Recovering a Kubernetes Crunchy Data PostgreSQL (PGO) cluster with no primary node

$ pgo test zercuritycluster : zercurity
Services
primary (10.103.189.241:5432): UP
pgbouncer (10.109.233.235:5432): UP
replica (10.109.210.110:5432): UP
Instances
replica (zercurity-758dd49969-nzltl): UP
replica (zercurity-erwj-5d5876dc64-gb676): UP
replica (zercurity-vcxh-574fd7cf8b-jdgxc): UP
$ pgo failover zercurity --query
$ pgo failover zercurity --target zercurity --force
WARNING: Are you sure? (yes/no): yes
Error: zercurity has not yet been upgraded. Please upgrade the cluster before running this Postgres Operator command.

Recovering our cluster manually

Fortunately, you’re able to failover a cluster manually using the patronictl failover command from with inside the PostgreSQL container.

$ kubectl -n pgo get po
$ kubectl -n pgo exec -i -t zercurity-758dd49969-nzltl  -- /bin/bash
$ patronictl failoverCandidate ['zercurity-nzltl', 'zercurity-erwj-wnlql', 'zercurity-vcxh-xmr64'] []: zercurity-nzltlCurrent cluster topology
+ Cluster: zercurity --+------------+---------+---------+----+-----+
| Member | Host | Role | State | T | Lag |
+----------------------+------------+---------+---------+---+------+
| zercurity-nzltl | 192.X.X.13 | Replica | running | 6 | 0 |
| zercurity-erwj-wnlql | 192.X.X.30 | Replica | running | 6 | 0 |
| zercurity-vcxh-xmr64 | 192.X.X.12 | Replica | running | 6 | 0 |
+----------------------+------------+---------+---------+----+-----+
Are you sure you want to failover cluster zercurity? [y/N]: y
$ patronictl list+ Cluster: zercurity --+------------+----------+---------+---+-----+
| Member | Host | Role | State | T | Lag |
+----------------------+------------+---------+----------+---+-----+
| zercurity-nzltl | 192.X.X.13 | Leader | running | 6 | |
| zercurity-erwj-wnlql | 192.X.X.30 | Replica | running | | |
| zercurity-vcxh-xmr64 | 192.X.X.12 | Replica | starting | | |
+----------------------+------------+---------+----------+---+-----+

Its all over!

We hope you found this helpful. Please feel free to get in touch if you have any questions.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Zercurity

Zercurity

Real-time security and compliance delivered.