K8SPSMDB-929: Add replicaSet horizons

[egegunes]

CHANGE DESCRIPTION

Problem:
If cluster is exposed, mongo clients can't connect to ReplicaSet with ?replicaSet=<rsName>.

Cause:
mongod doesn't know about external addresses.

Solution:
We need to allow users to configure split-horizon DNS. We're adding a new field (horizons) to ReplsetSpec, and users will be able to configure horizons:

  replsets:
  - name: rs0
    expose:
      enabled: true
      exposeType: LoadBalancer
    horizons:
      cluster1-rs0-0:
        external: rs0-0.egedemo.xyz
        external-2: rs0-0.egedemo2.xyz
      cluster1-rs0-1:
        external: rs0-1.egedemo.xyz
        external-2: rs0-1.egedemo2.xyz
      cluster1-rs0-2:
        external: rs0-2.egedemo.xyz
        external-2: rs0-2.egedemo2.xyz

which will turn into the following replica set configuration:

[
        {
                "_id" : 0,
                "host" : "cluster1-rs0-0.cluster1-rs0.psmdb.svc.cluster.local:27017",
                "arbiterOnly" : false,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 2,
                "tags" : {
                        "serviceName" : "cluster1",
                        "podName" : "cluster1-rs0-0"
                },
                "horizons" : {
                        "external" : "rs0-0.egedemo.xyz:27017",
                        "external-2" : "rs0-0.egedemo2.xyz:27017"
                },
                "secondaryDelaySecs" : NumberLong(0),
                "votes" : 1
        },
        {
                "_id" : 1,
                "host" : "cluster1-rs0-1.cluster1-rs0.psmdb.svc.cluster.local:27017",
                "arbiterOnly" : false,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 2,
                "tags" : {
                        "podName" : "cluster1-rs0-1",
                        "serviceName" : "cluster1"
                },
                "horizons" : {
                        "external" : "rs0-1.egedemo.xyz:27017",
                        "external-2" : "rs0-1.egedemo2.xyz:27017"
                },
                "secondaryDelaySecs" : NumberLong(0),
                "votes" : 1
        },
        {
                "_id" : 2,
                "host" : "cluster1-rs0-2.cluster1-rs0.psmdb.svc.cluster.local:27017",
                "arbiterOnly" : false,
                "buildIndexes" : true,
                "hidden" : false,
                "priority" : 2,
                "tags" : {
                        "podName" : "cluster1-rs0-2",
                        "serviceName" : "cluster1"
                },
                "horizons" : {
                        "external" : "rs0-2.egedemo.xyz:27017",
                        "external-2" : "rs0-2.egedemo2.xyz:27017"
                },
                "secondaryDelaySecs" : NumberLong(0),
                "votes" : 1
        }
]

Limitations:

• Connecting with horizon domains only supported if the client connects using TLS certificates.
• Duplicate domain names in horizons are forbidden by MongoDB.
• Using IP addresses in horizons are forbidden by MongoDB.
• Horizons needs to be set for all pods or none. You can't have horizons only for some pods.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?
• Are OpenShift compare files changed for E2E tests (compare/*-oc.yml)?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are the manifests (crd/bundle) regenerated if needed?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported MongoDB version?
• Does the change support oldest and newest supported Kubernetes version?

[egegunes]

@spron-in please review the cr.yaml and let me know if you want to change the interface.

[inelpandzic]

Maybe use DefaultMongoDBPort.

[egegunes]

I would but its a int32 and I need to convert it to string and this place will be very messy.

[llabordehpe]

This "horizons" feature should allow you to chose ports, and not force the exclusive use of port 27017.

[JNKPercona]

Test name	Status
arbiter	passed
balancer	passed
custom-replset-name	passed
cross-site-sharded	passed
data-at-rest-encryption	passed
data-sharded	passed
demand-backup	passed
demand-backup-eks-credentials	passed
demand-backup-physical	passed
demand-backup-physical-sharded	passed
demand-backup-sharded	passed
expose-sharded	passed
ignore-labels-annotations	passed
init-deploy	passed
finalizer	passed
limits	passed
liveness	passed
mongod-major-upgrade	passed
mongod-major-upgrade-sharded	passed
monitoring-2-0	passed
multi-cluster-service	passed
non-voting	passed
one-pod	passed
operator-self-healing-chaos	passed
pitr	passed
pitr-sharded	passed
pitr-physical	passed
recover-no-primary	passed
rs-shard-migration	passed
scaling	passed
scheduled-backup	passed
security-context	passed
self-healing-chaos	passed
service-per-pod	passed
serviceless-external-nodes	passed
smart-update	passed
split-horizon	passed
storage	passed
tls-issue-cert-manager	passed
upgrade	passed
upgrade-consistency	passed
upgrade-consistency-sharded	passed
upgrade-sharded	passed
users	passed
version-service	passed
We run 45 out of 45

commit: f790869
image: perconalab/percona-server-mongodb-operator:PR-1293-f7908692