Hey @AceHack , sorry for the delayed response. Things have been pretty crazy, and we have been blocked on the Kubebuilder v2 migration. Finally able to get into these other issues.

In general, Solr is relatively easy to run via multiple kubernetes clusters. What you need to do is make sure that you are using 1 zookeeper cluster for the solr clouds running in all kube clusters, so that they all share the same zookeeper connection information in their SolrCloud Specs. The data volumes don't have to be linked at all, since each Solr node manages it's data independently of other Solr Nodes.

We use different base ingress URLs to differentiate SolrCloud nodes in each kubernetes cluster.

So the solr-operator would be started in Kube cluster A with --base-ingress-domain a.test.com and in Kube cluster B with --base-ingress-domain b.test.com. Then when you create a SolrCloud example in each kube cluster (both connecting to the exact same zookeeper connection information), the nodes will register under their cluster-specific domain names. You just need to make sure that your DNS rules are set up correctly to route a.test.com and b.test.com to the correct ingress controllers.

We are hopefully going to be able to support more addressability options in the near future, such as ExternalDNS, and give you more control over how things are created/named.

We don't want to run this in multiple clusters, we have 1 kubernetes cluster that spans 3 AZs. Currently, we have 3 different stateful sets (1 per AZ) so that Solr can be AZ aware in its autoscaling policies. It also used to be a nightmare with PVs/PVCs until the WaitForFirstConsumer volumeBindingMode for StorageClasses came out. To really support multi-AZ, I should be able to set a solr autoscaling policy that can reference the different AZs within the cluster.

I would really like to set a policy that looks something like

{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": "#ANY"}
or

{"replica": "#EQUAL", "shard": "#EACH", "sysprop.zone": ["us-east-1a", "us-east-1b", "us-east-1c"]}

More info
https://lucene.472066.n3.nabble.com/Rule-based-replication-or-sharding-td4407125.html#a4407660

I completely understand, and don't want the solr-operator to make any assumptions on how people configure their Solr clouds or Kube clusters.

Hopefully we can find a way to pass Node props to the environment variables of Solr. If that's not going to be an option, we might have to look into creating multiple statefulSets for each SolrCloud with multiple AZs.

Linking discussion on creating Kube options to pass Node labels to Pods: kubernetes/kubernetes#40610

It's possible in an init container or part of the main container startup you can read what AZ you are in and write it to a file or something. Requires calling kubernetes APIs.

@sepulworld, to follow up on the post you made in the PR about autoscaling. I was doing some research and it looks like Kubernetes 1.16 gives us an additional pod Option: Pod.Spec. topologySpreadConstraints, documented here https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/

This would allow us to generically spread out pods across nodes (and AZs) without having to use autoscaling.

Without the solr Auto scaling policy being AZ aware, no matter how the pod spread out you can’t be sure the replicas are spread out across AZs.

Got a start on implementing this functionality. Let me know how y'all think it could be improved. This was just the first way I could get it done after tinkering for a while. I've very open to suggestions and improvements.

Also we should be good to go on the availability zone thing once the Labels PR & Collection Policy PR are merged, correct? Do y'all have an issue with me merging the policy PR now, or should it be blocked on this?

hi - is there any update on this?

Hey @alittlec , sorry didn't see your comment come in. Currently there are no plans to merge the currently linked PR. Instead in the v0.5.0 release we plan on support Pod Topology Spread Constraints.

The idea of autoscaling within Solr is completely changing from 8.x to 9.0, with the autoscaling framework deprecated and not recommended for production use. We will definitely need to investigate how the Solr Operator ties into the future autoscaling features starting with Solr 9.0.

Hi @HoustonPutman ,

I was wondering if you had a chance to investigate this concept within the operator for Solr 9.x?