Allow update of CassandraDatacenter to be processed to StatefulSets i…

[burmanm]

…f the Datacenter is running in unhealthy Kubernetes configuration. These could be scheduling related or pods crashing

What this PR does:

Which issue(s) this PR fixes:
Fixes #583

Checklist

• Changes manually tested
• Automated Tests added/updated
• Documentation added/updated
• CHANGELOG.md updated (not required for documentation PRs)
• CLA Signed: DataStax CLA

[adejanovski]

I was able to test this with a failed upgrade to a version of Cassandra that doesn't exist.
But I ran into a problem with the following scenario:

• I created a cluster with 1G for initial and max heap.
• Once it was started, I updated it with a 256M initial heap size and 128M max heap size.
• The pod never gets more than one restart, so any fix wouldn't be applied.
• I tried to use forceUpgradeRacks then, but it didn't apply the fix either:

  forceUpgradeRacks:
    - rack1
  managementApiAuth:
    insecure: {}
  racks:
    - name: rack1

Here's my full manifest:

apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: test-cluster
  namespace: cass-operator
spec:
  clusterName: development
  config:
    cassandra-yaml:
      authenticator: PasswordAuthenticator
      authorizer: CassandraAuthorizer
      num_tokens: 16
      role_manager: CassandraRoleManager
    jvm-server-options:
      initial_heap_size: 1G
      max_heap_size: 1G
  configBuilderResources: {}
  managementApiAuth:
    insecure: {}
  racks:
    - name: rack1
  resources:
    requests:
      cpu: 1500m
      memory: 2Gi
  serverType: cassandra
  serverVersion: 4.1.5
  size: 3
  storageConfig:
    cassandraDataVolumeClaimSpec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: standard