Revise default config

[davidumea]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

Release notes

Default Opensearch setup has been changed. Opensearch is now configured to use 3 master nodes, 0 data nodes and 0 client nodes by default. There's no easy migration path for the new configuration, if the old default Opensearch setup is used it should be added to the override config.
Proxy protocol is now enabled by default in ingress-nginx when using ElastX as infrastructure provider.
Fluentd audit is now enabled by default.
Log retention is now 30 days by default.

Platform Administrator notice

Default Opensearch setup has been changed. Opensearch is now configured to use 3 master nodes, 0 data nodes and 0 client nodes by default. There's no easy migration path for the new configuration, if the old default Opensearch setup is used it should be added to the override config.
Proxy protocol is now enabled by default in ingress-nginx when using ElastX as infrastructure provider.
Fluentd audit is now enabled by default.

What does this PR do / why do we need this PR?

Changed some default configuration

• Part of https://github.com/elastisys/ck8s-issue-tracker/issues/352

Information to reviewers

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests

[viktor-f]

Very nice changes

[aarnq]

How come these have preferred while the rest is strict?

[aarnq]

Also, is there a reason why we don't set the default opensearch setup with only master nodes?

[davidumea]

Also, is there a reason why we don't set the default opensearch setup with only master nodes?

I thought of that and asked around a bit, and I think the main issue would be that there's no known migration path.

[davidumea]

How come these have preferred while the rest is strict?

I don't know, but it should be aligned. Do you think we should have preferred or strict everywhere?

[aarnq]

The easy migration path is to inform and preserve the config for existing environments.
So merge and lift any of the config that has changed to the override config.

I think we want to prefer a strict one, else we might get strange scenarios where we have a skew. Because this will mostly impact during the creation of new ones, as afterwards the PVCs will control which zone they land on.

[aarnq]

@davidumea do you intend to write a migration snippet to preserve the opensearch setup if one already uses dedicated nodes?
Which would be to yq_dig and then yq_add the results for those options into the override config before init runs.

[davidumea]

👍 I added something for that 6302a38