feat: operator v2 (for Hubble control plane)

[huntergregory]

Description

Writing a new operator to support Hubble in non-Cilium clusters (see #417). This operator manages all CiliumIdentities and CiliumEndpoints for the cluster.

> ./retina-operator -h
Start Retina Operator

Usage:
  retina-operator [flags]
  retina-operator [command]

Available Commands:
  completion         Generate the autocompletion script for the specified shell
  help               Help about any command
  manage-cilium-crds Start the Retina operator for Hubble control plane

Flags:
      --config string            config file (default "retina/operator-config.yaml")
      --enable-leader-election   Enable leader election for controller manager.
  -h, --help                     help for retina-operator
      --metrics-addr string      The address the metric endpoint binds to. (default ":8080")
      --probe-addr string        The address the probe endpoint binds to. (default ":8081")

Use "retina-operator [command] --help" for more information about a command.

> ./retina-operator manage-cilium-crds  -h
Start the Retina operator for Hubble control plane

Usage:
  retina-operator manage-cilium-crds [flags]
  retina-operator manage-cilium-crds [command]

Available Commands:
  hive        Inspect the hive

Flags:
      --allocator-list-timeout duration                      timeout to list initial allocator state (default 3m0s)
      --cilium-endpoint-gc-interval duration                 GC interval for cilium endpoints (default 5m0s)
      --cluster-id uint32                                    Unique identifier of the cluster
      --cluster-name string                                  Name of the cluster (default "default")
      --config string                                        Configuration file (to configure the operator, this argument is required)
      --config-dir string                                    Configuration directory that contains a file for each option
      --controller-group-metrics strings                     List of controller group names for which to to enable metrics. Accepts 'all' and 'none'. The set of controller group names available is not guaranteed to be stable between Cilium versions.
  -D, --debug                                                Enable debugging mode
      --enable-cilium-endpoint-slice                         If set to true, the CiliumEndpointSlice feature is enabled. If any CiliumEndpoints resources are created, updated, or deleted in the cluster, all those changes are broadcast as CiliumEndpointSlice updates to all of the Cilium agents.
      --enable-k8s                                           Enable the k8s clientset (default true)
      --enable-k8s-api-discovery                             Enable discovery of Kubernetes API groups and resources with the discovery API
      --enable-k8s-endpoint-slice                            Enables k8s EndpointSlice feature in Cilium if the k8s cluster supports it (default true)
      --enable-metrics                                       Enable Prometheus metrics
      --enable-telemetry                                     enable telemetry (send logs and metrics to a remote server)
  -h, --help                                                 help for v2
      --identity-allocation-mode string                      Identity allocation mode (default "crd")

> ./retina-operator
Starting Retina Operator
ts=2024-06-17T17:30:29.829-0400 level=info caller=legacy/deployment.go:87 msg="Starting legacy operator undefined"

> ./retina-operator v2
...
time="2024-06-17T17:31:44-04:00" level=info msg="starting hive. Some logs will say 'cilium' since some code is derived from cilium" subsys=operator
time="2024-06-17T17:31:44-04:00" level=info msg=Invoked duration="66.167µs" function="github.com/microsoft/retina/operator/cmd/hubble.init.func10 (cmd/hubble/cells.go:47)" subsys=hive
ts=2024-06-17T17:31:44.919-0400 level=info caller=hubble/zap.go:60 msg="Traces telemetry initialized with zapai" version= appInsightsID=
ts=2024-06-17T17:31:44.919-0400 level=info caller=cell/invoke.go:40 msg=Invoked duration=0.016847234 subsys=hive function="github.com/microsoft/retina/operator/cmd/hubble.setupZapHook (cmd/hubble/zap.go:34)"
ts=2024-06-17T17:31:44.919-0400 level=info caller=cell/invoke.go:40 msg=Invoked function="pprof.init.func1 (pkg/pprof/cell.go:50)" subsys=hive duration=0.000154826

Related Issue

Fixes #419

Checklist

• I have read the contributing documentation.
• I signed and signed-off the commits (git commit -S -s ...). See this documentation on signing commits.
• I have correctly attributed the author(s) of the code.
• I have tested the changes locally.
• I have followed the project's style guidelines.
• I have updated the documentation, if necessary.
• I have added tests, if applicable.
  • Includes unit tests.
  • Will have to add e2e tests in follow-up PR.

Screenshots (if applicable) or Testing Completed

Additional Notes

[huntergregory]

Thanks Jacques for all the work here. I think we should revert all changes around the licenses, and I had some questions about other changes.

Also, can you add details in the PR description about your testing for legacy and new operator?

[huntergregory]

changes lgtm. Just ought to rebase on #432

[anubhabMajumdar]

Look good. Can you verify this works with the new retina-agent changes? Update Testing Done to show enrichment of flows works when using Hubble cli.

[jimassa]

Look good. Can you verify this works with the new retina-agent changes? Update Testing Done to show enrichment of flows works when using Hubble cli.

I added images in test section showing Hubble cli working with agent changes.