[Housekeeping] Make Ingress configuration more generic in flyte-core

[davidmirror-ops]

Describe the issue

When flyte-core is deployed, several resources exhibit a behavior that could affect the ability to troubleshoot effectively:

1. With albSSLRedirect: true it adds an Ingress rule that points to a non-existent service:

spec:
  rules:
  - host: flyte-the-hard-way.uniondemo.run
    http:
      paths:
      - backend:
          service:
            name: ssl-redirect
            port:
              name: use-annotation
        path: /*
        pathType: ImplementationSpecific

Then, a kubectl describe on the deployed Ingress shows an error:

Rules:
  Host                              Path  Backends
  ----                              ----  --------
  flyte-the-hard-way.uniondemo.run
                                    /*               ssl-redirect:use-annotation (<error: endpoints "ssl-redirect" not found>)
...

Was the idea to use the port from this annotation?:

alb.ingress.kubernetes.io/ssl-redirect: "443"

2. The base values.yaml file includes hardcoded annotations for the nginx Ingress controller:

annotations:
     nginx.ingress.kubernetes.io/app-root: /console

Even when using another controller, the annotations end up being applied to the deployed resource which could be confusing.

What if we do not do this?

It affects the User Experience and potentially causes unnecessary distraction during a troubleshooting process.

Related component(s)

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[eapolinario]

For the CRD issue, https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#some-caveats-and-explanations provides some explanations. The tldr is that helm3 doesn't have support for upgrading/deleting CRDs, only in a very limited fashion (namely non-templated CRDs, which is not the case of they flyteworkflows.flyte.lyft.com). Because of that I think we should discuss this in a separate issue. Can you open one, @davidmirror-ops ?