Multiple gateway providers in the same cluster #7

guicassolato · 2022-12-13T10:45:37Z

No description provided.

…ailure

eguzki · 2023-01-18T16:59:30Z

For me the gateway provider should be per gateway basis instead of per kuadrant instance basis. The gateway, represented by the Gateway API's Gateway resource, should specify the provider. Actually, the gatewayClassName was intended for that. I think that the kuadrant control plane should rely on attributes read from the gateway to know how to configure it. If the gatewayClassName is not enough, maybe kuadrant can define some label/annotation that must be in the gateway object to have kuadrant configure it for rate limiting and auth.

For me this is the most flexible way. Less prone to errors like trying to configure gateways with some provider specified in the Kuadrant CR and the actual provider is other one.

I do not know if it is a real use case, but kuadrant might be configuring multiple gateways managed by different gateway providers at the same time.

guicassolato · 2023-01-19T17:24:22Z

rfcs/0000-multiple-control-plane-providers.md

+
+```yaml
+apiGroup: maistra.io/v1
+kind: ServiceMeshControlPlane


Suggested change

kind: ServiceMeshControlPlane

kind: ServiceMeshMemberRoll

guicassolato · 2023-01-19T17:53:43Z

@eguzki

For me the gateway provider should be per gateway basis instead of per kuadrant instance basis. The gateway, represented by the Gateway API's Gateway resource, should specify the provider. Actually, the gatewayClassName was intended for that. I think that the kuadrant control plane should rely on attributes read from the gateway to know how to configure it.

The gatewayClassName is just the name of a GatewayClass resource, so we'd need to read this other resource. Unfortunately the only thing that the GatewayClass resource guarantees is another field called controllerName, which may or may not give enough hint to infer the provider.

If the gatewayClassName is not enough, maybe kuadrant can define some label/annotation that must be in the gateway object to have kuadrant configure it for rate limiting and auth.

An annotation containing the info we need to infer the provider is a valid approach; likely that would be in the GatewayClass instead of each Gateway, since all gateways under a same GatewayClass must be implemented by the same provider.

[This actually gives me another idea...]

For me this is the most flexible way. Less prone to errors like trying to configure gateways with some provider specified in the Kuadrant CR and the actual provider is other one.

You raise a good point here. By just dumping a list of gateway providers in the Kuadrant CR (as I originally proposed), we get nothing that tells us which gateways are provided by each provider. It may suffice to inject configurations that apply to the entire provider, but not enough to cover occasional differences between providers when configuring the gateways themselves.

[...and this is the idea:]

So perhaps a couple improvements to this proposal could be:

Instead of listing "gateway providers" in the Kuadrant CR, users could list GatewayClass resources;
All Gateway resources referring GatewayClasses listed in the Kuadrant CR are to be managed by Kuadrant (cluster admins can always create more GatewayClasses to split the sets of gateways that will be managed by Kuadrant from others the won't);
Together with each GatewayClass referenced in the Kuadrant CR, the user would also provide the additional reference to the provider.

I do not know if it is a real use case, but kuadrant might be configuring multiple gateways managed by different gateway providers at the same time.

I don't think this is possible as definition enforced by the gatewayClassName, which always points to one provider.

eguzki · 2023-01-19T18:12:47Z

I don't think this is possible as definition enforced by the gatewayClassName, which always points to one provider.

I was referring to having multiple gateways with a different class (i.e. different provider). For example, one managed by Istio while other managed by, let's say, Gateway Envoy control plane.

guicassolato · 2023-01-23T09:32:48Z

I don't think this is possible as definition enforced by the gatewayClassName, which always points to one provider.

I was referring to having multiple gateways with a different class (i.e. different provider). For example, one managed by Istio while other managed by, let's say, Gateway Envoy control plane.

My point was that the name of the class itself may not be enough to infer the kind of the provider.

maleck13

My preference here is to go with explicit configuration and a single provider per kuadrant instance

maleck13 · 2023-02-16T11:04:27Z

rfcs/0000-multiple-control-plane-providers.md

+**Example 1.** Kuadrant instance for gateways provided by 2 OSSM service meshes.
+
+In the example provided in the previous section with 2 OSSM service meshes listed in the `gatewayProviders` field of the `Kuadrant` CR, the Operator would look up for 2 `ServiceMeshControlPlane` resources, named `basic` and `enterprise`, defined in the `istio-system`. In each of these resources, the Operator would edit the `spec` field so the following entry is present:
+


Beyond the scope of this but I think it should be clear in any docs we have that by defining a gateway provider, you are asking the kuadrant operator to patch your config resource

maleck13 · 2023-02-16T12:03:06Z

rfcs/0000-multiple-control-plane-providers.md

+1. the Kuadrant policies (`RateLimitPolicy` and `AuthPolicy`) and their targeted network resources, that ultimately link all the way with gateways and the gateway providers; or
+2. (more directly) the `Kuadrant` custom resources and their corresponding gateway objects, linked via labels and label selectors (see [architecture#5:124-126](https://github.com/Kuadrant/architecture/blob/9545f760b66ff9aaddeeaab126d5276a086e228c/rfcs/0000-wire-policies-with-backend-services.md?plain=1#L124-L126)).
+
+While option 1 above implies a lazy configuration of the functional components into the gateway providers, option 2 perhaps could be implemented in a rather proactive fashion, such as the one proposed in this RFC.


not sure I am a fan of the lazy configuration. Or I should say, I like the hands off approach esp for developer exp, but I think I would want that to be an explicit opt in. It is the role of the cluster-admin to decide to install kuadrant and it should be up to that persona to decide which gateway providers it applies to. It feels like doing this lazy config moves that reponsibility (perhaps unexpectedly) into the hands of the person defining a RLP for example?

maleck13 · 2023-02-16T12:05:46Z

rfcs/0000-multiple-control-plane-providers.md

+# Unresolved questions
+[unresolved-questions]: #unresolved-questions
+
+1. Is having multiple service mesh control planes (i.e. multiple gateway providers) within the same cluster a real use case or rather just a technical possibility that we choose not to support?<br/>


it seems it is a real scenerio based on recent conversations where there may be two control planes one for the gateway and one for the mesh

maleck13 · 2023-02-16T12:06:28Z

rfcs/0000-multiple-control-plane-providers.md

+
+1. Is having multiple service mesh control planes (i.e. multiple gateway providers) within the same cluster a real use case or rather just a technical possibility that we choose not to support?<br/>
+  **_A:_**<br/><br/>
+2. Does a Kuadrant instance need to be able to target more than one control plane/gateway provider? (See [Rationale and alternatives](#rationale-and-alternative).)<br/>


This seems less clear. I see a clear use case for defining which provider an instance of Kuadrant should target, but less so the need for one instance of kuadrant to target multiple providers

maleck13 · 2023-02-16T12:08:01Z

rfcs/0000-multiple-control-plane-providers.md

+  **_A:_**<br/><br/>
+2. Does a Kuadrant instance need to be able to target more than one control plane/gateway provider? (See [Rationale and alternatives](#rationale-and-alternative).)<br/>
+  **_A:_**<br/><br/>
+3. Should targeting a control plane/gateway provider be required to be explicit (e.g. in a `Kuadrant` CR) or it can based on convention/heuristic (e.g. default to Istio)? _Note:_ I am personally inclined to it needs to be explicit, hence the proposal in this RFC.<br/>


I am a fan of explicit configuration for this type of thing over potentially hidden logic

maleck13 · 2023-02-16T12:09:03Z

rfcs/0000-multiple-control-plane-providers.md

+  **_A:_** Istio and OpenShift Service Mesh (OSSM).<br/><br/>
+5. What other gateway providers could we aim to support in the future? _Possible answers:_ [Envoy Gateway](https://gateway.envoyproxy.io/v0.2.0/), [Contour](https://projectcontour.io), other API gateways not based on Envoy, ...<br/>
+  **_A:_** Only gateways and perhaps sidecar proxies based on Envoy for quite some time, likely.<br/><br/>
+6. Is "gateway provider" an adequate term? What about supporting injecting Kuadrant into sidecar proxies in the future?<br/>


Gateway provider seems adequate a sidcar could be thought of as another form of gateway

maleck13 · 2023-02-16T12:09:37Z

rfcs/0000-multiple-control-plane-providers.md

+  **_A:_** Only gateways and perhaps sidecar proxies based on Envoy for quite some time, likely.<br/><br/>
+6. Is "gateway provider" an adequate term? What about supporting injecting Kuadrant into sidecar proxies in the future?<br/>
+  **_A:_** We'll stay with "gateway providers" for now. This can be changed in the future. Sidecars may become a thing one day; this should be either transparent to the scope of this RFC or changeable in the future as part of releasing a new API.<br/><br/>
+7. Should the reconciliation of a `Kuadrant` CR be considered successful only if all gateway providers indicated in the spec were successfuly configured for that instance of Kuadrant? What shall the Operator do in case some of the gateway providers to fail to be configured while others succeed? Shall the Operator roll back the successful configurations in case of partial failure? Should the Operator reflect the state of the reconciliation of each gateway provider listed in the spec in the status subresource of the `Kuadrant` CR? (See [Troubleshooting](#troubleshooting) for possible reasons for the reconciliation to fail.)<br/>


Yes I think so and it is probably another good reason to have one provider per instance

rfc: Multiple gateway providers in the same cluster

2feb371

guicassolato added the RFC Request For Comments label Dec 13, 2022

guicassolato self-assigned this Dec 13, 2022

guicassolato added 3 commits December 14, 2022 17:39

added troubleshooting section and question on what to do in case of f…

1b037c2

…ailure

added the alternative of implicit reference to the gateway providers

cd0b225

unresolved questions cleanup

65731d4

guicassolato commented Jan 19, 2023

View reviewed changes

guicassolato mentioned this pull request Jan 31, 2023

Multiple kuadrant instances #5

Closed

guicassolato added the target/current label Feb 9, 2023

maleck13 reviewed Feb 16, 2023

View reviewed changes

didierofrivia mentioned this pull request May 9, 2023

Fix Mesh provider selection Kuadrant/kuadrant-operator#183

Closed

alexsnaps mentioned this pull request Jul 6, 2023

Define how Kuadrant lets the user configure "some" of the underlying services Kuadrant/kuadrant-operator#164

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multiple gateway providers in the same cluster #7

Multiple gateway providers in the same cluster #7

guicassolato commented Dec 13, 2022

eguzki commented Jan 18, 2023 •

edited

Loading

guicassolato Jan 19, 2023

guicassolato commented Jan 19, 2023 •

edited

Loading

eguzki commented Jan 19, 2023

guicassolato commented Jan 23, 2023

maleck13 left a comment

maleck13 Feb 16, 2023

maleck13 Feb 16, 2023

maleck13 Feb 16, 2023

maleck13 Feb 16, 2023

maleck13 Feb 16, 2023

maleck13 Feb 16, 2023

maleck13 Feb 16, 2023

		Example 1. Kuadrant instance for gateways provided by 2 OSSM service meshes.

		In the example provided in the previous section with 2 OSSM service meshes listed in the `gatewayProviders` field of the `Kuadrant` CR, the Operator would look up for 2 `ServiceMeshControlPlane` resources, named `basic` and `enterprise`, defined in the `istio-system`. In each of these resources, the Operator would edit the `spec` field so the following entry is present:

Multiple gateway providers in the same cluster #7

Are you sure you want to change the base?

Multiple gateway providers in the same cluster #7

Conversation

guicassolato commented Dec 13, 2022

eguzki commented Jan 18, 2023 • edited Loading

Choose a reason for hiding this comment

guicassolato commented Jan 19, 2023 • edited Loading

eguzki commented Jan 19, 2023

guicassolato commented Jan 23, 2023

maleck13 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eguzki commented Jan 18, 2023 •

edited

Loading

guicassolato commented Jan 19, 2023 •

edited

Loading