Cannot create an HTTPRoute to ServiceExport/Import when underlying k8s service has the same name across multiple clusters

[evgeniy-khatko]

flowchart TD
subgraph config-cluster["configcluster&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;"];
    HTTPRoute --> service-X-import
    HTTPRoute -..-> service-X-import
end
subgraph cluster2["cluster2&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;"];
    service-X-import --> service-X-export --> service-X --> pods-X
end
subgraph cluster1["cluster1&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<br>&nbsp;"];
    service-X-import --> .service-X-export -->.service-X --> .pods-X
end

Loading

Precondition

As presented on visual, we have 2 separate clusters: cluster1 and cluster2. It's not uncommon to have the same service with the same name to be deployed into multiple clusters to achieve HA/DR capabilities. Note that service name is the same in both cluster - specifically "service-X".

Use case

We want to have an HTTPRoute in a "config" cluster to route requests evenly to service-X in both clusters. To achieve that, we create ServiceExports in cluster1 and cluster2. Note that current implementation would look at ServiceExport-s names to map them to respective Services in the cluster. Then we want to create ServiceImports in a "config" cluster. Note that current implementation would look at ServiceImport names to map them to respective ServiceExports from cluster1 and cluster2. Then we want to create an HTTPRoute with 2 backendRefs - one for ServiceImport of service-X from cluster1 and one for ServiceImport of service-X from cluster2. It would look similar to

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: route-to-service-X-accross-clusters
spec:
  parentRefs:
    - name: service-network-name
      sectionName: http[s]
  rules:
    - backendRefs:
        - name: service-X-import
          kind: ServiceImport
          port: 80
        - name: service-X-import
          kind: ServiceImport
          port: 80

Problem

Because resource names are used to map to actual services, we have a conflict in the routing manifest with two identical service imports representing the same service-X deployed to multiple clusters 1 and 2.

Potential solution

Use annotations on resources to map to Services and fallback to resource names (for backward compatibility). The manifests would look like:

---
apiVersion: application-networking.k8s.aws/v1alpha1
kind: ServiceExport
metadata:
  name: service-X-export-1
  annotations:
    application-networking.k8s.aws/federation: "amazon-vpc-lattice"
    application-networking.k8s.aws/service-name: "service-X"

---
apiVersion: application-networking.k8s.aws/v1alpha1
kind: ServiceExport
metadata:
  name: service-X-export-2
  annotations:
    application-networking.k8s.aws/federation: "amazon-vpc-lattice"
    application-networking.k8s.aws/service-name: "service-X"

---
apiVersion: application-networking.k8s.aws/v1alpha1
kind: ServiceImport
metadata:
  name: service-X-import-1
  annotations:
    application-networking.k8s.aws/export-name: "service-X-export-1"
spec:
  ports:
    - port: 80
      protocol: TCP
  type: Headless

---
apiVersion: application-networking.k8s.aws/v1alpha1
kind: ServiceImport
metadata:
  name: service-X-import-2
  annotations:
    application-networking.k8s.aws/export-name: "service-X-export-2"
spec:
  ports:
    - port: 80
      protocol: TCP
  type: Headless

---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: route-to-service-X-accross-clusters
spec:
  parentRefs:
    - name: service-network-name
      sectionName: http[s]
  rules:
    - backendRefs:
        - name: service-X-import-1
          kind: ServiceImport
          port: 80
        - name: service-X-import-2
          kind: ServiceImport
          port: 80

[evgeniy-khatko]

This is where mapping between Service inside target group and ServiceExport is hardcoded: https://github.com/aws/aws-application-networking-k8s/blob/main/pkg/gateway/model_build_targetgroup.go#L177.

Potentially this might be the only place where changes need to be done. If we decouple Service and ServiceExport, the rest becomes unblocked autometicaly.