Help - Adding non-Kubernetes workloads to your mesh #12349
Replies: 11 comments 9 replies
-
|
Hi there. Thank you for the detailed description of the issue. We can help with that. So here are a couple of thoughts. The panics that you see in the non k8s proxy Reaching in-cluster workloads from the non-k8s workload. Logs in k8s proxy Let us know when you check these things and we can take it further. |
Beta Was this translation helpful? Give feedback.
-
|
@zaharidichev - Thanks for the prompt reply, I was able to resolve the issue. Below are the details.
There was a typo in the LINKERD2_PROXY_IDENTITY_SPIRE_SOCKET from spire-agent logs Corrected I've corrected that and restarted the proxy. Not seeing any panic errors now.
I was able to connect to the external-workload from
The only thing that I noticed is, I am seeing my linkerd-proxy logs on the external-workload flooded with these Am I missing something in the config? |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, I am glad things have been resolved. Let me answer your questions one by one:
Configuring it as root was just for the purpose of the demo. There is no restriction as to which user the proxy runs as. In order to configure IPtables you need a privileged user, but this can be done out of band.
This is correct
This is interesting. What happens there is that the proxy tries to contact the policy controller and provides it with a string that describes the workload. What is the value of {
"ns": "mixed-env",
"external_workload": "external-workload"
}In the docs, there are a number of excape chars in order to make it work. Do these get processed correctly on your environment. Is the policy controller logging some errors where it cannot find the workload? |
Beta Was this translation helpful? Give feedback.
-
|
The environment vars look ok to me, apart from the spacing, not sure if that matters. Also, the traffic is not balanced between the external-workload and in-cluster workload. As per the docs, I have create a service that selects over both the machine as well as an in-cluster workload |
Beta Was this translation helpful? Give feedback.
-
The traffic should be balanced if both endpoints ara available. Due to the fact that this is a latency-sensitive LB, it could happen that one endpoint is favored over another for a period of time. Try scaling the in-cluster workload to 0 and you should observe receiving responses from the EC2 instance only.
They look fine but the policy controller seems to not be able to find these workloads. Are there any logs in the policy controller that look suspicious? |
Beta Was this translation helpful? Give feedback.
-
I scaled down the in-cluster workload, however, I noticed that the legacy-app svc doesn't have any endpoints, when scaled up both the After scaling down And requests fail due to endpoints not being available. Is is because these services use a common label Shouldn't the external-workloads service
I believe the problem was with the External Workload CRDs version as policy controller pods were complaining about meshTLS not being found(see error snippet below), while the external workload is created using
The |
Beta Was this translation helpful? Give feedback.
-
|
I was able to resolve this problem, by running the external-workload as a non-root user(since the proxy is configured using root user which ignores all traffic from the root. Thanks a lot @zaharidichev for you prompt response 🥂 . |
Beta Was this translation helpful? Give feedback.
-
|
I would like a response on the service endpoints for external-workloads. |
Beta Was this translation helpful? Give feedback.
-
Your ExternalWorkload should have a apiVersion: workload.linkerd.io/v1beta1
kind: ExternalWorkload
metadata:
creationTimestamp: "2024-03-27T09:51:51Z"
generation: 1
name: test-123
namespace: default
resourceVersion: "65937"
uid: d98788e3-4ce8-4312-9862-a17502f36d5d
spec:
meshTLS:
identity: test-123.default.serviceaccount.identity.linkerd.cluster.local
serverName: test-123.default.cluster.local
ports:
- name: null
port: 80
protocol: null
workloadIPs:
- ip: 192.0.2.0
status:
conditions:
- lastTransitionTime: "2024-03-27T09:51:51Z"
message: Workload created
reason: WorkloadCreated
status: "True"
type: ReadyYou can edit your resource and add a |
Beta Was this translation helpful? Give feedback.
-
|
Hi there, it seems to me you are looking at endpoints. The controller works by creating slices so you should try: Also, yes you need to have a Ready status on this workload resource in order for it to be reflected in the endpoint slices list. Can you verify that all works? |
Beta Was this translation helpful? Give feedback.
-
|
I've added that but I still don't see any endpoints for the service Still no endpoints I've even added a additional label selector |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I do see an endpointslices now |
Beta Was this translation helpful? Give feedback.
-
|
However, requests from the EC2 instances are still failing with the From proxy logs on the Ec2 instance |
Beta Was this translation helpful? Give feedback.
-
|
You are targeting the service that selects only over pods that are in the cluster. Try targeting http://legacy-app.mixed-env.svc.cluster.local:80/who-am-i |
Beta Was this translation helpful? Give feedback.
-
|
Actually, what are you trying to do here? You are hitting the legacy-app-cluster from the VM and it is failing while the app is scaled down. It should fail. There is no pod running I suppose |
Beta Was this translation helpful? Give feedback.
-
|
That was a typo mistake on my end, I used the correct endpoint now and it still doesn't work |
Beta Was this translation helpful? Give feedback.
-
|
Alright this is a known limitation. If you are on a VM and you are targeting a service on the cluster that points back to the VM itself, this traffic pattern will not work. Is there a particular usecase that you are trying to solve for or just experimenting? |
Beta Was this translation helpful? Give feedback.
-
|
You can load balance between a VM and an in-cluster pod if your client is not the VM (it could be another VM or another pod in the cluster) |
Beta Was this translation helpful? Give feedback.
-
|
I am just experimenting here, I saw this in the proxy logs as well as it was trying to route the request to the service and back to the VM, which is why it is failing. I understand this now. Thanks for the explanation. Also for a This describes creating a Server resource to deny traffic, however our default policy is set to |
Beta Was this translation helpful? Give feedback.
-
|
@ranjit-se7en it is not really mandatory for a default policy to be configured. Traffic is denied on the inbound side, though, not on the outbound side (we don't limit where we can establish connections, we limit who can establish connections to us). If you want to configure a deny policy to play with this, here's an example set of resources: apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
annotations:
# Default here says: if there's no other policy created that
# targets this Server, deny everything
# this is _not_ mandatory though
config.linkerd.io/default-inbound-policy: deny
name: <NAME>
namespace: <NAMESPACE>
spec:
externalWorkloadSelector:
matchLabels:
<REPLACE-WITH-YOUR-LABELS>
port: <REPLACE-WITH-YOUR-PORT>
proxyProtocol: HTTP/1
This is the easiest way to get started, just make sure your Server targets a set of workloads using the externalWorkloadSelector. If you want to instead target pods (i.e. restrict traffic to pod from VMs) then use If you want something more fine grained, apply a policy that targets this server: ---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
name: policy-<name>
namespace: <namespace>
spec:
requiredAuthenticationRefs:
- group: policy.linkerd.io
kind: MeshTLSAuthentication
name: policy-<name>-mtls
targetRef:
group: policy.linkerd.io
kind: Server
name: <server-name>
---
apiVersion: policy.linkerd.io/v1alpha1
kind: MeshTLSAuthentication
metadata:
name: policy-<name>-mtls
namespace: <namespace>
spec:
identities:
# Name and namespace of svc acc you want to allow. This can also be
# a spiffe identity
- "<name>.<namespace>.serviceaccount.identity.linkerd.cluster.local"Hope this makes sense. All of these examples are to limit traffic to a VM from a set of a pods. |
Beta Was this translation helpful? Give feedback.
-
We are working on a POC to mesh
external-workloads, and referenced this https://linkerd.io/2.15/tasks/adding-non-kubernetes-workloads/# guide. However, it's not working as intended and we would like some help.We are using a spire server/agent running on the same EC2 instance with
aws_iidnode attestation and AWS PCA upstream.Additionally, we have connectivity between the EC2 instance and K8s worker nodes.
Versions
Spire agent is attested
external-workload is registered
However, We are seeing admin panic errors in the linkerd-proxy logs on EC2. Note we used the same linkerd-proxy version which is installed in the K8s cluster.
If we use the service endpoint IPs to make a curl request from EC2 external-workload instance it goes through/
If we make a curl request to the EC2 external-workload instance IP, from the client pod it doesn't work and times out.
Seeing the admin panicked errors in the EC2 linkerd-proxy logs where external-workload is running.
Followed by steam of unknown service errors in the EC2 linkerd-proxy logs
Looking at the K8s linkerd-proxy sidecar logs from the client pod. It shows a similar error as seen from the linkerd-proxy logs on the EC2 instance.
Beta Was this translation helpful? Give feedback.
All reactions