Port forwarding failing to start (using Argo Rollouts)

[mkonecny-atlassian]

Expected behavior

When using skaffold dev --port-forward the ports I have setup in my YAML would automatically port-forward.

Actual behavior

We are using Argo Rollout in our Helm chart (not sure if this matters). When using the above I see the following:

Saving 1 charts
Deleting outdated charts
NAME: dx-ref-java-quarkus-im
LAST DEPLOYED: Mon Jan 13 16:57:25 2025
NAMESPACE: dx-ref-java-quarkus-im
STATUS: deployed
REVISION: 1
TEST SUITE: None
Waiting for deployments to stabilize...
Deployments stabilized in 12.8455ms
port forwarding pod-dx-ref-java-quarkus-im-webserver-dx-ref-java-quarkus-im-7070 got terminated: output: Error from server (NotFound): pods "dx-ref-java-quarkus-im-webserver" not found

port forwarding service-dx-ref-java-quarkus-im-service-canary-dx-ref-java-quarkus-im-80 got terminated: output: error: unable to forward port because pod is not running. Current status=Pending

port forwarding service-dx-ref-java-quarkus-im-service-dx-ref-java-quarkus-im-80 got terminated: output: error: unable to forward port because pod is not running. Current status=Pending

Listing files to watch...
 - docker.atl-paas.net/atlassian/dx-ref-java-quarkus-im
Press Ctrl+C to exit
Watching for changes...
[microservice] INFO exec -a "java" java -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -cp "." -jar /deployments/quarkus-run.jar
[microservice] INFO running in /deployments
[microservice] __  ____  __  _____   ___  __ ____  ______
[microservice]  --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
[microservice]  -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
[microservice] --\___\_\____/_/ |_/_/|_/_/|_|\____/___/
[microservice] 2025-01-13 15:57:27,776 INFO  [io.und.websockets] (main) UT026003: Adding annotated server endpoint class com.atlassian.nebulae.micros.service.websocket.WebsocketResource for path /ws/{id}
[microservice] 2025-01-13 15:57:28,501 INFO  [io.quarkus] (main) dx-ref-java-quarkus-im 0.0.238-SNAPSHOT on JVM (powered by Quarkus 3.17.5) started in 1.056s. Listening on: http://0.0.0.0:7070
[microservice] 2025-01-13 15:57:28,501 INFO  [io.quarkus] (main) Profile local activated.
[microservice] 2025-01-13 15:57:28,501 INFO  [io.quarkus] (main) Installed features: [agroal, amazon-dynamodb, amazon-s3, amazon-sns, amazon-sqs, cdi, elasticsearch-rest-client, jdbc-postgresql, narayana-jta, redis-client, resteasy, resteasy-client, resteasy-jackson, smallrye-context-propagation, vertx, websockets, websockets-client]

Information

• Skaffold version: v2.13.2
• Operating system: Darwin 24.1.0
• Installed via: Homebrew
• Contents of skaffold.yaml:

apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: dx-ref-java-quarkus-im
build:
  artifacts:
    - image: <redacted>
      custom:
        buildCommand: |
          ./mvnw package -Dmaven.test.skip -T 1C -Pnebulae
          docker tag <redacted> $IMAGE
        dependencies:
          paths:
            - pom.xml
            - src/main
            - src/test
deploy:
  helm:
    releases:
      - name: dx-ref-java-quarkus-im
        chartPath: helm
        valuesFiles:
          - helm/values.yaml
        namespace: dx-ref-java-quarkus-im

## User defined port forwards, currently not working
portForward:
  - resourceType: pod
    resourceName:  dx-ref-java-quarkus-im-webserver
    namespace: dx-ref-java-quarkus-im
    port: 7070

Additional logs:

DEBU[0004] found open port: 7070                         subtask=-1 task=DevLoop
DEBU[0004] Running command: [kubectl --context kind-nebulae port-forward --pod-running-timeout 1s --namespace dx-ref-java-quarkus-im pod/dx-ref-java-quarkus-im-webserver 7070:7070]  subtask=pod/dx-ref-java-quarkus-im-webserver task=PortForward
DEBU[0004] port forwarding pod-dx-ref-java-quarkus-im-webserver-dx-ref-java-quarkus-im-7070 got terminated: exit status 1, output: Error from server (NotFound): pods "dx-ref-java-quarkus-im-webserver" not found  subtask=pod/dx-ref-java-quarkus-im-webserver task=PortForward
port forwarding pod-dx-ref-java-quarkus-im-webserver-dx-ref-java-quarkus-im-7070 got terminated: output: Error from server (NotFound): pods "dx-ref-java-quarkus-im-webserver" not found

I am surprised that the deployment is "stabilized" in 12ms before the pod is actually running. Note that the Pod is healthy in ~2s but port forward needs to be manually setup. The name is dx-ref-java-quarkus-im-webserver-6fb9db8544-6xx4s (note the hash) and it's not controlled by Deployment (but by a ReplicaSet which is controlled by ArgoRollout).

[kallangerard]

You're trying to target a pod named dx-ref-java-quarkus-im-webserver, but that's not the named of the pod. The name of the pod will be randomly generated by the replicaset, like dx-ref-java-quarkus-im-webserver-6fb9db8544-6xx4s.

You could try using a service instead for the port forward. Or see if kubectl port-forward accepts Argo Rollout resource types.

See https://kubernetes.io/docs/reference/kubectl/generated/kubectl_port-forward/

[mkonecny-atlassian]

@kallangerard the generated name was something that I thought would be an issue, but using the service doesn't work either:

port forwarding service-dx-ref-java-quarkus-im-service-canary-dx-ref-java-quarkus-im-80 got terminated: output: error: unable to forward port because pod is not running. Current status=Pending

port forwarding service-dx-ref-java-quarkus-im-service-dx-ref-java-quarkus-im-80 got terminated: output: error: unable to forward port because pod is not running. Current status=Pending

Despite the fact that the service exists. The setup in skaffold yaml is

portFroward:
  - resourceType: service
    resourceName: dx-ref-java-quarkus-im-service
    namespace: dx-ref-java-quarkus-im
    port: 80
    localPort: 8080

See:

kubectl get services -n dx-ref-java-quarkus-im
NAME                                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
dx-ref-java-quarkus-im-debug            ClusterIP   10.96.148.188   <none>        5005/TCP   2d
dx-ref-java-quarkus-im-service          ClusterIP   10.96.63.4      <none>        80/TCP     2m51s
dx-ref-java-quarkus-im-service-canary   ClusterIP   10.96.219.240   <none>        80/TCP     2m51s

Note that forwarding to a service WORKS after the first redeploy (if I rebuild the main docker image). At that point there is still an old Pod around and I suppose that is why the port forwarding to the service works:

Port forwarding service/dx-ref-java-quarkus-im-service in namespace dx-ref-java-quarkus-im, remote port 80 -> http://127.0.0.1:8080
Port forwarding service/dx-ref-java-quarkus-im-service-canary in namespace dx-ref-java-quarkus-im, remote port 80 -> http://127.0.0.1:4503

[kallangerard]

Is the pod actually running when you try that?

error: unable to forward port because pod is not running. Current status=Pending

[mkonecny-atlassian]

The pod turns to be running later, see the deployment logs in the issue summary. So it looks to me like the port forwarding to pod (or a service) is attempted once the "deployment stabilizes" (not sure what the condition is):

Waiting for deployments to stabilize...
Deployments stabilized in 12.8455ms

But the service logs are shown after this message. I am not sure why/how this works though because in k9s I see that the pod is first in the Pending state until the liveness and readiness probe pass (takes a few seconds) after which it turns to Running. However the port forwarding failure message is printed way ahead of that event.

Could it be that skaffold is looking for a Deployment object (which does not exist in my case, due to Argo Rollout supplementing that) and if it cannot find it, it just passes this condition as NO-OP? Fix for this would be to look for a ReplicaSet maybe.

[mkonecny-atlassian]

I am wondering why does the Deployment stabilized in 12.8455ms message occur, the pods do not yet exist as I can see when polling via k9s. Can you please point me to the piece of code responsible for this detection? I think this is the issue, it tries to continue to setup port forwarding too soon.

[mkonecny-atlassian]

Any suggestions here @kallangerard ? Thanks!

[mkonecny-atlassian]

Looks like there is a feature added by @ericzzzzzzz in 97c9c14#diff-ec89f93c5bc490d5558545dc8338f1ff94cf25bfc84469a6266a2ddae41e3f68R778 that allows CRDs to be ?monitored? for status, though this is not fitting my bill.

When I debug status_check.go I can see that the s.manifests passed to getCustomResources is always empty in my case.
I am not great with Go, but it looks like the check in statusCheck method is looking only at Deployments, StatefulSets and StandalonePods, and somewhat at the Custom Resources, but it's not able to validate that Argo Rollout is not yet ready.

I have also tried with

{
  "selectors": [
    {
      "group": "argoproj.io",
      "kind": "Rollout"
    }
  ]
}

and passing this file into skaffold dev --port-forward --status-check-selectors=status-check-selector.json but also then, the whole status checking just finished in few milliseconds while the pods were still starting.

Is there a mechanism to monitor these rollouts instead of Deployments? I can potentially even monitor "non-standalone" Pods with Skaffold Run-ID but that would mean making a change in the source and I doubt this type of contribution is what will be accepted in main.

[mkonecny-atlassian]

I can confirm that if I make change in pod.go:

func (s *standalonePodsSelector) Select(ctx context.Context, ns string, opts metav1.ListOptions) ([]v1.Pod, error) {
	pods, err := s.k.CoreV1().Pods(ns).List(ctx, opts)
	if err != nil {
		return nil, err
	}
	var filtered []v1.Pod
	for _, po := range pods.Items {
		// deployments defining pods directly don't have owner references
		// if metav1.GetControllerOfNoCopy(&po) == nil {
		filtered = append(filtered, po)
		// }
	}
	return filtered, nil
}

and include all pods (not just standalone pods) then the detection works fine:

Release "dx-ref-java-quarkus-im" has been upgraded. Happy Helming!
NAME: dx-ref-java-quarkus-im
LAST DEPLOYED: Wed Feb 12 12:18:41 2025
NAMESPACE: dx-ref-java-quarkus-im
STATUS: deployed
REVISION: 6
TEST SUITE: None
Waiting for deployments to stabilize...
 - pods: 0/1 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "dx-ref-java-quarkus-im-webserver-748f75c597-jb9w8-service-volume". preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
    - dx-ref-java-quarkus-im:pod/dx-ref-java-quarkus-im-webserver-748f75c597-jb9w8: 0/1 nodes are available: waiting for ephemeral volume controller to create the persistentvolumeclaim "dx-ref-java-quarkus-im-webserver-748f75c597-jb9w8-service-volume". preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
 - pods is ready.
Deployments stabilized in 10.164 seconds
Port forwarding service/dx-ref-java-quarkus-im-service in namespace dx-ref-java-quarkus-im, remote port 80 -> http://127.0.0.1:8083
Port forwarding service/dx-ref-java-quarkus-im-debug in namespace dx-ref-java-quarkus-im, remote port 5005 -> http://127.0.0.1:5008
Port forwarding service/dx-ref-java-quarkus-im-service-canary in namespace dx-ref-java-quarkus-im, remote port 80 -> http://127.0.0.1:4504
Watching for changes...

I am happy to create a PR for this, but I am not clear why was the StandalonePod option even chosen.

[mkonecny-atlassian]

@gsquared94 you have added the filter for standalone pods in 14314b4#diff-3f3b4b2f9bde6d865f17be301760aa1f5ec701f66ea7091a3513c2e58f5c2b6f

Any reason only standalone pods are monitored and not any pods? For some reason the pods created via Argo Rollouts are not monitored and a quick fix is removing the filter above...