Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Authentication issue on Azure devops with the git artifact example #11705

Open
MarcoFabbri-orienteed opened this issue Aug 29, 2023 · 8 comments · May be fixed by #13875
Open

Authentication issue on Azure devops with the git artifact example #11705

MarcoFabbri-orienteed opened this issue Aug 29, 2023 · 8 comments · May be fixed by #13875
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc P3 Low priority solution/suggested A solution to the bug has been suggested. Someone needs to implement it. type/bug

Comments

@MarcoFabbri-orienteed
Copy link

MarcoFabbri-orienteed commented Aug 29, 2023 •
edited
Loading

Hi all,
the issue 10600 is closed but I still have an authentication issue on Azure devops with the git artifact example

https://github.com/argoproj/argo-workflows/blob/master/examples/input-artifact-git.yaml

artifact argo-source failed to load: failed to clone "https://<org>@dev.azure.com/<org>/folder/_git/customerdata": unexpected client error: unexpected requesting "https://<org>@dev.azure.com/<org>/folder/_git/customerdata/git-upload-pack" status code: 400

This error is thrown using Argo Workflow 3.4.10: any idea?

Thanks

Originally posted by @MarcoFabbri-orienteed in #10600 (comment)
@MarcoFabbri-orienteed MarcoFabbri-orienteed changed the title Azure devops git repo unable to use git artifact Authentication issue on Azure devops with the git artifact example Aug 29, 2023
@stale

This comment was marked as resolved.

Sign in to view
@stale stale bot added the problem/stale This has not had a response in some time label Sep 17, 2023
@terrytangyuan terrytangyuan removed the problem/stale This has not had a response in some time label Sep 20, 2023
@agilgur5 agilgur5 added type/bug area/artifacts S3/GCP/OSS/Git/HDFS etc P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important P3 Low priority and removed P2 Important. All bugs with >=3 thumbs up that aren’t P0 or P1, plus: Any other bugs deemed important labels Sep 26, 2023
@mfabbri
Copy link

mfabbri commented Nov 1, 2023

Hi @agilgur5 thank you for marking this as a bug. Do you have news on this issue?

@agilgur5
Copy link
Contributor

agilgur5 commented Nov 1, 2023 •
edited
Loading

No. Any updates would be in the issue. Please see https://sindresorhus.com/blog/issue-bumping.

You also did not fill out the bug report template, which does not help.

This error is thrown using Argo Workflow 3.4.10: any idea?

The fix to #10600, #10953, was not released until 3.5.0-rc1. So I would suggest trying with 3.5.0

@agilgur5 agilgur5 added the problem/more information needed Not enough information has been provide to diagnose this issue. label Nov 1, 2023
@mfabbri
Copy link

mfabbri commented Nov 2, 2023 •
edited by agilgur5
Loading

you are right @agilgur5 , let me provide more information

Pre-requisites

  • I have double-checked my configuration
  • I can confirm the issues exists when I tested with :3.5.0
  • I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?
To connect to an Azure devops repository and clone the content as artifacts. Following this generic example
https://github.com/argoproj/argo-workflows/blob/master/examples/input-artifact-git.yaml

Version

Version{Version:v3.5.0,BuildDate:2023-10-13T13:06:32Z,GitCommit:bf735a2e861d6b1c686dd4a076afc3468aa89c4a,GitTag:v3.5.0,GitTreeState:clean,GoVersion:go1.21.3,Compiler:gc,Platform:linux/amd64

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-artifact-git-
spec:
  entrypoint: git-clone
  templates:
  - name: git-clone
    inputs:
      artifacts:
      - name: argo-source
        path: /src
        git:
          repo: https://dev.azure.com/<projectFolder>/_git/<repoName>
          branch: main
          usernameSecret:
            name: azuredevops-git-secret
            key: username
          passwordSecret:
            name: azuredevops-git-secret
            key: password
    container:
      image: golang:1.10
      command: [sh, -c]
      args: ["git status && ls && cat VERSION"]
      workingDir: /src

The secret was created with

apiVersion: v1
data:
  password: <password>
  username: YXJnbw==
kind: Secret
metadata:
  name: azuredevops-git-secret
  namespace: argo
type: Opaque

Logs from the workflow controller

time="2023-11-02T10:00:33.013Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=warning msg="Non-transient error: configmaps \"artifact-repositories\" not found"
time="2023-11-02T10:00:33.033Z" level=info msg="resolved artifact repository" artifactRepositoryRef=default-artifact-repository
time="2023-11-02T10:00:33.033Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=info msg="Pod node input-artifact-git-rrch5-hmhhj initialized Pending" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.033Z" level=error msg="was unable to obtain node for " namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg="Created pod: input-artifact-git-rrch5-hmhhj (input-artifact-git-rrch5-hmhhj)" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg=reconcileAgentPod namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:33.129Z" level=info msg="Workflow to be dehydrated" Workflow Size=2054
time="2023-11-02T10:00:33.140Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=148216596 workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="Processing workflow" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=input-artifact-git-rrch5-hmhhj        
time="2023-11-02T10:00:43.131Z" level=info msg="Pod failed: Error (exit code 1): artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400" displayName=input-artifact-git-rrch5-hmhhj namespace=argo pod=input-artifact-git-rrch5-hmhhj templateName=git-clone workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="marking node as failed since init container has non-zero exit code" namespace=argo new.phase=Failed workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.131Z" level=info msg="node changed" namespace=argo new.message="Error (exit code 1): artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400" new.phase=Failed new.progress=0/1 nodeID=input-artifact-git-rrch5-hmhhj old.message= old.phase=Pending old.progress=0/1 workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg=reconcileAgentPod namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Updated phase Running -> Failed" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Updated message  -> Error (exit code 1): artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>\/git-upload-pack\" status code: 400" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Marking workflow completed" namespace=argo workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.132Z" level=info msg="Workflow to be dehydrated" Workflow Size=3085
time="2023-11-02T10:00:43.137Z" level=info msg="cleaning up pod" action=deletePod key=argo/input-artifact-git-rrch5-hmhhj-1340600742-agent/deletePod
time="2023-11-02T10:00:43.142Z" level=info msg="Workflow update successful" namespace=argo phase=Failed resourceVersion=148216674 workflow=input-artifact-git-rrch5-hmhhj
time="2023-11-02T10:00:43.143Z" level=info msg="Queueing Failed workflow argo/input-artifact-git-rrch5-hmhhj for delete in 168h0m0s due to TTL"     
time="2023-11-02T10:00:43.170Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/input-artifact-git-rrch5-hmhhj/labelPodCompleted 
time="2023-11-02T10:02:09.213Z" level=info msg="Alloc=47344 TotalAlloc=31311568 Sys=127816 NumGC=1407 Goroutines=175"

Logs from in your workflow's init container

time="2023-11-02T10:00:37.183Z" level=info msg="Starting Workflow Executor" version=v3.5.0
time="2023-11-02T10:00:37.186Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2023-11-02T10:00:37.186Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=input-artifact-git-rrch5-hmhhj template="{\"name\":\"git-clone\",\"inputs\":{\"artifacts\":[{\"name\":\"argo-source\",\"path\":\"/src\",\"git\":{\"repo\":\"https://dev.azure.com/*/_git/<repoName>\",\"usernameSecret\":{\"name\":\"azuredevops-git-secret\",\"key\":\"username\"},\"passwordSecret\":{\"name\":\"azuredevops-git-secret\",\"key\":\"password\"},\"branch\":\"main\"}}]},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"golang:1.10\",\"command\":[\"sh\",\"-c\"],\"args\":[\"git status \\u0026\\u0026 ls \\u0026\\u0026 cat VERSION\"],\"workingDir\":\"/src\",\"resources\":{}}}" version="&Version{Version:v3.5.0,BuildDate:2023-10-13T13:06:32Z,GitCommit:bf735a2e861d6b1c686dd4a076afc3468aa89c4a,GitTag:v3.5.0,GitTreeState:clean,GoVersion:go1.21.3,Compiler:gc,Platform:linux/amd64,}"
time="2023-11-02T10:00:37.241Z" level=info msg="Start loading input artifacts..."
time="2023-11-02T10:00:37.241Z" level=info msg="Downloading artifact: argo-source"
time="2023-11-02T10:00:37.339Z" level=info msg="Load artifact" artifactName=argo-source duration=97.805551ms error="failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400" key=
time="2023-11-02T10:00:37.339Z" level=error msg="executor error: artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400"
time="2023-11-02T10:00:37.340Z" level=info msg="Alloc=10152 TotalAlloc=15800 Sys=23909 NumGC=4 Goroutines=5"
time="2023-11-02T10:00:37.340Z" level=fatal msg="artifact argo-source failed to load: failed to clone \"https://dev.azure.com/*/_git/<repoName>\": unexpected client error: unexpected requesting \"https://dev.azure.com/*/_git/<repoName>/git-upload-pack\" status code: 400"

I hope this is enough to understand the issue.

Thanks

@mfabbri mfabbri mentioned this issue Nov 2, 2023
Closed
3 tasks
@agilgur5
Copy link
Contributor

agilgur5 commented Nov 2, 2023 •
edited
Loading

Thanks for the update @mfabbri. Could you please add code blocks and syntax highlighting (i.e. ```yaml) to your comment? It looks like it wasn't formatted correctly.

So you're still having this issue in 3.5 then, correct?
@rohankmr414 would you be able to take a look at this? I think some of the go-git changes were still in-progress when you last worked on this issue (I know #11149 never quite got completed, but don't know the full context)

@mfabbri
Copy link

mfabbri commented Nov 3, 2023

You're right @agilgur5 : I didn't remember the syntax. Now it's fixed. Please @rohankmr414 , ask me any additional details you need about the test I performed. Thank you both

@agilgur5
Copy link
Contributor

agilgur5 commented Nov 5, 2023

I think I might have found the issue. #10953 merely bumped go-git, but it actually requires some code on our side to add to workaround the Azure issue. go-git/go-git#613 did not entirely fix go-git/go-git#64 -- it provided a workaround, and that workaround requires some manual code. There's an example in the go-git PR that requires implementing; one implementation of it can be seen in kluctl/kluctl#637

That should be fairly simple to add. @mfabbri would you be interested in contributing that? If not, I can probably take this on

@agilgur5 agilgur5 mentioned this issue Nov 5, 2023
Merged
@agilgur5 agilgur5 removed the problem/more information needed Not enough information has been provide to diagnose this issue. label Nov 5, 2023
@mfabbri
Copy link

mfabbri commented Nov 5, 2023

Thank you for the proposal @agilgur5 but I don't think I have enough knowledge to proceed with this implementation. Can you take it?

@agilgur5 agilgur5 self-assigned this Nov 5, 2023
@agilgur5 agilgur5 added the solution/suggested A solution to the bug has been suggested. Someone needs to implement it. label Jan 7, 2024
clkamp pushed a commit to clkamp/argo-workflows that referenced this issue Jan 31, 2024
fix: Fixes git over azure devops Fixes argoproj#11705
6827e88 
Signed-off-by: Christian Lütke Stetzkamp <[email protected]>
@clkamp clkamp mentioned this issue Jan 31, 2024
Closed
clkamp pushed a commit to clkamp/argo-workflows that referenced this issue Jan 31, 2024
fix: Fixes git over azure devops Fixes argoproj#11705
cf3f23a 
Signed-off-by: Christian Lütke Stetzkamp <[email protected]>
@agilgur5 agilgur5 mentioned this issue Feb 1, 2024
Closed
7 tasks
clkamp pushed a commit to clkamp/argo-workflows that referenced this issue Feb 12, 2024
fix: Fixes git over azure devops Fixes argoproj#11705
6d0a832 
Signed-off-by: Christian Lütke Stetzkamp <[email protected]>
clkamp pushed a commit to clkamp/argo-workflows that referenced this issue Mar 25, 2024
fix: Fixes git over azure devops Fixes argoproj#11705
ba1fdaa 
Signed-off-by: Christian Lütke Stetzkamp <[email protected]>
@tooptoop4 tooptoop4 linked a pull request Nov 6, 2024 that will close this issue
Open
@agilgur5 agilgur5 removed their assignment Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/artifacts S3/GCP/OSS/Git/HDFS etc P3 Low priority solution/suggested A solution to the bug has been suggested. Someone needs to implement it. type/bug
Projects
None yet
Milestone
No milestone
4 participants
@mfabbri @terrytangyuan @agilgur5 @MarcoFabbri-orienteed