[GitLab] Token provided by GCM for gitlab.com does not work for Source Link request.

[chuckries]

GCM for GitLab uses the repository_write scope for its tokens. This provides read/write access for Git over HTTP operations but does not provide repository read operations via the GitLab API. This means that GCM does not provide tokens that can be used to authenticate Source Link requests.

This isn't a bug, per se, as GCM is using the correct scope for pure git.exe operations. This is its intended purpose. It is, however, a missed opportunity. Visual Studio leverages GCM to authenticate Source Link requests, and expanding the scope used could allow Visual Studio to use Source Link for private gitlab.com repos, which has been a gap for several releases.

A more in depth write up can be found here: dotnet/sourcelink#281 (comment)

I am hoping somebody from GCM or GitLab can speak to this. My testing of GCM token's against GitLab API requests makes me believe that the scope is the issue, but I'm not 100% sure.

Thanks!

[chuckries]

I was able to confirm that adding repository_read scope to the tokens provided by GCM allows the token to be used to authenticate the repository files api, which would make the tokens usable by Source Link.

[ldennington]

Mentioning @hickford for comment.

[hickford]

Do Source Link users know they're using Git indirectly? As a Git user, a request from GCM for ever-so-slightly broader than necessary permissions wouldn't bother me. Source Link users however might be confused to read "an application called Git Credential Manager is requesting access to your account" when using Source Link.

@chuckries Can you work around the issue by creating a 'Source Link' application with broader scope at https://gitlab.com/-/profile/applications and configuring GCM following https://github.com/GitCredentialManager/git-credential-manager/blob/main/docs/gitlab.md ? It might also be necessary to expand GitLabOAuthScopes according to a new config variable.

---

https://docs.gitlab.com/ee/integration/oauth_provider.html#authorized-applications

Scope	Description
read_repository	Grants read-only access to repositories on private projects using Git-over-HTTP or the Repository Files API.
write_repository	Grants read-write access to repositories on private projects using Git-over-HTTP (not using the API).

[Bene81]

According to my researches regarding using sourcelink to gitlab from VS debugger following issues persist:

• VS Debugger requires plain text/source files, thus the raw file URL
• repository files API returns JSON, in my case with base64 encoded content, see Repository Files API
• repository files API refers to commits using the ref query parameter
• PDB sourcelink is refering to repository root, as seen here or any resulting PDB, the relative file paths are then appended.
  • I don't see adding query parameters will do the trick
  • as far as I read somewhere VS debugger has some issues with query parameters (correct me please, if I am wrong)
• repository files API requires a PRIVATE_TOKEN or ACCESS_TOKEN. A BEARER Token as provided by the OAuth API is not sufficient.
  • at least not according to my API tests -> results in 401 Unauthorized

At this point of my research I think what needs to be done to allow sourcelink working with gitlab would be to allow OAuth application an access right to read_raw_files.
This access right would still need to be invented and implemented over at gitlab.org.

Best way to get sourcelink debugging nowadays seems to be the workarounds with

• GitLabProxy
• Authentication from within VS integrated web browser

[ldennington]

We're going to plan to add the read_repository scope to tokens, as it's a subset of permissions granted by write_repository anyway. Hopefully this will unblock the described Source Link scenarios.

[chuckries]

@ldennington thanks for the info. from what I recall from looking at this, the issue for source link is that the write_reposository scope for OAuth tokens provides read-write access only for Git-over-HTTP (essentially git.exe operations) while read_respository scope provides read access for both Git-over-HTTP as well as the Repository Files API. This is documented here.

The read access to the Repository Files API is what would be needed for Source Link. If you are saying that this line will be amended to include both scopes in tokens obtained through GCM, then I believe this should work for source link.

[Bene81]

@ldennington @chuckries
I do not see that extending the scope to contain read_repository also will do the the trick for the following reasons:

• I tried it on our companies network using sourcelink from visual studio 2019
• I tried accessing the Repository Files API using PostMan, it returns the source file base64 encoded in JSON, not a plain text file, see result file sample below:
• PDBs generated with sourcelink contain repository links to raw files of commit. Relative paths are appended, depending on the symbols debugged
  • see PDB extract below
  • e.g. http://gitlab/ABC/SymbolPackage/-/raw/e4dd6ade0d14b620d8e30eff993909049ad8bfae/src/Project/Dummy.cs
• sourcelink needs to map this to the Repository Files API calls containing query parameters,
  • e.g. GET https://gitlab/api/v4/projects/13083/repository/files/app%2Fmodels%2Fkey%2Erb?ref=e4dd6ade0d14b620d8e30eff993909049ad8bfae
  • in general GET /projects/:id/repository/files/:file_path
  • project ID :id is managed by gitlab, I don't see, how sourcelink will obtain this, especially if origin url is using project name like SymbolPackage see example below.

I admit, the blackbox for me is sourcelink itself. I do not now, what excactly it does.

• if it contains a special gitlab handler, that is transcoding the raw file urls into Repository API URLs
• if it parses API result files, extracting embedded content

it will work. Otherwise support has to come from gitlab offering access to the raw file url via OAUTH token.

What I can say for sure is, it did not work in my tests, when I granted myself also the read_repository rights.

Sample Repository Files API result:

{
  "file_name": "Dummy.cs",
  "file_path": "src/Project//Dummy.cs",
  "size": 1234,
  "encoding": "base64",
  "content": "IyA9PSBTY2hlbWEgSW5mb3...",
  "content_sha256": "4c294617b60715c1d218e61164a3abd4808a4284cbc30e6728a01ad9aada4481",
  "ref": "main",
  "blob_id": "79f7bbd25901e8334750839545a9bd021f0e4c83",
  "commit_id": "d5a3ff139356ce33e37e73add446f16869741b50",
  "last_commit_id": "e4dd6ade0d14b620d8e30eff993909049ad8bfae",
  "execute_filemode": false
}

Source Link Prefix Mapping in PDB:

{
    "documents":
    {
          "D:\\Project\\SymbolPackage\\*":"http://gitlab/ABC/SymbolPackage/-/raw/e4dd6ade0d14b620d8e30eff993909049ad8bfae/*"
    }
}

[chuckries]

@Bene81 you are correct that adjusting the scope is not the only fix required for Source Link to work with private gitlab.com repos, but it is a necessary piece. It is also the piece that I can't change; I can make the necessary changes in VS Source Link.

1. A while back I cloned the GCM repo and changed the necessary application id's locally which allowed me to register it as my own application with my own GitLab account. This in turn allowed me to adjust the scope of the tokens, which allowed me to test the addition of read_repository
2. With this token, I tested the Repository Files API for getting a raw file. This I believe returns a flat text file that Source Link expects, not a json blob.

Once the correct authentication is available, I intend to:

1. Get the dotnet/sourcelink GitLab package to change to api based URL's as opposed to the current web based URL's.
2. Add a mechanism in VS to map the old URL's into the API url's, as you suggest. According to the docs, the "id" field of the url can be a string or an id, which should allow us to use the pieces from the existing URL.

[Bene81]

@chuckries
Sounds good. Altogether it really might work.
I recall though that I had issues with just the project names in the API calls. But maybe I tricked myself there. I will need to retry this at some point. I'll keep you posted then.
I am really looking forward to sourcelink being able to deal with gitlab.

[ldennington]

@chuckries I'm unable to mention you in the PR for some reason, but wanted to give you a heads up that it is open!