repos require https:// prefixed

[jnaulty]

I was testing out the latest changes and think the drop-down selection for adding + changing repos is really cool.

I noticed a small issue with the user-interface and bubbling up failures.
I tried to use the repo github.com/diem/diem but it failed somewhat silently after hitting the 'refresh'

Here are the kubectl logs deployment/backend --follow after hitting refresh

Feb 19 18:01:02.499  INFO rocket::server: GET /refresh?repo=github.com/diem/diem application/json:    
Feb 19 18:01:02.500  INFO _: Matched: GET /refresh?<repo> (refresh)    
Feb 19 18:01:02.505  INFO _: Outcome: Success    
Feb 19 18:01:02.505  INFO metrics::analysis: getting github.com/diem/diem repo
Feb 19 18:01:02.506  INFO metrics::analysis: cloning github.com/diem/diem into /app/metrics/repos/d7f2b20002483a071f6202e94c88c123
Feb 19 18:01:02.506  INFO _: Response succeeded.    
Feb 19 18:01:02.512  INFO metrics::analysis: pulling latest changes
Feb 19 18:01:02.516 ERROR metrics: metrics failed to terminate: No such file or directory (os error 2)
Feb 19 18:01:41.256  INFO rocket::server: GET /dependencies?repo=github.com/diem/diem application/json:    
Feb 19 18:01:41.256  INFO rocket::server: GET /repos application/json:    
Feb 19 18:01:41.256  INFO _: Matched: GET /dependencies?<repo> (dependencies)    
Feb 19 18:01:41.256  INFO _: Matched: GET /repos (repos)    
Feb 19 18:01:41.258  INFO _: Outcome: Success    
Feb 19 18:01:41.258  INFO _: Response succeeded.    
Feb 19 18:01:41.260  INFO _: Outcome: Success    
Feb 19 18:01:41.260  INFO _: Response succeeded.   

When I used the full url (https://github.com/diem/diem), things worked as expected
Good logs:

Feb 19 18:06:16.523  INFO rocket::server: GET /refresh?repo=https://github.com/diem/diem application/json:    
Feb 19 18:06:16.523  INFO _: Matched: GET /refresh?<repo> (refresh)    
Feb 19 18:06:16.530  INFO metrics::analysis: getting https://github.com/diem/diem repo
Feb 19 18:06:16.530  INFO metrics::analysis: cloning https://github.com/diem/diem into /app/metrics/repos/5160e7f3745a28604dbaba920a202a2d
Feb 19 18:06:16.530  INFO _: Outcome: Success    
Feb 19 18:06:16.531  INFO _: Response succeeded.    
Feb 19 18:06:25.572  INFO metrics::analysis: pulling latest changes
Feb 19 18:06:26.528  INFO metrics::analysis: current commit: 509c430af9fbc8091894674c11504e55b5969d14

Feb 19 18:06:26.676  INFO metrics::rust: 1. fetching dependencies...
Feb 19 18:06:26.676  INFO metrics::rust: parsing Cargo.toml with guppy...
Feb 19 18:06:26.676  INFO metrics::rust::guppy: obtaining dependencies from "/app/metrics/repos/5160e7f3745a28604dbaba920a202a2d/Cargo.toml"
Feb 19 18:06:38.234  INFO metrics::rust::guppy: guppy cargo settings: CargoOptions {
    version: V1,
    include_dev: false,
    initials_platform: Standard,
    host_platform: None,
    target_platform: None,
    omitted_packages: {},
}
Feb 19 18:06:41.772  INFO metrics::rust::guppy: guppy cargo settings: CargoOption

[mimoo]

So I looked into sanitizing the repo URL earlier, and ended up in a rabbit hole about how git itself validates git repo URLs. It's apparently very lax, and you can have all sorts of URLs in there which make sense if you're doing something locally. For example, repo would work (and probably should work because you might be hosting your stuff in a weird way :D)

[mimoo]

(also, ssh:// should work as well)

[AnomalRoil]

Then we might want to change the way we detect diem too:

whackadep/web-backend/metrics/src/analysis.rs

Line 98 in 3ed09ae

let is_diem = repo_url == "https://github.com/diem/diem.git";

[mimoo]

yeah we could detect the SSH url as well (although that wouldn't work atm)

btw this is unrelated to the original issue so I'll close this : o

[AnomalRoil]

The initial issue still exists, it got me yesterday

[mimoo]

we can look more into git URL sanitization, but from my glance it's going to be very simple and will allow something like github.com/diem/diem (not sure though)

[AnomalRoil]

Yup, as long as we do not "blindly" accept an url that won't work afterwards, such as we currently do with github.com/diem/diem: it gets listed but we cannot run analysis on it.

[mimoo]

Yeah good point! Maybe git has a command that checks if it can clone a repo?

git ls-remote https://github.com/git/git

seems to work, the problem is that it'll sometimes ask for user + password... cc @jnaulty we should make sure that'll play nice with the GITHUB_TOKEN approach to cloning private repos.