-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implemented Artifactory upload and download handling #30
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There appears to be quite a bit of functionality and many changes that are not directly related to the Artifactory / caching functionality, which makes this PR a bit hard to read and review.
Also, can we get the writeup of what we decided would be the first cut of this functionality? Without that, @rabadin can't properly evaluate this code.
I assume that you notice that the CI tests are failing, too. You'll want to make test
before you push again and make sure everything passes.
Finally, let's talk in the next meeting about how to add some unit tests for this functionality.
Note that this PR now has a (simple) conflict that has to be fixed. Github won't run the CI on a PR that has conflicts, which is why there's currently no CI results here. |
082c98b
to
c389fb2
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Need to remove the a640.... binary.
FWIW: Using git add -a
or git commit -a
is dangerous. It is much safer to git add ...list of files...
or git add -p
to select precisely which things you want to add to the staging area.
internal/lock.go
Outdated
} | ||
defer file.Close() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks like it yes. A small one since this would be GCed but still.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks generally okay... but I really don't understand why you're adding so much changes that are completely unrelated to the task at hand and that, in most cases, are not real improvements. See my comments inline.
Also: before this change the coverage is 81.8& and after it is 80.4%... could you improve the coverage of the newly-added code to ensure that the coverage doesn't go down with this change?
It looks like there were some bad merges that were not fixed properly before committing. For example, I see a full Open MPI tarball added in 4033f12. Word to the wise: you should avoid using Additionally, |
internal/lock.go
Outdated
return err | ||
} | ||
tmpPath := tmpFile.Name() | ||
defer os.Remove(tmpPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're doing a remove... and on line 270 you're doing a rename... why?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have resolved this without giving me a good explanation... re-opening...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was to trying to make sure the temporary file is deleted if something goes wrong. If everything works fine, the file get renamed and moved to where it needs to be.
Is there a better way to do this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes: just remove all this code and re-use the code that exists; you only need to add support for a bearer token:
diff --git a/internal/resource.go b/internal/resource.go
index a377e87..aab11b0 100644
--- a/internal/resource.go
+++ b/internal/resource.go
@@ -45,17 +45,20 @@ func NewResourceFromUrl(urls []string, algo string, tags []string, filename stri
}
// getUrl downloads the given resource and returns the path to it.
-func getUrl(u string, fileName string, ctx context.Context) (string, error) {
+func getUrl(u string, fileName string, bearer string, ctx context.Context) (string, error) {
_, err := url.Parse(u)
if err != nil {
return "", fmt.Errorf("invalid url '%s': %s", u, err)
}
log.Debug().Str("URL", u).Msg("Downloading")
- err = requests.
+ req := requests.
URL(u).
Header("Accept", "*/*").
- ToFile(fileName).
- Fetch(ctx)
+ ToFile(fileName)
+ if bearer != "" {
+ req.Header("Authorization", fmt.Sprintf("Bearer %s", bearer))
+ }
+ err = req.Fetch(ctx)
if err != nil {
return "", fmt.Errorf("failed to download '%s': %s", u, err)
}
@@ -64,12 +67,12 @@ func getUrl(u string, fileName string, ctx context.Context) (string, error) {
}
// GetUrlToDir downloads the given resource to the given directory and returns the path to it.
-func GetUrlToDir(u string, targetDir string, ctx context.Context) (string, error) {
+func GetUrlToDir(u string, targetDir string, bearer string, ctx context.Context) (string, error) {
// create temporary name in the target directory.
h := sha256.New()
h.Write([]byte(u))
fileName := filepath.Join(targetDir, fmt.Sprintf(".%s", hex.EncodeToString(h.Sum(nil))))
- return getUrl(u, fileName, ctx)
+ return getUrl(u, fileName, bearer, ctx)
}
// GetUrlWithDir downloads the given resource to a temporary file and returns the path to it.
@@ -79,7 +82,7 @@ func GetUrltoTempFile(u string, ctx context.Context) (string, error) {
log.Fatal().Err(err)
}
fileName := file.Name()
- return getUrl(u, fileName, ctx)
+ return getUrl(u, fileName, "", ctx)
}
func (l *Resource) Download(dir string, mode os.FileMode, ctx context.Context) error {
@@ -92,7 +95,7 @@ func (l *Resource) Download(dir string, mode os.FileMode, ctx context.Context) e
for _, u := range l.Urls {
// Download file in the target directory so that the call to
// os.Rename is atomic.
- lpath, err := GetUrlToDir(u, dir, ctx)
+ lpath, err := GetUrlToDir(u, dir, "", ctx)
if err != nil {
downloadError = err
continue
@amdurh05 Also, you have a random 11M binary file in here. |
@eeasaa01 Btw, you have a conflict that you need to fix by rebasing your work onto the latest main. |
@eeasaa01 I suggested doing a rebase. You did a merge. That's going to potentially complicate the process when you squash all your changes into just one commit. |
I did the merge through the GitHub website, as it was the only option I saw there—apologies for that. Also, I now see the tests failing in two files: lock_test and download_test. What do you recommend to fix these?
…________________________________
From: Raphaël ***@***.***>
Sent: Monday, December 2, 2024 2:58 PM
To: cisco-open/grabit ***@***.***>
Cc: Assaid, Amin ***@***.***>; Mention ***@***.***>
Subject: Re: [cisco-open/grabit] Implemented Artifactory upload and download handling (PR #30)
CAUTION: This email originated from outside of our organization. Do not click links, open attachments, or respond unless you recognize the sender's email address and know the contents are safe.
@eeasaa01<https://github.com/eeasaa01> I suggested doing a rebase. You did a merge. That's going to potentially complicate the process when you squash all your changes into just one commit.
—
Reply to this email directly, view it on GitHub<#30 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BDAAXH4UKL5I25KLEMMLA2T2DS3VZAVCNFSM6AAAAABRGFOQCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJSGY2DSMRZGA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
In the future: although GitHub does have the option for Rebasing (instead of Merging), you might be better served doing the operation on the command line or in your local IDE -- it might be slightly less confusing as to what is actually happening. As we talked about in our status meetings, merging the target branch into your PR branch frequently leads to pain in the long run. It should be avoided whenever possible. Rebase will generally spare you that pain.
Your PR removed the 5th argument from
|
672863d
to
c7aaa93
Compare
@eeasaa01 and I are DM'ing in Teams to try to clean this up. I just took a first step here, and did the following:
We still have a problematic merge in the branch on this commit. The merge from eeasaa01#1 contains a problematic merge from main as well, which is still causing more conflicts. Still need to resolve this... |
3520e1d
to
971b190
Compare
.github/workflows/ci.yml
Outdated
@@ -19,6 +19,8 @@ jobs: | |||
go-version-file: 'go.mod' | |||
|
|||
- name: Test | |||
env: | |||
GRABIT_ARTIFACTORY_TOKEN: team2grabit |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that's a good idea. You want to control this in code via https://pkg.go.dev/testing#B.Setenv. This way you can also test what happens when it's not set.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought this way it may work, but okay, I will use your suggestion right away.
9566766
to
2123068
Compare
internal/lock.go
Outdated
|
||
err := deleteCache(artifactoryURL, token) | ||
if err != nil { | ||
log.Warn().Msg("Warning: Unable to delete from Artifcatory") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typo: Artifcatory
internal/lock.go
Outdated
@@ -89,7 +176,7 @@ func strToFileMode(perm string) (os.FileMode, error) { | |||
|
|||
// Download gets all the resources in this lock file and moves them to | |||
// the destination directory. | |||
func (l *Lock) Download(dir string, tags []string, notags []string, perm string, status bool) error { | |||
func (l *Lock) Download(dir string, tags []string, notags []string, perm string, someBool bool) error { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's this change, why the name change?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn’t really change it. When I was testing, the function needed five arguments, so I added a temporary one called someBool. Later, when I added Dylan’s code, I saw it should be status, so I fixed it.
internal/lock.go
Outdated
return err | ||
} | ||
tmpPath := tmpFile.Name() | ||
defer os.Remove(tmpPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have resolved this without giving me a good explanation... re-opening...
internal/lock.go
Outdated
return err | ||
} | ||
tmpPath := tmpFile.Name() | ||
defer os.Remove(tmpPath) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes: just remove all this code and re-use the code that exists; you only need to add support for a bearer token:
diff --git a/internal/resource.go b/internal/resource.go
index a377e87..aab11b0 100644
--- a/internal/resource.go
+++ b/internal/resource.go
@@ -45,17 +45,20 @@ func NewResourceFromUrl(urls []string, algo string, tags []string, filename stri
}
// getUrl downloads the given resource and returns the path to it.
-func getUrl(u string, fileName string, ctx context.Context) (string, error) {
+func getUrl(u string, fileName string, bearer string, ctx context.Context) (string, error) {
_, err := url.Parse(u)
if err != nil {
return "", fmt.Errorf("invalid url '%s': %s", u, err)
}
log.Debug().Str("URL", u).Msg("Downloading")
- err = requests.
+ req := requests.
URL(u).
Header("Accept", "*/*").
- ToFile(fileName).
- Fetch(ctx)
+ ToFile(fileName)
+ if bearer != "" {
+ req.Header("Authorization", fmt.Sprintf("Bearer %s", bearer))
+ }
+ err = req.Fetch(ctx)
if err != nil {
return "", fmt.Errorf("failed to download '%s': %s", u, err)
}
@@ -64,12 +67,12 @@ func getUrl(u string, fileName string, ctx context.Context) (string, error) {
}
// GetUrlToDir downloads the given resource to the given directory and returns the path to it.
-func GetUrlToDir(u string, targetDir string, ctx context.Context) (string, error) {
+func GetUrlToDir(u string, targetDir string, bearer string, ctx context.Context) (string, error) {
// create temporary name in the target directory.
h := sha256.New()
h.Write([]byte(u))
fileName := filepath.Join(targetDir, fmt.Sprintf(".%s", hex.EncodeToString(h.Sum(nil))))
- return getUrl(u, fileName, ctx)
+ return getUrl(u, fileName, bearer, ctx)
}
// GetUrlWithDir downloads the given resource to a temporary file and returns the path to it.
@@ -79,7 +82,7 @@ func GetUrltoTempFile(u string, ctx context.Context) (string, error) {
log.Fatal().Err(err)
}
fileName := file.Name()
- return getUrl(u, fileName, ctx)
+ return getUrl(u, fileName, "", ctx)
}
func (l *Resource) Download(dir string, mode os.FileMode, ctx context.Context) error {
@@ -92,7 +95,7 @@ func (l *Resource) Download(dir string, mode os.FileMode, ctx context.Context) e
for _, u := range l.Urls {
// Download file in the target directory so that the call to
// os.Rename is atomic.
- lpath, err := GetUrlToDir(u, dir, ctx)
+ lpath, err := GetUrlToDir(u, dir, "", ctx)
if err != nil {
downloadError = err
continue
internal/artifactory_cache_test.go
Outdated
// TestDownloadWithArtifactoryCache verifies downloading resources with caching enabled. | ||
func TestDownloadWithArtifactoryCache(t *testing.T) { | ||
// Sub-test to verify behavior when NO_CACHE_UPLOAD is set. | ||
t.Run("NO_CACHE_UPLOAD", func(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We discussed this: there is no need for NO_CACHE_UPLOAD
. So this test can go.
internal/artifactory_cache_test.go
Outdated
// TestAddWithArtifactoryCache verifies adding a resource with caching enabled. | ||
func TestAddWithArtifactoryCache(t *testing.T) { | ||
// Sub-test to verify behavior when the token is not set. | ||
t.Run("TokenNotSet", func(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the benefit of using t.Run
if you only have one t.Run
per test?
internal/artifactory_cache_test.go
Outdated
// Clear the GRABIT_ARTIFACTORY_TOKEN environment variable. | ||
t.Setenv("GRABIT_ARTIFACTORY_TOKEN", "") | ||
|
||
// Setup a simple HTTP handler that always returns "test content". |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is all this setup really necessary if you're only testing the case where GRABIT_ARTIFACTORY_TOKEN isn't set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rabadin Looks like without defining "t.Setenv("GRABIT_ARTIFACTORY_TOKEN", "")" the test fails when I run "make test" but passes when I run the test directly. Do you have any recommendations for this?
Went ahead and pushed this change to see if it passes the CI.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure what you mean by "when I run the test directly". Do you mean running this individual test in isolation or manually testing things?
I see that the tests pass right but you're not testing all the functionality, which is reflected in the new coverage number (with this PR): 75.8%.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rabadin I mean when I run the test in isolation.
I think one way to up the testing coverage could be testing the normal Artifactory download functionality, I don't think this is currently being tested. If you think this would be a good test to add I can work on adding it when I get home this evening.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By definition the UT coverage is the coverage of the UT.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well, yes, you clearly need to test:
a) the happy path
- Adding a resources with a cache
- Downloading a resource with a cache, getting the data from the cache (using mocking, as discussed)
- Deleting a resource with a cache, deleting the cache (using mocking, as discussed)
b) the non happy path
- Adding resource with a cache but no token env var
- Downloading a resource with a cache, but the cache GET errors
- Deleting a resource with a cache, and deleting the cache works
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't look like the progress bar code was re-integrated properly. When I run:
$ ./grabit download --status
✔[████████████████████] 3/3 Resources 126 MB / 126 MB 13s elapsed
I see an empty status bar for 13 seconds (and 0/3 Resources 0 B / 126 MB) and then all the sudden it's full / done. Something is not right -- it should be continually updating with its actual status.
The SL seems to be working with normal HTTP resources. I do not have Artifactory set up on my machine -- does this only happen when downloading from Artifactory? The SL should be updating properly if Increment is called after every successful download. |
@MDylanWilliams I think this was a bad test on my part. With more proper testing on my side, I think you're right -- it is working properly. Sorry for the confusion! |
I tested incorrectly -- the status bar does appear to be working properly.
72ff3bd
to
811e0f4
Compare
- Added GRABIT_ARTIFACTORY_TOKEN support - Implemented cache download priority - Added fallback to source URL - Added validation at each step - Improved error handling and logging - Integrated cache upload functionality Co-authored-by: Amin Assaid <[email protected]> Co-authored-by: Amy Druham <[email protected]>
811e0f4
to
ae77c89
Compare
- Rename flag and internal arguments to clarify that this caching is specific to artifactory - Refactor tests to cover both the happy path and critical error paths - Move code and test to resource*.go - Add integrity checking for cached artifacts
Implemented Artifactory Cache Integration
Changes
Testing Done
Changes Made Based on Feedback
Checklist