diff --git a/README.md b/README.md index cc70a6a..0004510 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,14 @@ # fetch -fetch downloads all or a subset of files or folders from a specific git commit, branch or tag of a GitHub repo. +fetch downloads all or a subset of files or folders from a specific git commit, branch, tag, or release of a GitHub +repo. #### Features - Download from a specific git commit SHA. - Download from a specific git tag. - Download from a specific git branch. +- Download a binary asset from a specific release. - Download from public repos. - Download from private repos by specifying a [GitHub Personal Access Token](https://help.github.com/articles/creating-an-access-token-for-command-line-use/). - Download a single file, a subset of files, or all files from the repo. @@ -15,40 +17,52 @@ fetch downloads all or a subset of files or folders from a specific git commit, ## Motivation [Gruntwork](http://gruntwork.io) helps software teams get up and running on AWS with DevOps best practices and world-class -infrastructure in about 2 weeks. Sometimes we publish scripts that clients use in their infrastructure, and we want clients -to auto-download the latest non-breaking version of a script when we publish updates. In addition, for security reasons, -we wish to verify the integrity of the git commit being downloaded. +infrastructure in about 2 weeks. Sometimes we publish scripts and binaries that clients use in their infrastructure, +and we want clients to auto-download the latest non-breaking version of that script or binary when we publish updates. +In addition, for security reasons, we wish to verify the integrity of the git commit being downloaded. ## Installation -Download the binary from the [GitHub Releases](https://github.com/gruntwork-io/fetch/releases) tab. +Download the fetch binary from the [GitHub Releases](https://github.com/gruntwork-io/fetch/releases) tab. ## Assumptions -fetch assumes that a repo's tags are in the format `vX.Y.Z` or `X.Y.Z` to support Semantic Versioning parsing. Repos that -use git tags not in this format cannot be used with fetch. +fetch assumes that a repo's tags are in the format `vX.Y.Z` or `X.Y.Z` to support Semantic Versioning parsing. Repos +that use git tags not in this format cannot currently be used with fetch. ## Usage #### General Usage ``` -fetch --repo= --tag= [] +fetch [OPTIONS] ``` -- `` - **Optional**. - If blank, all files in the repo will be downloaded. Otherwise, only files located in the given path - will be downloaded. - - Example: `/` Download all files in the repo - - Example: `/folder` Download only files in the `/folder` path and below from the repo -- `` - **Required**. - The local path where all files should be downloaded. - - Example: `/tmp` Download all files to `/tmp` +The supported options are: -Run `fetch --help` to see more information about the flags, some of which are required. See [Tag Constraint Expressions](#tag-constraint-expressions) -for examples of tag constraints you can use. +- `--repo` (**Required**): The fully qualified URL of the GitHub repo to download from (e.g. https://github.com/foo/bar). +- `--tag` (**Optional**): The git tag to download. Can be a specific tag or a [Tag Constraint + Expression](#tag-constraint-expressions). +- `--branch` (**Optional**): The git branch from which to download; the latest commit in the branch will be used. If + specified, will override `--tag`. +- `--commit` (**Optional**): The SHA of a git commit to download. If specified, will override `--branch` and `--tag`. +- `--source-path` (**Optional**): The source path to download from the repo (e.g. `--source-path=/folder` will download + the `/folder` path and all files below it). By default, all files are downloaded from the repo unless `--source-path` + or `--release-asset` is specified. This option can be specified more than once. +- `--release-asset` (**Optional**): The name of a release asset--that is, a binary uploaded to a [GitHub + Release](https://help.github.com/articles/creating-releases/)--to download. This option can be specified more than + once. It only works with the `--tag` option. +- `--github-oauth-token` (**Optional**): A [GitHub Personal Access + Token](https://help.github.com/articles/creating-an-access-token-for-command-line-use/). Required if you're + downloading from private GitHub repos. **NOTE:** fetch will also look for this token using the `GITHUB_OAUTH_TOKEN` + environment variable, which we recommend using instead of the command line option to ensure the token doesn't get + saved in bash history. + +The supported arguments are: + +- `` (**Required**): The local path where all files should be downloaded (e.g. `/tmp`). + +Run `fetch --help` to see more information about the flags. #### Usage Example 1 @@ -58,7 +72,7 @@ Download `/modules/foo/bar.sh` from a GitHub release where the tag is the latest fetch \ --repo="https://github.com/gruntwork-io/script-modules" \ --tag="~>0.1.5" \ -/modules/foo/bar.sh \ +--source-path="/modules/foo/bar.sh" \ /tmp/bar ``` @@ -70,7 +84,7 @@ Download all files in `/modules/foo` from a GitHub release where the tag is exac fetch \ --repo="https://github.com/gruntwork-io/script-modules" \ --tag="0.1.5" \ -/modules/foo \ +--source-path="/modules/foo" \ /tmp ``` @@ -80,10 +94,11 @@ fetch \ Download all files from a private GitHub repo using the GitHUb oAuth Token `123`. Get the release whose tag is exactly `0.1.5`, and save the files to `/tmp`: ``` +GITHUB_OAUTH_TOKEN=123 + fetch \ --repo="https://github.com/gruntwork-io/script-modules" \ --tag="0.1.5" \ ---github-oauth-token="123" \ /tmp ``` @@ -112,6 +127,17 @@ fetch \ ``` +#### Usage Example 6 + +Download the release asset `foo.exe` from a GitHub release where the tag is exactly `0.1.5`, and save it to `/tmp`: + +``` +fetch \ +--repo="https://github.com/gruntwork-io/script-modules" \ +--tag="0.1.5" \ +--release-asset="foo.exe" \ +/tmp + #### Tag Constraint Expressions diff --git a/fetch_error.go b/fetch_error.go index 98ea1cc..bf75257 100644 --- a/fetch_error.go +++ b/fetch_error.go @@ -23,6 +23,9 @@ func newError(errorCode int, details string) *FetchError { } func wrapError(err error) *FetchError { + if err == nil { + return nil + } return &FetchError{ errorCode: -1, details: err.Error(), diff --git a/github.go b/github.go index e908943..55e2de7 100644 --- a/github.go +++ b/github.go @@ -6,11 +6,15 @@ import ( "bytes" "encoding/json" "regexp" + "os" + "io" ) type GitHubRepo struct { + Url string // The URL of the GitHub repo Owner string // The GitHub account name under which the repo exists Name string // The GitHub repo name + Token string // The personal access token to access this repo (if it's a private repo) } // Represents a specific git commit. @@ -40,35 +44,37 @@ type GitHubTagsCommitApiResponse struct { Url string // The URL at which additional API information can be found for the given commit } +// Modeled directly after the api.github.com response (but only includes the fields we care about). For more info, see: +// https://developer.github.com/v3/repos/releases/#get-a-release-by-tag-name +type GitHubReleaseApiResponse struct { + Id int + Url string + Name string + Assets []GitHubReleaseAsset +} + +// The "assets" portion of the GitHubReleaseApiResponse. Modeled directly after the api.github.com response (but only +// includes the fields we care about). For more info, see: +// https://developer.github.com/v3/repos/releases/#get-a-release-by-tag-name +type GitHubReleaseAsset struct { + Id int + Url string + Name string +} + // Fetch all tags from the given GitHub repo func FetchTags(githubRepoUrl string, githubToken string) ([]string, *FetchError) { var tagsString []string - repo, err := ParseUrlIntoGitHubRepo(githubRepoUrl) + repo, err := ParseUrlIntoGitHubRepo(githubRepoUrl, githubToken) if err != nil { return tagsString, wrapError(err) } - // Make an HTTP request, possibly with the gitHubOAuthToken in the header - httpClient := &http.Client{} - - req, err := MakeGitHubTagsRequest(repo, githubToken) + url := createGitHubRepoUrlForPath(repo, "tags") + resp, err := callGitHubApi(repo, url, map[string]string{}) if err != nil { - return tagsString, wrapError(err) - } - - resp, err := httpClient.Do(req) - if err != nil { - return tagsString, wrapError(err) - } - if resp.StatusCode != 200 { - // Convert the resp.Body to a string - buf := new(bytes.Buffer) - buf.ReadFrom(resp.Body) - respBody := buf.String() - - // We leverage the HTTP Response Code as our ErrorCode here. - return tagsString, newError(resp.StatusCode, fmt.Sprintf("Received HTTP Response %d while fetching releases for GitHub URL %s. Full HTTP response: %s", resp.StatusCode, githubRepoUrl, respBody)) + return tagsString, err } // Convert the response body to a byte array @@ -78,8 +84,7 @@ func FetchTags(githubRepoUrl string, githubToken string) ([]string, *FetchError) // Extract the JSON into our array of gitHubTagsCommitApiResponse's var tags []GitHubTagsApiResponse - err = json.Unmarshal(jsonResp, &tags) - if err != nil { + if err := json.Unmarshal(jsonResp, &tags); err != nil { return tagsString, wrapError(err) } @@ -91,7 +96,7 @@ func FetchTags(githubRepoUrl string, githubToken string) ([]string, *FetchError) } // Convert a URL into a GitHubRepo struct -func ParseUrlIntoGitHubRepo(url string) (GitHubRepo, error) { +func ParseUrlIntoGitHubRepo(url string, token string) (GitHubRepo, *FetchError) { var gitHubRepo GitHubRepo regex, regexErr := regexp.Compile("https?://(?:www\\.)?github.com/(.+?)/(.+?)(?:$|\\?|#|/)") @@ -105,26 +110,97 @@ func ParseUrlIntoGitHubRepo(url string) (GitHubRepo, error) { } gitHubRepo = GitHubRepo{ + Url: url, Owner: matches[1], Name: matches[2], + Token: token, } return gitHubRepo, nil } +// Download the release asset with the given id and return its body +func DownloadReleaseAsset(repo GitHubRepo, assetId int, destPath string) *FetchError { + url := createGitHubRepoUrlForPath(repo, fmt.Sprintf("releases/assets/%d", assetId)) + resp, err := callGitHubApi(repo, url, map[string]string{"Accept": "application/octet-stream"}) + if err != nil { + return err + } + + return writeResonseToDisk(resp, destPath) +} -// Return an HTTP request that will fetch the given GitHub repo's tags, possibly with the gitHubOAuthToken in the header -func MakeGitHubTagsRequest(repo GitHubRepo, gitHubToken string) (*http.Request, error) { - var request *http.Request +// Get information about the GitHub release with the given tag +func GetGitHubReleaseInfo(repo GitHubRepo, tag string) (GitHubReleaseApiResponse, *FetchError) { + release := GitHubReleaseApiResponse{} - request, err := http.NewRequest("GET", fmt.Sprintf("https://api.github.com/repos/%s/%s/tags", repo.Owner, repo.Name), nil) + url := createGitHubRepoUrlForPath(repo, fmt.Sprintf("releases/tags/%s", tag)) + resp, err := callGitHubApi(repo, url, map[string]string{}) if err != nil { - return request, wrapError(err) + return release, err } - if gitHubToken != "" { - request.Header.Set("Authorization", fmt.Sprintf("token %s", gitHubToken)) + // Convert the response body to a byte array + buf := new(bytes.Buffer) + buf.ReadFrom(resp.Body) + jsonResp := buf.Bytes() + + if err := json.Unmarshal(jsonResp, &release); err != nil { + return release, wrapError(err) } - return request, nil + return release, nil } + +// Craft a URL for the GitHub repos API of the form repos/:owner/:repo/:path +func createGitHubRepoUrlForPath(repo GitHubRepo, path string) string { + return fmt.Sprintf("repos/%s/%s/%s", repo.Owner, repo.Name, path) +} + +// Call the GitHub API at the given path and return the HTTP response +func callGitHubApi(repo GitHubRepo, path string, customHeaders map[string]string) (*http.Response, *FetchError) { + httpClient := &http.Client{} + + request, err := http.NewRequest("GET", fmt.Sprintf("https://api.github.com/%s", path), nil) + if err != nil { + return nil, wrapError(err) + } + + if repo.Token != "" { + request.Header.Set("Authorization", fmt.Sprintf("token %s", repo.Token)) + } + + for headerName, headerValue := range customHeaders { + request.Header.Set(headerName, headerValue) + } + + resp, err := httpClient.Do(request) + if err != nil { + return nil, wrapError(err) + } + if resp.StatusCode != 200 { + // Convert the resp.Body to a string + buf := new(bytes.Buffer) + buf.ReadFrom(resp.Body) + respBody := buf.String() + + // We leverage the HTTP Response Code as our ErrorCode here. + return nil, newError(resp.StatusCode, fmt.Sprintf("Received HTTP Response %d while fetching releases for GitHub URL %s. Full HTTP response: %s", resp.StatusCode, repo.Url, respBody)) + } + + return resp, nil +} + +// Write the body of the given HTTP response to disk at the given path +func writeResonseToDisk(resp *http.Response, destPath string) *FetchError { + out, err := os.Create(destPath) + if err != nil { + return wrapError(err) + } + + defer out.Close() + defer resp.Body.Close() + + _, err = io.Copy(out, resp.Body) + return wrapError(err) +} \ No newline at end of file diff --git a/github_test.go b/github_test.go index 20931ba..6f33c9a 100644 --- a/github_test.go +++ b/github_test.go @@ -3,6 +3,8 @@ package main import ( "testing" "os" + "reflect" + "io/ioutil" ) func TestGetListOfReleasesFromGitHubRepo(t *testing.T) { @@ -52,19 +54,20 @@ func TestParseUrlIntoGitHubRepo(t *testing.T) { repoUrl string owner string name string + token string }{ - {"https://github.com/brikis98/ping-play", "brikis98", "ping-play"}, - {"http://github.com/brikis98/ping-play", "brikis98", "ping-play"}, - {"https://github.com/gruntwork-io/script-modules", "gruntwork-io", "script-modules"}, - {"http://github.com/gruntwork-io/script-modules", "gruntwork-io", "script-modules"}, - {"http://www.github.com/gruntwork-io/script-modules", "gruntwork-io", "script-modules"}, - {"http://www.github.com/gruntwork-io/script-modules/", "gruntwork-io", "script-modules"}, - {"http://www.github.com/gruntwork-io/script-modules?foo=bar", "gruntwork-io", "script-modules"}, - {"http://www.github.com/gruntwork-io/script-modules?foo=bar&foo=baz", "gruntwork-io", "script-modules"}, + {"https://github.com/brikis98/ping-play", "brikis98", "ping-play", ""}, + {"http://github.com/brikis98/ping-play", "brikis98", "ping-play", ""}, + {"https://github.com/gruntwork-io/script-modules", "gruntwork-io", "script-modules", ""}, + {"http://github.com/gruntwork-io/script-modules", "gruntwork-io", "script-modules", ""}, + {"http://www.github.com/gruntwork-io/script-modules", "gruntwork-io", "script-modules", ""}, + {"http://www.github.com/gruntwork-io/script-modules/", "gruntwork-io", "script-modules", ""}, + {"http://www.github.com/gruntwork-io/script-modules?foo=bar", "gruntwork-io", "script-modules", "token"}, + {"http://www.github.com/gruntwork-io/script-modules?foo=bar&foo=baz", "gruntwork-io", "script-modules", "token"}, } for _, tc := range cases { - repo, err := ParseUrlIntoGitHubRepo(tc.repoUrl) + repo, err := ParseUrlIntoGitHubRepo(tc.repoUrl, tc.token) if err != nil { t.Fatalf("error extracting url %s into a GitHubRepo struct: %s", tc.repoUrl, err) } @@ -76,6 +79,14 @@ func TestParseUrlIntoGitHubRepo(t *testing.T) { if repo.Name != tc.name { t.Fatalf("while extracting %s, expected name %s, received %s", tc.repoUrl, tc.name, repo.Name) } + + if repo.Url != tc.repoUrl { + t.Fatalf("while extracting %s, expected url %s, received %s", tc.repoUrl, tc.repoUrl, repo.Url) + } + + if repo.Token != tc.token { + t.Fatalf("while extracting %s, expected token %s, received %s", tc.repoUrl, tc.token, repo.Token) + } } } @@ -91,9 +102,104 @@ func TestParseUrlThrowsErrorOnMalformedUrl(t *testing.T) { } for _, tc := range cases { - _, err := ParseUrlIntoGitHubRepo(tc.repoUrl) + _, err := ParseUrlIntoGitHubRepo(tc.repoUrl, "") if err == nil { t.Fatalf("Expected error on malformed url %s, but no error was received.", tc.repoUrl) } } +} + +func TestGetGitHubReleaseInfo(t *testing.T) { + t.Parallel() + + token := os.Getenv("GITHUB_OAUTH_TOKEN") + + expectedFetchTestPrivateRelease := GitHubReleaseApiResponse{ + Id: 3064041, + Url: "https://api.github.com/repos/gruntwork-io/fetch-test-private/releases/3064041", + Name: "v0.0.2", + Assets: []GitHubReleaseAsset{ + { + Id: 1872521, + Url: "https://api.github.com/repos/gruntwork-io/fetch-test-private/releases/assets/1872521", + Name: "test-asset.png", + }, + }, + } + + expectedFetchTestPublicRelease := GitHubReleaseApiResponse{ + Id: 3065803, + Url: "https://api.github.com/repos/gruntwork-io/fetch-test-public/releases/3065803", + Name: "v0.0.3", + Assets: []GitHubReleaseAsset{}, + } + + cases := []struct { + repoUrl string + repoToken string + tag string + expected GitHubReleaseApiResponse + }{ + {"https://github.com/gruntwork-io/fetch-test-private", token, "v0.0.2", expectedFetchTestPrivateRelease}, + {"https://github.com/gruntwork-io/fetch-test-public", "", "v0.0.3", expectedFetchTestPublicRelease}, + } + + for _, tc := range cases { + repo, err := ParseUrlIntoGitHubRepo(tc.repoUrl, tc.repoToken) + if err != nil { + t.Fatalf("Failed to parse %s into GitHub URL due to error: %s", tc.repoUrl, err.Error()) + } + + resp, err := GetGitHubReleaseInfo(repo, tc.tag) + if err != nil { + t.Fatalf("Failed to fetch GitHub release info for repo %s due to error: %s", tc.repoToken, err.Error()) + } + + if !reflect.DeepEqual(tc.expected, resp) { + t.Fatalf("Expected GitHub release %s but got GitHub release %s", tc.expected, resp) + } + } +} + +func TestDownloadReleaseAsset(t *testing.T) { + t.Parallel() + + token := os.Getenv("GITHUB_OAUTH_TOKEN") + + cases := []struct { + repoUrl string + repoToken string + tag string + assetId int + }{ + {"https://github.com/gruntwork-io/fetch-test-private", token, "v0.0.2", 1872521}, + {"https://github.com/gruntwork-io/fetch-test-public", "", "v0.0.2", 1872641}, + } + + for _, tc := range cases { + repo, err := ParseUrlIntoGitHubRepo(tc.repoUrl, tc.repoToken) + if err != nil { + t.Fatalf("Failed to parse %s into GitHub URL due to error: %s", tc.repoUrl, err.Error()) + } + + tmpFile, tmpErr := ioutil.TempFile("", "test-download-release-asset") + if tmpErr != nil { + t.Fatalf("Failed to create temp file due to error: %s", tmpErr.Error()) + } + + if err := DownloadReleaseAsset(repo, tc.assetId, tmpFile.Name()); err != nil { + t.Fatalf("Failed to download asset %s to %s from GitHub URL %s due to error: %s", tc.assetId, tmpFile.Name(), tc.repoUrl, err.Error()) + } + + defer os.Remove(tmpFile.Name()) + + if !fileExists(tmpFile.Name()) { + t.Fatalf("Got no errors downloading asset %s to %s from GitHub URL %s, but %s does not exist!", tc.assetId, tmpFile.Name(), tc.repoUrl, tmpFile.Name()) + } + } +} + +func fileExists(path string) bool { + _, err := os.Stat(path) + return err == nil } \ No newline at end of file diff --git a/main.go b/main.go index ba2732e..a9a8114 100644 --- a/main.go +++ b/main.go @@ -5,35 +5,66 @@ import ( "github.com/codegangsta/cli" "fmt" "errors" + "path" ) +type FetchOptions struct { + RepoUrl string + CommitSha string + BranchName string + TagConstraint string + GithubToken string + SourcePaths []string + ReleaseAssets []string + LocalDownloadPath string +} + +const OPTION_REPO = "repo" +const OPTION_COMMIT = "commit" +const OPTION_BRANCH = "branch" +const OPTION_TAG = "tag" +const OPTION_GITHUB_TOKEN = "github-oauth-token" +const OPTION_SOURCE_PATH = "source-path" +const OPTION_RELEASE_ASSET = "release-asset" + +const ENV_VAR_GITHUB_TOKEN = "GITHUB_OAUTH_TOKEN" + func main() { app := cli.NewApp() app.Name = "fetch" - app.Usage = "download a file or folder from a specific release of a public or private GitHub repo subject to the Semantic Versioning constraints you impose" - app.UsageText = "fetch [global options] [] \n (See https://github.com/gruntwork-io/fetch for examples, argument definitions, and additional docs.)" + app.Usage = "download a file, folder, or release asset from a specific release of a public or private GitHub repo subject to the Semantic Versioning constraints you impose" + app.UsageText = "fetch [global options] \n (See https://github.com/gruntwork-io/fetch for examples, argument definitions, and additional docs.)" app.Version = getVersion(Version, VersionPrerelease) app.Flags = []cli.Flag{ cli.StringFlag{ - Name: "repo", + Name: OPTION_REPO, Usage: "Required. Fully qualified URL of the GitHub repo.", }, cli.StringFlag{ - Name: "commit", + Name: OPTION_COMMIT, Usage: "The specific git commit SHA to download. If specified, will override --branch and --tag.", }, cli.StringFlag{ - Name: "branch", + Name: OPTION_BRANCH, Usage: "The git branch from which to download the commit; the latest commit in th branch will be used. If specified, will override --tag.", }, cli.StringFlag{ - Name: "tag", + Name: OPTION_TAG, Usage: "The specific git tag to download, expressed with Version Constraint Operators.\n\tIf left blank, fetch will download the latest git tag.\n\tSee https://github.com/gruntwork-io/fetch#version-constraint-operators for examples.", }, cli.StringFlag{ - Name: "github-oauth-token", - Usage: "Required for private repos. A GitHub Personal Access Token (https://help.github.com/articles/creating-an-access-token-for-command-line-use/).", + Name: OPTION_GITHUB_TOKEN, + Usage: "A GitHub Personal Access Token, which is required for downloading from private repos.", + EnvVar: ENV_VAR_GITHUB_TOKEN, + }, + cli.StringSliceFlag{ + Name: OPTION_SOURCE_PATH, + Usage: "The source path to download from the repo. If this or --release-asset aren't specified, all files are downloaded. Can be specified more than once.", + }, + cli.StringSliceFlag{ + Name: OPTION_RELEASE_ASSET, + Usage: "The name of a release asset--that is, a binary uploaded to a GitHub Release--to download. Only works with --tag. Can be specified more than once.", }, } @@ -54,40 +85,13 @@ func runFetchWrapper (c *cli.Context) { // Run the fetch program func runFetch (c *cli.Context) error { - - // Validate required flags - if c.String("repo") == "" { - return fmt.Errorf("The --repo flag is required. Run \"fetch --help\" for full usage info.") - } - - repoUrl := c.String("repo") - commitSha := c.String("commit") - branchName := c.String("branch") - tagConstraint := c.String("tag") - githubToken := c.String("github-oauth-token") - - // Validate args - if len(c.Args()) == 0 || len(c.Args()) > 2 { - return fmt.Errorf("Missing required arguments. Run \"fetch --help\" for full usage info.") - } - - var repoDownloadFilter string - var localFileDst string - - // Assume the arg is missing, so set a default - if len(c.Args()) == 1 { - repoDownloadFilter = "/" - localFileDst = c.Args()[0] - } - - // We have two args so load both - if len(c.Args()) == 2 { - repoDownloadFilter = c.Args()[0] - localFileDst = c.Args()[1] + options := parseOptions(c) + if err := validateOptions(options); err != nil { + return err } // Get the tags for the given repo - tags, err := FetchTags(repoUrl, githubToken) + tags, err := FetchTags(options.RepoUrl, options.GithubToken) if err != nil { if err.errorCode == INVALID_GITHUB_TOKEN_OR_ACCESS_DENIED { return errors.New(getErrorMessage(INVALID_GITHUB_TOKEN_OR_ACCESS_DENIED, err.details)) @@ -99,7 +103,7 @@ func runFetch (c *cli.Context) error { } // Find the specific release that matches the latest version constraint - latestTag, err := getLatestAcceptableTag(tagConstraint, tags) + latestTag, err := getLatestAcceptableTag(options.TagConstraint, tags) if err != nil { if err.errorCode == INVALID_TAG_CONSTRAINT_EXPRESSION { return errors.New(getErrorMessage(INVALID_TAG_CONSTRAINT_EXPRESSION, err.details)) @@ -109,17 +113,75 @@ func runFetch (c *cli.Context) error { } // Prepare the vars we'll need to download - repo, goErr := ParseUrlIntoGitHubRepo(repoUrl) - if goErr != nil { + repo, err := ParseUrlIntoGitHubRepo(options.RepoUrl, options.GithubToken) + if err != nil { return fmt.Errorf("Error occurred while parsing GitHub URL: %s", err) } + // If no release assets and no source paths are specified, then by default, download all the source files from + // the repo + if len(options.SourcePaths) == 0 && len(options.ReleaseAssets) == 0 { + options.SourcePaths = []string{"/"} + } + + // Download any requested source files + if err := downloadSourcePaths(options.SourcePaths, options.LocalDownloadPath, repo, latestTag, options.BranchName, options.CommitSha); err != nil { + return err + } + + // Download any requested release assets + if err := downloadReleaseAssets(options.ReleaseAssets, options.LocalDownloadPath, repo, latestTag); err != nil { + return err + } + + return nil +} + +func parseOptions(c *cli.Context) FetchOptions { + return FetchOptions{ + RepoUrl: c.String(OPTION_REPO), + CommitSha: c.String(OPTION_COMMIT), + BranchName: c.String(OPTION_BRANCH), + TagConstraint: c.String(OPTION_TAG), + GithubToken: c.String(OPTION_GITHUB_TOKEN), + SourcePaths: c.StringSlice(OPTION_SOURCE_PATH), + ReleaseAssets: c.StringSlice(OPTION_RELEASE_ASSET), + LocalDownloadPath: c.Args().First(), + } +} + +func validateOptions(options FetchOptions) error { + if options.RepoUrl == "" { + return fmt.Errorf("The --%s flag is required. Run \"fetch --help\" for full usage info.", OPTION_REPO) + } + + if options.LocalDownloadPath == "" { + return fmt.Errorf("Missing required arguments specifying the local download path. Run \"fetch --help\" for full usage info.") + } + + if options.TagConstraint == "" && options.CommitSha == "" && options.BranchName == "" { + return fmt.Errorf("You must specify exactly one of --%s, --%s, or --%s. Run \"fetch --help\" for full usage info.", OPTION_TAG, OPTION_COMMIT, OPTION_BRANCH) + } + + if len(options.ReleaseAssets) > 0 && options.TagConstraint == "" { + return fmt.Errorf("The --%s flag can only be used with --%s. Run \"fetch --help\" for full usage info.", OPTION_RELEASE_ASSET, OPTION_TAG) + } + + return nil +} + +// Download the specified source files from the given repo +func downloadSourcePaths(sourcePaths []string, destPath string, githubRepo GitHubRepo, latestTag string, branchName string, commitSha string) error { + if len(sourcePaths) == 0 { + return nil + } + // We want to respect the GitHubCommit Hierarchy of "CommitSha > GitTag > BranchName" // Note that CommitSha or BranchName may be blank here if the user did not specify values for these. // If the user specified no value for GitTag, our call to getLatestAcceptableTag() above still gave us some value // So we can guarantee (at least logically) that this struct instance is in a valid state right now. gitHubCommit := GitHubCommit{ - Repo: repo, + Repo: githubRepo, GitTag: latestTag, BranchName: branchName, CommitSha: commitSha, @@ -127,31 +189,71 @@ func runFetch (c *cli.Context) error { // Download that release as a .zip file if gitHubCommit.CommitSha != "" { - fmt.Printf("Downloading git commit \"%s\" of %s ...\n", gitHubCommit.CommitSha, repoUrl) + fmt.Printf("Downloading git commit \"%s\" of %s ...\n", gitHubCommit.CommitSha, githubRepo.Url) } else if gitHubCommit.BranchName != "" { - fmt.Printf("Downloading latest commit from branch \"%s\" of %s ...\n", gitHubCommit.BranchName, repoUrl) + fmt.Printf("Downloading latest commit from branch \"%s\" of %s ...\n", gitHubCommit.BranchName, githubRepo.Url) } else if gitHubCommit.GitTag != "" { - fmt.Printf("Downloading tag \"%s\" of %s ...\n", latestTag, repoUrl) + fmt.Printf("Downloading tag \"%s\" of %s ...\n", latestTag, githubRepo.Url) } else { return fmt.Errorf("The commit sha, tag, and branch name are all empty.") } - localZipFilePath, err := downloadGithubZipFile(gitHubCommit, githubToken) + localZipFilePath, err := downloadGithubZipFile(gitHubCommit, githubRepo.Token) if err != nil { return fmt.Errorf("Error occurred while downloading zip file from GitHub repo: %s", err) } defer cleanupZipFile(localZipFilePath) // Unzip and move the files we need to our destination - fmt.Printf("Extracting files from %s to %s ...\n", repoDownloadFilter, localFileDst) - if goErr = extractFiles(localZipFilePath, repoDownloadFilter, localFileDst); goErr != nil { - return fmt.Errorf("Error occurred while extracting files from GitHub zip file: %s", goErr) + for _, sourcePath := range sourcePaths { + fmt.Printf("Extracting files from %s to %s ...\n", sourcePath, destPath) + if err := extractFiles(localZipFilePath, sourcePath, destPath); err != nil { + return fmt.Errorf("Error occurred while extracting files from GitHub zip file: %s", err.Error()) + } } fmt.Println("Download and file extraction complete.") return nil } +// Download the specified binary files that were uploaded as release assets to the specified GitHub release +func downloadReleaseAssets(releaseAssets []string, destPath string, githubRepo GitHubRepo, latestTag string) error { + if len(releaseAssets) == 0 { + return nil + } + + release, err := GetGitHubReleaseInfo(githubRepo, latestTag) + if err != nil { + return err + } + + for _, assetName := range releaseAssets { + asset := findAssetInRelease(assetName, release) + if asset == nil { + return fmt.Errorf("Could not find asset %s in release %s", assetName, latestTag) + } + + assetPath := path.Join(destPath, asset.Name) + fmt.Printf("Downloading release asset %s to %s\n", asset.Name, assetPath) + if err := DownloadReleaseAsset(githubRepo, asset.Id, assetPath); err != nil { + return err + } + } + + fmt.Println("Download of release assets complete.") + return nil +} + +func findAssetInRelease(assetName string, release GitHubReleaseApiResponse) *GitHubReleaseAsset { + for _, asset := range release.Assets { + if asset.Name == assetName { + return &asset + } + } + + return nil +} + // Delete the given zip file. func cleanupZipFile(localZipFilePath string) error { err := os.Remove(localZipFilePath)