Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SHA224, SHA384 and SHA512 source hashes. #4793

Merged
merged 19 commits into from
Nov 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions conda_build/metadata.py
Original file line number Diff line number Diff line change
Expand Up @@ -566,7 +566,10 @@ def parse(data, config, path=None):
"url": None,
"md5": str,
"sha1": None,
"sha224": None,
"sha256": None,
"sha384": None,
"sha512": None,
"path": str,
"path_via_symlink": None,
"git_url": str,
Expand Down
31 changes: 17 additions & 14 deletions conda_build/source.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@
ext_re = re.compile(r"(.*?)(\.(?:tar\.)?[^.]+)$")


ACCEPTED_HASH_TYPES = ("md5", "sha1", "sha224", "sha256", "sha384", "sha512")


def append_hash_to_fn(fn, hash_value):
return ext_re.sub(rf"\1_{hash_value[:10]}\2", fn)

Expand All @@ -66,16 +69,16 @@ def download_to_cache(cache_folder, recipe_path, source_dict, verbose=False):
source_dict["fn"] if "fn" in source_dict else basename(source_urls[0])
)
hash_added = False
for hash_type in ("md5", "sha1", "sha256"):
if hash_type in source_dict:
if source_dict[hash_type] in (None, ""):
raise ValueError(f"Empty {hash_type} hash provided for {fn}")
fn = append_hash_to_fn(fn, source_dict[hash_type])
hash_added = True
break

for hash_type in sorted(set(source_dict).intersection(ACCEPTED_HASH_TYPES)):
if source_dict[hash_type] in (None, ""):
raise ValueError(f"Empty {hash_type} hash provided for {fn}")
fn = append_hash_to_fn(fn, source_dict[hash_type])
hash_added = True
break
else:
log.warning(
f"No hash (md5, sha1, sha256) provided for {unhashed_fn}. Source download forced. "
f"No hash {ACCEPTED_HASH_TYPES} provided for {unhashed_fn}. Source download forced. "
"Add hash to recipe to use source cache."
)
path = join(cache_folder, fn)
Expand Down Expand Up @@ -116,16 +119,16 @@ def download_to_cache(cache_folder, recipe_path, source_dict, verbose=False):
raise RuntimeError(f"Could not download {url}")

hashed = None
for tp in ("md5", "sha1", "sha256"):
if tp in source_dict:
expected_hash = source_dict[tp]
hashed = compute_sum(path, tp)

for hash_type in set(source_dict).intersection(ACCEPTED_HASH_TYPES):
if hash_type in source_dict:
expected_hash = source_dict[hash_type]
hashed = compute_sum(path, hash_type)
if expected_hash != hashed:
rm_rf(path)
raise RuntimeError(
f"{tp.upper()} mismatch: '{hashed}' != '{expected_hash}'"
f"{hash_type.upper()} mismatch: '{hashed}' != '{expected_hash}'"
)
break
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting, we were only computing the weakest one and ignoring the rest?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that's fun.


# this is really a fallback. If people don't provide the hash, we still need to prevent
# collisions in our source cache, but the end user will get no benefit from the cache.
Expand Down
3 changes: 3 additions & 0 deletions docs/source/resources/define-metadata.rst
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,10 @@ Source from tarball or zip archive
url: https://pypi.python.org/packages/source/b/bsdiff4/bsdiff4-1.1.4.tar.gz
md5: 29f6089290505fc1a852e176bd276c43
sha1: f0a2c9a30073449cfb7d171c57552f3109d93894
sha224: ebf3e3b54353146ca21128ed6399739663a1256a223f438ed0223845
sha256: 5a022ff4c1d1de87232b1c70bde50afbb98212fd246be4a867d8737173cf1f8f
sha384: 23eee6ee2e5d1054780e331857589bfba098255a88ae4edd47102fce676694ce0f543dc5c0d27c51f77cc4546d4e74c0
sha512: b968c7dc99132252a83b175a96ec75ec842edf9e2494db2c07b419e61a0b1cf6984e7c544452f9ab56aa8581caf966c0f6933fc22a071ccc4fbb5d22b363fe54

If an extracted archive contains only 1 folder at its top level, its contents
will be moved 1 level up, so that the extracted package contents sit in the
Expand Down
19 changes: 19 additions & 0 deletions news/4793-support-sha224-sha384-and-sha512
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
### Enhancements

* Added SHA224, SHA384 and SHA512 support for validating downloaded sources. (#4793)

### Bug fixes

* Verify all source hashes when multiple are defined (not just the first one). (#4793)

### Deprecations

* <news item>

### Docs

* <news item>

### Other

* <news item>
3 changes: 3 additions & 0 deletions tests/test-recipes/metadata/source_url/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ source:
url: https://github.com/conda/conda-build/archive/1.8.1.tar.gz
md5: 0bf1f3598a659a0e8fb5ee6bbb3fd9fd
sha1: c464a8995ad6bbf0480abd2883876cc9b4913fa7
sha224: 96d76b37dcc8c28577ae5776e3aa3eb3b057af60983ce4005bbd3d61
sha256: f82b0bd5c809c9a7c7256c26364a0065e57732788b7a74c7ea2169135ed2f598
sha384: 5b407afd0c41028bd2443b13a1e34092053452db9e1872eed427f9a6042626837d16f0268568e8e54a07d173a3c80e6b
sha512: 6bf7d22fc65c111402d22fd27bbcf7ea36eec03efded33ece74a54bf30f0cb87b6f31b789276e990dd6e9e41ac78739eac8c6b2954144096ccf0bb2ec1cc4fd7

requirements:
build:
Expand Down
Loading