Sporadic MA manifest IT failure due to sequencing_protocol #6697
Assignees
Comments
hannes-ucsc
changed the title
|
I would also like to request these changes to be made to the IT: Index: test/integration_test.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/test/integration_test.py b/test/integration_test.py
--- a/test/integration_test.py (revision dfb41a228ef60b0f3d8ceb968dc94808edfa542b)
+++ b/test/integration_test.py (date 1731220150195)
@@ -1532,7 +1532,7 @@
for bundle in hit['bundles']
}
- managed_access_bundles = set.union(*(
+ ma_bundles = set.union(*(
bundle_uuids(file)
for file in files
if len(file['sources']) == 1
@@ -1542,8 +1542,8 @@
files_url = furl(url=endpoint, path='index/files', args=params)
response = self._get_url_json(GET, files_url)
public_bundle = self.random.choice(sorted(bundle_uuids(one(response['hits']))))
- self.assertNotIn(public_bundle, managed_access_bundles)
- all_bundles = {public_bundle, *managed_access_bundles}
+ self.assertNotIn(public_bundle, ma_bundles)
+ all_bundles = {public_bundle, *ma_bundles}
filters = {
special_fields.bundle_uuid: {
@@ -1611,26 +1611,27 @@
'format': format.value,
'filters': json.dumps({special_fields.bundle_uuid: {'is': list(bundles)}})
})
- content = BytesIO(self._get_url_content(PUT, manifest_url))
- return {
+ readers = {
ManifestFormat.verbatim_jsonl: read_verbatim_jsonl_manifest,
ManifestFormat.verbatim_pfb: read_verbatim_pfb_manifest
- }[format](content)
+ }
+ content = BytesIO(self._get_url_content(PUT, manifest_url))
+ return readers[format](content)
for format in ManifestFormat.verbatim_jsonl, ManifestFormat.verbatim_pfb:
if format in metadata_plugin.manifest_formats:
with self.subTest(format=format):
- unauthorized = get_verbatim_manifest(format, all_bundles)
+ public = get_verbatim_manifest(format, all_bundles)
with self._service_account_credentials:
- authorized = get_verbatim_manifest(format, all_bundles)
- private_only = get_verbatim_manifest(format, managed_access_bundles)
- self.assertSetEqual(private_only, authorized - unauthorized)
+ all = get_verbatim_manifest(format, all_bundles)
+ ma_only = get_verbatim_manifest(format, ma_bundles)
+ self.assertSetEqual(ma_only, all - public)
if ManifestFormat.curl in metadata_plugin.manifest_formats:
# Create a single-file curl manifest and verify that the OAuth2
# token is present on the command line
- managed_access_file_id = one(self.random.choice(files)['files'])['uuid']
- filters = {'fileId': {'is': [managed_access_file_id]}}
+ ma_file_id = one(self.random.choice(files)['files'])['uuid']
+ filters = {'fileId': {'is': [ma_file_id]}}
manifest_url.set(args=dict(catalog=catalog,
filters=json.dumps(filters),
format='curl')) |
|
This protocol ( Thus, the protocol being present in all three manifests (private source only, public source only, and both) is the correct and expected behavior. The problem lies in the test condition, which seems to assume that entities cannot occur in more than one source. Apparently, they can, albeit rarely. How we adjust the test condition depends on the pervasiveness of entities that break this assumption. If only protocols are affected, then we can simply filter protocol entities out of the manifests before comparing the set difference. If, for example, donors or samples can also occur in multiple sources, then we may want to consider a more substantial redesign for this test. |
|
From the DCP/2 specs
I think what we are observing is disallowed in the spec. I think the fix is to remove the snapshot for one of the projects. In order to decide which one, we should investigate a bit more, to understand how this happened, and if |
hannes-ucsc
added
bug
|
Spike to retrieve the snapshot table rows for that sequencing protocol entity and to look for equivalent snapshots in prod using the project title and shortname. |
|
I got this, too, running IT locally. |
|
Same type of failure for a GitLab.dev IT: |
|
Sequencing protocol 76a4bca7 was found in 2 devprod |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
https://gitlab.dev.singlecell.gi.ucsc.edu/ucsc/azul/-/jobs/103847
In both subtests it's a particular
sequencing_protocolentity that's inprivate_onlybut not inauthorized - unauthorized.The text was updated successfully, but these errors were encountered: