CMR-10238: Adding the virtual catalog 'ALL'

[doug-newman-nasa]

Overview

What is the feature?

Allows users to perform collection searches across all providers.

What is the Solution?

Add the virtual catalog 'ALL' which will return collection inventories without a provider clause in the collection query

What areas of the application does this impact?

• /stac - adds the 'all' child
• /stac/ALL - returns collections across all providers using each collection's provider to formulate rel=child links.
• /stac/ALL/collections - returns detailed collections across all providers using each collection's provider to formulate
• rel=item links.
• /stac/ALL/search - disallowed

Testing

Reproduction steps

• SIT
• Any collection

1. Navigate to cmr.sit.earthdata.nasa.gov/stac
2. Observe child link
   { "rel": "child", "title": "all", "type": "application/json", "href": "https://cmr.sit.earthdata.nasa.gov/stac/ALL" }
3. Navigate to cmr.sit.earthdata.nasa.gov/stac/ALL
4. Observe absence of rel=search links
5. Observe standard rel=child links for each collection but note that the route to those collections is via their provider and not 'all'
   { "rel": "child", "href": "https://cmr.sit.earthdata.nasa.gov/stac/AMD_KOPRI/collections/Test%201_1.2", "title": "\"The Omnivores Dilemma\": The Effect of Autumn Diet on Winter Physiology and Condition of Juvenile Antarctic Krill", "type": "application/json" },
6. Navigate to cmr.sit.earthdata.nasa.gov/stac/ALL/collections
7. Observe, for each collection, the presence of a rel=items link that refers to the collection provider rather than 'all'. For example:
   { "rel": "items", "href": "https://cmr.sit.earthdata.nasa.gov/stac/AMD_KOPRI/collections/Test%201_1.2/items", "type": "application/geo+json", "title": "Collection Items" }
8. Navigate to cmr.sit.earthdata.nasa.gov/stac/ALL/collections/Test%201_1.2
9. Observe a 404 error
10. Navigate to cmr.sit.earthdata.nasa.gov/stac/ALL/search
11. Observe a 404 error
12. Navigate to cmr.sit.earthdata.nasa.gov/stac/ALL/collections/Test%201_1.2/items
13. Observe a 404 error
14. Navigate to cmr.sit.earthdata.nasa.gov/stac/ALL/collections?q=amazonia&limit=12
15. Observe collection with ID: AMZ1-WFI-L4-SR-1_NA
16. Observe link within that collection of rel=items with href=https://data.inpe.br/bdc/stac/v1/collections/AMZ1-WFI-L4-SR-1/items

Attachments

Radian Earth integrations:

Checklist

• I have added automated tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 93.33333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 88.39%. Comparing base (a5aff1b) to head (e727079).

Files with missing lines	Patch %	Lines
src/routes/healthcheck.ts	0.00%	2 Missing ⚠️
src/domains/collections.ts	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #379      +/-   ##
==========================================
+ Coverage   88.22%   88.39%   +0.17%     
==========================================
  Files          24       24              
  Lines        1180     1215      +35     
  Branches      261      269       +8     
==========================================
+ Hits         1041     1074      +33     
- Misses        139      141       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles

[eudoroolivares2016]

We could combine the logic in here within the existing validateProvider and just add another branch for the else conditional if (providerId == ALL_PROVIDER.toString()) { next( new ItemNotFound(This operation is not allowed for the ${ALL_PROVIDER.toString()} Catalog.) ); }
but, I also see the validity in keeping it separate because otherwise it does get a bit unruly. That said it does mean we are passing two very similar validations across the routes

[doug-newman-nasa]

ValidateProvider applies to all routes (except two health routes). ValidateNotAllProvider only applies to some (we do not allow search routes and item routes for the ALL provider) so I don't think we can combine. . This is why I separated them. I think validateProvider is not necessary (see https://bugs.earthdata.nasa.gov/browse/CMR-10264) so we would probably remove that in the future. validateProvider checks to see if the provider exists via a call to CMR and then, if it does, adds it to the request. This decreases the efficiency of a valid call. An invalid provider should be caught at the graphQL call instead imho.

[dmistry1]

Here we can simplify the condition by directly comparing mergedQuery.provider to ALL_PROVIDER

Suggested change

	if ("provider" in mergedQuery && mergedQuery.provider == ALL_PROVIDER)
	if (mergedQuery.provider === ALL_PROVIDER) {
	delete mergedQuery.provider;
	}

[doug-newman-nasa]

If I do that (which I did originally) I get the following error:
src/routes/catalog.ts:122:21 - error TS2339: Property 'provider' does not exist on type '{}'.

[william-valencia]

Other than my latest comments. Looks pretty good. Will you be able to do a team PR and demo the functionality for us?

[eudoroolivares2016]

Sent a PM for how to potentially resolve the typing issue though that is a minor point. I tested this locally for a while and did the test in the PR and didn't see any issues