Extracting useful metadata for validating load requests versus model output submissions

[annakrystalli]

As the hubmeta files will be used for both validating requests to load data as well as validating model output submissions, I just wanted to clarify I've understood the different requirements (which affect the level of parsing of metadata) in each case and outline some limitations I can foresee:

Validating load data requests

• required and optional argument values aren't significant in this case so task id parameters and output types (and type ids) can be flattened to a single vector of values that arguments can be validated against.
• In the case of optional values however, the metadata cannot guarantee a specific combination of argument values (i.e. a specific combination of origin_date, location, horizon and output type and type_id will return data and searching the hub is the only way to extract such data availability. Is this a correct understanding and is that a problem? Do we just return No data found which is easy but not so helpful for the user?

Validating model output submissions

Here required and optional become important and contents of submissions need to be validated separately against required (for which entries must be present) and optional (for which entries must match the optional values.

In any case, I'm thinking of writing smaller helper functions to extract individual argument values in different formats (flat for data loading validation vs semi structured with optional & required fields for submission validation) and appending unit and format metadata as attributes where available.

Here's an example:

x <- jsonlite::read_json(
    system.file("hub_1/hubmeta.json",
                package = "hubUtils"))[[1]]

hubUtils:::get_task_id_params(x, "location", flatten = TRUE)
#>  [1] "01" "02" "04" "05" "06" "08" "09" "10" "11" "12" "13" "15" "16" "17" "18"
#> [16] "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30" "31" "32" "33"
#> [31] "34" "35" "36" "37" "38" "39" "40" "41" "42" "44" "45" "46" "47" "48" "49"
#> [46] "50" "51" "53" "54" "55" "56" "72" "78" "US"
hubUtils:::get_task_id_params(x, "location", flatten = FALSE)
#> $required
#> NULL
#> 
#> $optional
#>  [1] "01" "02" "04" "05" "06" "08" "09" "10" "11" "12" "13" "15" "16" "17" "18"
#> [16] "19" "20" "21" "22" "23" "24" "25" "26" "27" "28" "29" "30" "31" "32" "33"
#> [31] "34" "35" "36" "37" "38" "39" "40" "41" "42" "44" "45" "46" "47" "48" "49"
#> [46] "50" "51" "53" "54" "55" "56" "72" "78" "US"

^{Created on 2022-10-03 with reprex v2.0.2}

Thoughts welcome!

[elray1]

This all looks good to me. In instances where no data are found for a particular query, I think I'd vote for returning a data frame with zero rows.