Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

implement ACTRIS-EBAS reader #46

Open
jgriesfeller opened this issue Aug 6, 2024 · 1 comment · May be fixed by #48
Open

implement ACTRIS-EBAS reader #46

jgriesfeller opened this issue Aug 6, 2024 · 1 comment · May be fixed by #48

Comments

@jgriesfeller
Copy link
Member

Implement a reader for the ACTRIS EBAS data format via the web API:
Simple example file (sulphate_corrected @ Birkenes) is here:

netcdf NO0001R.20010101070000.20181210133000.filter_3pack.sulphate_corrected.aerosol.1y.1d.NO01L_f3p_d_001.NO01L_IC.lev2 {
dimensions:
	time = 365 ;
	metadata_time = 1 ;
	tbnds = 2 ;
	sulphate_corrected_qc_flags = 1 ;
variables:
	double time(time) ;
		time:standard_name = "time" ;
		time:long_name = "time of measurement" ;
		time:units = "days since 1900-01-01 00:00:00 UTC" ;
		time:axis = "T" ;
		time:calendar = "gregorian" ;
		time:bounds = "time_bnds" ;
		time:cell_methods = "mean" ;
	double time_bnds(time, tbnds) ;
		time_bnds:standard_name = "time" ;
		time_bnds:long_name = "time bounds for measurement" ;
		time_bnds:units = "days since 1900-01-01 00:00:00 UTC" ;
	double metadata_time(metadata_time) ;
		metadata_time:standard_name = "time" ;
		metadata_time:long_name = "time of ebas metadata intervals" ;
		metadata_time:units = "days since 1900-01-01 00:00:00 UTC" ;
		metadata_time:axis = "T" ;
		metadata_time:calendar = "gregorian" ;
		metadata_time:bounds = "metadata_time_bnds" ;
		metadata_time:cell_methods = "mean" ;
	double metadata_time_bnds(metadata_time, tbnds) ;
		metadata_time_bnds:standard_name = "time" ;
		metadata_time_bnds:long_name = "time bounds for ebas metadata intervals" ;
		metadata_time_bnds:units = "days since 1900-01-01 00:00:00 UTC" ;
		metadata_time_bnds:calendar = "gregorian" ;
	double sulphate_corrected(time) ;
		sulphate_corrected:_FillValue = NaN ;
		string sulphate_corrected:standard_name = "mass_concentration_of_sulphate_corrected_for_seaspray_expressed_as_sulphur_in_dry_aerosol_particles_in_air" ;
		sulphate_corrected:missing_value = NaN ;
		string sulphate_corrected:units = "ug S/m3" ;
		sulphate_corrected:ancillary_variables = "sulphate_corrected_qc sulphate_corrected_ebasmetadata" ;
		sulphate_corrected:cell_methods = "time: mean" ;
	string sulphate_corrected_ebasmetadata(metadata_time) ;
		sulphate_corrected_ebasmetadata:long_name = "ebas metadata for different time intervals; json encoded" ;
	int sulphate_corrected_qc(sulphate_corrected_qc_flags, time) ;
		sulphate_corrected_qc:_FillValue = 0 ;
		sulphate_corrected_qc:standard_name = "status_flag" ;
		sulphate_corrected_qc:missing_value = 0 ;
		sulphate_corrected_qc:units = "1" ;

// global attributes:
		:Conventions = "CF-1.7, ACDD-1.3" ;
		:featureType = "timeSeries" ;
		string :title = "Ground based in situ observations of sulphate_corrected at Birkenes (NO0001R) using filter_3pack" ;
		string :keywords = "Birkenes, NO0001R, CAMP, sulphate_corrected, aerosol, NILU, EMEP, mass_concentration_of_sulphate_corrected_for_seaspray_expressed_as_sulphur_in_dry_aerosol_particles_in_air" ;
		string :id = "NO0001R.20010101070000.20181210133000.filter_3pack.sulphate_corrected.aerosol.1y.1d.NO01L_f3p_d_001.NO01L_IC.lev2.nc" ;
		:naming_authority = "no.nilu.ebas" ;
		string :project = "CAMP, EMEP, NILU" ;
		:acknowledgement = "" ;
		string :license = "CAMP: Public open access. We encourage contacting data originators if substatial use of individual time series is planned (fair use data policy)., EMEP: Public open access. We encourage contacting data originators if substatial use of individual time series is planned (fair use data policy)., NILU: Public open access. We encourage contacting data originators if substatial use of individual time series is planned (fair use data policy)." ;
		string :summary = "Ground based in situ observations of sulphate_corrected at Birkenes (NO0001R) using filter_3pack. These measurements are gathered as a part of the following projects CAMP, EMEP, NILU and they are stored in the EBAS database (http://ebas.nilu.no/). Parameters measured are: sulphate_corrected in aerosol (mass_concentration_of_sulphate_corrected_for_seaspray_expressed_as_sulphur_in_dry_aerosol_particles_in_air)" ;
		:source = "surface observation" ;
		string :institution = "NO01L, Norwegian Institute for Air Research, NILU, Atmosphere and Climate Department, Instituttveien 18, 2007, Kjeller, Norway" ;
		:processing_level = "2: final (Physical parameters, aggregated (if needed), info on variability recommended, quality assured by human inspection. This is the typical EBAS data level meant for dissemination and long term preservation.)" ;
		:date_created = "2018-12-10T13:30:00 UTC" ;
		:date_metadata_modified = "2018-12-10T13:30:00 UTC" ;
		:creator_name = "Jan Erik Hanssen" ;
		:creator_type = "person" ;
		:creator_email = "" ;
		string :creator_institution = "\"Norwegian Institute for Air Research, Atmosphere and Climate Department, NILU\"" ;
		string :contributor_name = "Jan Erik Hanssen" ;
		:contributor_role = "data submitter" ;
		:publisher_type = "institution" ;
		:publisher_name = "NILU - Norwegian Institute for Air Research, ATMOS, EBAS" ;
		:publisher_institution = "NILU - Norwegian Institute for Air Research, ATMOS, EBAS" ;
		:publisher_email = "[email protected]" ;
		:publisher_url = "http://ebas.nilu.no/" ;
		:geospatial_bounds = "POINT Z (58.383333 8.25 190.0)" ;
		:geospatial_bounds_crs = "EPSG:4979" ;
		:geospatial_lat_min = 58.383333 ;
		:geospatial_lat_max = 58.383333 ;
		:geospatial_lon_min = 8.25 ;
		:geospatial_lon_max = 8.25 ;
		:geospatial_vertical_min = 190. ;
		:geospatial_vertical_max = 190. ;
		:geospatial_vertical_positive = "up" ;
		:time_coverage_start = "2001-01-01T07:00:00 UTC" ;
		:time_coverage_end = "2002-01-01T07:00:00 UTC" ;
		:time_coverage_duration = "P0001-00-00T00:00:00" ;
		:time_coverage_resolution = "P0000-00-01T00:00:00" ;
		:timezone = "UTC" ;
		string :ebas_metadata = "{\n    \"Data definition\": \"EBAS_1.1\", \n    \"Set type code\": \"TU\", \n    \"Timezone\": \"UTC\", \n    \"File name\": \"NO0001R.20010101070000.20181210133000.filter_3pack.sulphate_corrected.aerosol.1y.1d.NO01L_f3p_d_001.NO01L_IC.lev2.nc\", \n    \"File creation\": \"20190514064331\", \n    \"Startdate\": \"20010101070000\", \n    \"Revision date\": \"20181210133000\", \n    \"Statistics\": \"arithmetic mean\", \n    \"Data level\": \"2\", \n    \"Period code\": \"1y\", \n    \"Resolution code\": \"1d\", \n    \"Station code\": \"NO0001R\", \n    \"Platform code\": \"NO0001S\", \n    \"Station name\": \"Birkenes\", \n    \"Station WDCA-ID\": \"GAWANO__BIR\", \n    \"Station GAW-ID\": \"BIR\", \n    \"Station GAW-Name\": \"Birkenes Atmospheric Observatory\", \n    \"Station other IDs\": \"201(NILUDB)\", \n    \"Station land use\": \"Forest\", \n    \"Station setting\": \"Rural\", \n    \"Station GAW type\": \"R\", \n    \"Station WMO region\": \"6\", \n    \"Station latitude\": \"58.383333\", \n    \"Station longitude\": \"8.25\", \n    \"Station altitude\": \"190.0 m\", \n    \"Regime\": \"IMG\", \n    \"Component\": \"sulphate_corrected\", \n    \"Unit\": \"ug S/m3\", \n    \"Matrix\": \"aerosol\", \n    \"Laboratory code\": \"NO01L\", \n    \"Instrument type\": \"filter_3pack\", \n    \"Instrument name\": \"f3p_d_001\", \n    \"Method ref\": \"NO01L_IC\", \n    \"Organization\": \"NO01L, Norwegian Institute for Air Research, NILU, Atmosphere and Climate Department, Instituttveien 18, , 2007, Kjeller, Norway\", \n    \"Frameworks\": \"CAMP EMEP NILU\", \n    \"Originator\": \"Hanssen, Jan Erik, , Norwegian Institute for Air Research, NILU, Atmosphere and Climate Department, Instituttveien 18, , 2007, Kjeller, Norway\", \n    \"Submitter\": \"Hanssen, Jan Erik, , Norwegian Institute for Air Research, NILU, Atmosphere and Climate Department, Instituttveien 18, , 2007, Kjeller, Norway\"\n}" ;

Search API description is here:
https://prod-actris-md.nilu.no/index.html
@jgriesfeller
Copy link
Member Author

jgriesfeller commented Aug 28, 2024 •
edited
Loading

Results from the 3rd annual ACTRIS-Norway meeting:
Notes are here:, most of the stuff from there is also below

Links

General points:

  • The stability of the search API and the data distribution via thredds is unknown at this point. We need to hit it to find the limits.
  • Caching is necessary
  • Try big request first, if they don't work, try smaller ones, e.g. country by country. The search API can provide a list of countries.
  • start simple (aerocom variable vmro3) and increase complexity afterwards

Some detailed points:

  • file names have no meaning
  • variable names have no meaning; to get do a final selection of which variable to use, one has to interpret the attribute ebas_metadata. Usually that is a variable attribute, but it can also be a global attribute (as in the example above). Search order: first variable attributes, then global attributes
  • at this point the search for files to download has to be done using ACTRIS vocabulary, but the files themselves contain the old EBAS vocabulary, This will change by the end of the year, but we have to implement a reader now to find potential errors and needed extensions as we are the main user for now.
  • There's no way of searching for the matrix in the search API (called submatrix by the Nilu people). The search API will be extended at the next revision to support that (next year).
  • There's no way to distinguish between sulphate_total and sulplate_corrected components, since the ACTRIS vocabulary does not distiunguish between the two. The search API will be extended to make distinguishing possible (next year).

@jgriesfeller jgriesfeller linked a pull request Aug 29, 2024 that will close this issue
Draft
@jgriesfeller jgriesfeller linked a pull request Aug 29, 2024 that will close this issue
46 implement actris ebas reader #48
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

46 implement actris ebas reader metno/pyaro-readers
1 participant
@jgriesfeller