Data Providers Guide
This guide is for data custodians providing data that will be shown on NationalMap.
- Overview
- Paths to putting a data set on the NationalMap
- Trying your data set in NationalMap
- Cross-origin resource sharing (CORS)
- Performance
- Adding a data set via Data.gov.au
- Adding a data set to a state or territory group
- Adding a data set to National Data Sets or elsewhere
TerriaJS, the software that drives NationalMap, can display data from a wide range of data sources. The key principles are:
- data is loaded directly from your data source into the user's web browser. NationalMap does not host any data itself.
- metadata comes from your data source, but can be overridden in a data source configuration file.
- the better the metadata at your service end, the easier it is to display in NationalMap.
There are three main types:
- "web" services: 1 URL points to one data set served via a web service, such as WMS, WFS, ArcGIS MapServer, etc.
- "catalogue group" services: 1 URL points to a collection of data sets, such as through WMS GetCapabilities or CKAN.
- "file" services: 1 URL points to a file which is completely downloaded into the browser by HTTP and then displayed.
There are three main paths for getting a data set on to the NationalMap:
- Add the data set to data.gov.au, following conventions defined below. Such data sets will appear on the NationalMap in the Data.gov.au section of the Data Catalogue.
- Follow a process specific to each state or territory, described below, to add the data set to a state or territory group in the Data Catalogue.
- Request that the data set be added somewhere else in the Data Catalogue, such as to the curated National Data Sets section or as a new top-level group, using the process described below. The Department of Communications has final authority over the organization and contents of the Data Catalogue.
Whichever path you choose, the first step is try your data set in NationalMap and verify that it works well. NationalMap makes it easy in most cases to try out new data sets without any involvement from the Department of Communications or the NationalMap administrators.
The easiest way to try a data set in NationalMap is via the "Add Data" feature.
The following types of web services are supported by entering a URL for the web service in the Add Data panel. Click the name of a web service below for details about the features and limitations of NationalMap's support for each type of web service.
- Web Map Service (WMS)
- Web Feature Service (WFS)
- Web Map Tile Service (WMTS)
- Esri ArcGIS Server
- OpenStreetMap-like Server
After you enter a URL and click the Add button, the web service will be added to the Data Catalogue under the User-Added Data group.
You may then enable one or more data sets by checking the box next to it in the Data Catalogue. A data set that works well in this way is straightforward to add to the official NationalMap data catalogue.
In addition to web services, data sets can be added to NationalMap as individual files that are fully downloaded by the web browser and then displayed. The following types of files are supported. Click the name of a file type for details about the features and limitations of NationalMap's support for each.
- Keyhole Markup Language (KML), including KMZ
- GeoJSON
- Cesium Language (CZML)
- Comma-Separated Values (CSV)
- GPS Exchange Format (GPX)
- Others such as Esri Shapefiles
A data file can be tested in NationalMap by:
- hosting it on a web server and entering its URL in the Add Data panel.
- clicking the Browse to select a local data file to upload link on the Add Data panel and selecting the file from your local system.
- Dragging the file from your local system onto the map.
Please note that only the first option will work in Internet Explorer 9. Internet Explorer 10+ and any recent version of Chrome, Firefox, or Safari will support all three options.
Because these kinds of data files must be loaded completely before they can be shown on the map, it is important that their size be reasonable. We recommend that single-file data sets be less than a couple of megabytes in size. If you normally use the 3D viewer (the default on systems that support it), we recommend you try your data set in the 2D view as well, because it is often slower with large data sets.
For data sets that are too large or slow as single files, consider hosting them using a WMS server. Data.gov.au can do this for you very easily.
NationalMap supports some additional data set types that cannot be added with Add Data. If you have data in one of these formats, you will need to write a TerriaJS Catalog File to try the data set in NationalMap.
Experiencing difficulty adding your data set to NationalMap? The first thing to try is opening your web browser's debug console. You can access it as follows:
- Google Chrome - Hamburger menu in the top right, More tools, JavaScript console
- Mozilla Firefox - Hamburger menu in the top right, Developer, Web console
- Internet Explorer 11 - Gear button in the top right, F12 Developer Tools, Console (at the top of the developer tools window that appears)
Then, reload the page and try adding your data set again. A message in the console may give you a hint about what is going wrong. It may also be helpful to use your web browser's Network debugging tools to inspect the requests that NationalMap is making to your server and the responses that it is getting back.
By far the most common problem when attempting to add data to NationalMap is a CORS failure. See below for information about CORS.
If you're stuck, contact [email protected] for assistance.
NationalMap is a web application, so it must play by the rules of the web. One of those rules is the restrictions that web browsers place on sharing content across origins. For example, in the absence of special configuration, a web application running on one domain, such as nationalmap.gov.au, is not permitted to access resources hosted on another domain, such as data.gov.au. NationalMap will display an error when attempting to access such a resource, and your web browser's console will display a message indicating a Cross-Origin Resource Sharing failure.
The "special configuration" mentioned above is a change to the HTTP headers returned by the server hosting the data set. The server should include this HTTP header, which gives web applications running on other domains permission to use the resource:
Access-Control-Allow-Origin: *
Instructions for configuring a large variety of web servers to include this header can be found at enablecors.org.
Including this header in the response is safe and desirable for all requests for public resources. You can learn more about CORS from the Mozilla Developer Network.
[add info about testing performance here. Details about tuning specific types of servers should be added to the TerriaJS page about the type of server]
[Also mention caching]