aodn_wave_dm results differ with tos7

[jonescc]

Running this harvester on 9-nec-hob gives different results (timestamps) to 6-nec-hob:

--- /home/craigj/git-repos/aodn/utilities/pipeline/talend_test/harvest_results/6-nec-hob.emii.org.au/aodn_wave_dm/aodn_wave_dm/aodn_wave_dm_timeseries_map.csv
+++ /home/craigj/git-repos/aodn/utilities/pipeline/talend_test/harvest_results/9-nec-hob.emii.org.au/aodn_wave_dm/aodn_wave_dm/aodn_wave_dm_timeseries_map.csv
@@ -1,5 +1,5 @@
 file_id,url,size,institution,platform_code,site_code,site_name,instrument_type,instrument_model,time_coverage_start,time_coverage_end,platform_time_start,platform_time_end,latitude_min,latitude_max,longitude_min,longitude_max,depth,geom
-1,NSW-OEH/Manly_Hydraulics_Laboratory/Wave/Byron_Bay/IMOS_ANMN-NSW_W_19930721T010000Z_WAVEBYB_WAVERIDER_FV01_END-19931110T210000Z.nc,223095,Manly Hydraulics Laboratory (MHL),WAVEBYB,,Byron Bay,,,1993-07-21 01:00:00,1993-11-10 21:00:00,1993-07-20 15:00:00,1993-11-10 10:00:00,-28.67944,-28.67944,-28.67944,153.73667,,0101000020E610000038BEF6CC923763402198A3C7EFAD3CC0
-3,Department_of_Transport-Western_Australia/Waverider_Buoys/DELAYED/Cape_Naturaliste_02/NAT46/DOT-WA_W_20111231T160000Z_NAT46_WAVERIDER_FV01_END-20121230T211700Z.nc,1422895,"Department of Transport (DoT), Western Australian Government",NAT46,NAT46,Cape Naturaliste 02,Directional,Directional Waverider MK III,2011-12-31 16:00:00,2012-12-30 21:17:00,2011-12-31 05:00:00,2012-12-30 10:17:00,-33.534722,-33.534722,-33.534722,114.764444,,0101000020E6100000E6E786A6ECB05C40D2393FC571C440C0
-4,Bureau_of_Meteorology/Waverider_Buoys/DELAYED/Cape_Sorell/BOM_W_20041231T140000Z_SORELL_WAVERIDER_FV01_END-20051231T133000Z.nc,1376607,Australian Bureau of Meteorology (BoM),55026,SORELL,Cape Sorell,Non-Directional,Waverider FL,2004-12-31 14:00:00,2005-12-31 13:30:00,2004-12-31 03:00:00,2005-12-31 02:30:00,-42.12,-42.12,-42.12,145.03,,0101000020E6100000295C8FC2F52062408FC2F5285C0F45C0
-5,Department_of_Environment_and_Science-Queensland/Waverider_Buoys/DELAYED/Gladstone/DES-QLD_W_20141231T140000Z_Gladstone_WAVERIDER_FV01_END-20151231T123000Z.nc,1101340,"Department of Environment and Science (DES), Queensland Government",55038,Gladstone,Gladstone,Directional,Waverider Mk3 0.9 Cu,2014-12-31 14:00:00,2015-12-31 12:30:00,2014-12-31 03:00:00,2015-12-31 01:30:00,-23.89561,-23.89561,-23.89561,151.502383,,0101000020E61000002062838513F0624077F86BB246E537C0
+1,NSW-OEH/Manly_Hydraulics_Laboratory/Wave/Byron_Bay/IMOS_ANMN-NSW_W_19930721T010000Z_WAVEBYB_WAVERIDER_FV01_END-19931110T210000Z.nc,223095,Manly Hydraulics Laboratory (MHL),WAVEBYB,,Byron Bay,,,1993-07-21 01:00:00,1993-11-10 21:00:00,1993-07-21 01:00:00,1993-11-10 21:00:00,-28.67944,-28.67944,-28.67944,153.73667,,0101000020E610000038BEF6CC923763402198A3C7EFAD3CC0
+3,Department_of_Transport-Western_Australia/Waverider_Buoys/DELAYED/Cape_Naturaliste_02/NAT46/DOT-WA_W_20111231T160000Z_NAT46_WAVERIDER_FV01_END-20121230T211700Z.nc,1422895,"Department of Transport (DoT), Western Australian Government",NAT46,NAT46,Cape Naturaliste 02,Directional,Directional Waverider MK III,2011-12-31 16:00:00,2012-12-30 21:17:00,2011-12-31 16:00:00,2012-12-30 21:17:00,-33.534722,-33.534722,-33.534722,114.764444,,0101000020E6100000E6E786A6ECB05C40D2393FC571C440C0
+4,Bureau_of_Meteorology/Waverider_Buoys/DELAYED/Cape_Sorell/BOM_W_20041231T140000Z_SORELL_WAVERIDER_FV01_END-20051231T133000Z.nc,1376607,Australian Bureau of Meteorology (BoM),55026,SORELL,Cape Sorell,Non-Directional,Waverider FL,2004-12-31 14:00:00,2005-12-31 13:30:00,2004-12-31 14:00:00,2005-12-31 13:30:00,-42.12,-42.12,-42.12,145.03,,0101000020E6100000295C8FC2F52062408FC2F5285C0F45C0
+5,Department_of_Environment_and_Science-Queensland/Waverider_Buoys/DELAYED/Gladstone/DES-QLD_W_20141231T140000Z_Gladstone_WAVERIDER_FV01_END-20151231T123000Z.nc,1101340,"Department of Environment and Science (DES), Queensland Government",55038,Gladstone,Gladstone,Directional,Waverider Mk3 0.9 Cu,2014-12-31 14:00:00,2015-12-31 12:30:00,2014-12-31 14:00:00,2015-12-31 12:30:00,-23.89561,-23.89561,-23.89561,151.502383,,0101000020E61000002062838513F0624077F86BB246E537C0

[ggalibert]

@bpasquer can you please have a look at this?

Looks like the following dataset NSW-OEH/Manly_Hydraulics_Laboratory/Wave/Byron_Bay/IMOS_ANMN-NSW_W_19930721T010000Z_WAVEBYB_WAVERIDER_FV01_END-19931110T210000Z.nc is not harvested properly.

[jonescc]

Looking at this with @bpasquer it looks like the 9-nec-hob result is actually the correct one. Need to check what the situation is in production.

Looks like we are extracting values with AT TIME ZONE 'UTC' in the harvester and inserting them into a TIMESTAMP WITH TIME ZONE column. As discussed elsewhere we should probably stick with values WITH TIME ZONE in the tables and when processing in the harvester and convert to 'UTC' in views used by geoserver to work around time zone processing issues there.

[bpasquer]

Issue is present in Prod. Will open an issue in content for reference until TOS 7 is in prod and we reprocess the impacted file(s).