cdscan fails on grib control files

[chengzhuzhang]

It was reported by @tangq at E3SM-Project/e3sm-unified#75
I tried to reproduce the problem with data on Cori NERSC with cdms3.0.0 and 3.1.4. It turned out only the newer cdms3.1.4 had this problem.

[jasonb5]

Original issue:

cdscan has been used to merge multiple grib control files into a single file. However, it fails with the newer versions in the e3sm_unified. The follow errors appears with e3sm_unified_1.2.6 and 1.3.0.

This is likely an issue with newer cdscan version. An older version from uvcdat 2.4.1 on the other machine works fine.

The files to reproduce this issue is at cori:/global/cscratch1/sd/tang30/ECMWF_Interim/download/2014

[tang30@cori06 2014]$ cdscan -x tst.xml *.ctl
Finding common directory ...
Common directory:
Scanning files ...
UVTQPS-20140101_T255.grib.ctl
Setting reference time units to
Traceback (most recent call last):
File "/global/cfs/cdirs/e3sm/software/anaconda_envs/base/envs/e3sm_unified_1.2.6/bin/cdscan", line 1840, in
main(sys.argv)
File "/global/cfs/cdirs/e3sm/software/anaconda_envs/base/envs/e3sm_unified_1.2.6/bin/cdscan", line 1282, in main
timeIsLinear = (referenceTime[0].lower().split() in
IndexError: string index out of range

[forsyth2]

@chengzhuzhang @tangq /global/cscratch1/sd/tang30/ECMWF_Interim/download/2014 doesn't seem to exist on Cori. Is there another set of files I can use to reproduce the error? Thanks!

[chengzhuzhang]

Hi Jason, thank you for getting to it! I don't have a copy of the data. @tangq should have a copy somewhere on cori...

[tangq]

I thought the files were not needed any more.

I copied some grib control files to the same directory.

[chengzhuzhang]

@chengzhuzhang @tangq /global/cscratch1/sd/tang30/ECMWF_Interim/download/2014 doesn't seem to exist on Cori. Is there another set of files I can use to reproduce the error? Thanks!
Thank you @forsyth2 !

[forsyth2]

@jasonb5 I put vartime.units in a list (https://github.com/CDAT/cdms/blob/master/Lib/cdscan.py#L1280) but that leads to the following error (line numbers are off from master because of print statements I added):

CDMS error: Error on relative units conversion, string = 

Traceback (most recent call last):
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/bin/cdscan", line 1846, in <module>
    main(sys.argv)
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/bin/cdscan", line 1311, in main
    startindex = timeindex(
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/bin/cdscan", line 269, in timeindex
    tval = cdtime.reltime(value, units)
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/lib/python3.8/site-packages/cdtime/compreltime.py", line 254, in reltime
    return _cdtime.reltime(value, units)
ValueError: Invalid relative time units

So, I tried putting vartime.units in a list for the timeindex call (https://github.com/CDAT/cdms/blob/master/Lib/cdscan.py#L1309):

Traceback (most recent call last):
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/bin/cdscan", line 1846, in <module>
    main(sys.argv)
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/bin/cdscan", line 1311, in main
    startindex = timeindex(
  File "/global/homes/f/forsyth/.conda/envs/test-cdms2/bin/cdscan", line 268, in timeindex
    if units.find(" as ") == -1:
AttributeError: 'list' object has no attribute 'find'

So, I'm a bit unclear as to what type is expected for vartime.units. It seems like it should be a list for timeIsLinear (https://github.com/CDAT/cdms/blob/master/Lib/cdscan.py#L1284) but something else for timeindex (https://github.com/CDAT/cdms/blob/master/Lib/cdscan.py#L267).

In fact, the issue may actually be in cdtime, not cdms, but I'm not entirely sure.

[forsyth2]

The units appear to be an empty string. Is that why we'd get Invalid relative time units? I still get that error if I change the string.

[jasonb5]

@forsyth2 vartime.units should be a string and will fail if it's empty. That's interesting though that setting it doesn't work.

[chengzhuzhang]

Have to go back to use an old env to do this function again yesterday. I thought about this issue when Sasha and Ghaleb brought up backward compatibility at today's meeting. Just to pump up the issue and have it under the radar.