Running the test case stops at the Augur Module

[sg19dev]

Hello,

I am trying to run the test case for Reeds. If I run it to 2030, it shows up the following errors in the augurs log:

ERROR:gdxpds.special:Unable to convert numpy special values to GDX special values.num_dims: 3, dataframe:
r rr trtype Value
0 p60 p61 AC c_bool(False)
1 p60 p62 AC c_bool(False)
2 p60 p63 AC c_bool(False)
3 p61 p60 AC c_bool(False)
4 p61 p62 AC c_bool(False)
5 p61 p63 AC c_bool(False)
6 p61 p64 AC c_bool(False)
7 p61 p65 AC c_bool(False)
8 p62 p60 AC c_bool(False)
9 p62 p61 AC c_bool(False)
10 p63 p60 AC c_bool(False)
11 p63 p61 AC c_bool(False)
12 p63 p64 AC c_bool(False)
13 p64 p61 AC c_bool(False)
14 p64 p63 AC c_bool(False)
15 p64 p65 AC c_bool(False)
16 p64 p67 AC c_bool(False)
17 p65 p61 AC c_bool(False)
18 p65 p64 AC c_bool(False)
19 p65 p67 AC c_bool(False)
20 p67 p64 AC c_bool(False)
21 p67 p65 AC c_bool(False)
ERROR:gdxpds.gdx:Unable to write routes, , Parameter, 22 records, 3 dims ['r', 'rr', 'trtype'], loaded to ReEDS_Augur\augur_data\osprey_inputs_2020.gdx
ERROR:main:2020
ERROR:main:Traceback (most recent call last):
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py", line 166, in _na_arithmetic_op
result = func(left, right)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\computation\expressions.py", line 239, in evaluate
return _evaluate(op, op_str, a, b) # type: ignore[misc]
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\computation\expressions.py", line 128, in _evaluate_numexpr
result = _evaluate_standard(op, op_str, a, b)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\computation\expressions.py", line 69, in _evaluate_standard
return op(a, b)
TypeError: unsupported operand type(s) for -: 'c_bool' and 'float'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\Saahil2\Documents\GitHub\ReEDS_OpenAccess_manual\runs\v20230619_130858_ercot_seq\Augur.py", line 180, in
ReEDS_Augur(next_year, prev_year, scen)
File "C:\Users\Saahil2\Documents\GitHub\ReEDS_OpenAccess_manual\runs\v20230619_130858_ercot_seq\Augur.py", line 57, in ReEDS_Augur
reeds_inputs = prep_data()
File "C:\Users\Saahil2\Documents\GitHub\ReEDS_OpenAccess_manual\runs\v20230619_130858_ercot_seq\ReEDS_Augur\A_prep_data.py", line 229, in prep_data
gdx.write(
File "C:\Users\Saahil2\anaconda3\lib\site-packages\gdxpds\gdx.py", line 294, in write
symbol.write(index=i)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\gdxpds\gdx.py", line 1075, in write
to_write = self.dataframe.copy() if (self.data_type in (GamsDataType.Set, GamsDataType.Alias)) else special.convert_np_to_gdx_svs(self.dataframe, self.num_dims)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\gdxpds\special.py", line 101, in convert_np_to_gdx_svs
values[(values - NUMPY_SPECIAL_VALUES[-1]).abs() < NUMPY_SPECIAL_VALUES[-1]] = SPECIAL_VALUES[4]
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\ops\common.py", line 69, in new_method
return method(self, other)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\arraylike.py", line 100, in sub
return self._arith_method(other, operator.sub)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\frame.py", line 6866, in _arith_method
new_data = self._dispatch_frame_op(other, op, axis=axis)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\frame.py", line 6893, in _dispatch_frame_op
bm = self._mgr.apply(array_op, right=right)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\internals\managers.py", line 325, in apply
applied = b.apply(f, **kwargs)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\internals\blocks.py", line 381, in apply
result = func(self.values, **kwargs)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py", line 224, in arithmetic_op
res_values = _na_arithmetic_op(left, right, op)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py", line 173, in _na_arithmetic_op
result = _masked_arith_op(left, right, op)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\core\ops\array_ops.py", line 131, in _masked_arith_op
result[mask] = op(xrav[mask], y)
TypeError: unsupported operand type(s) for -: 'c_bool' and 'float'

The Gamslog stops at:

*** Status: Normal completion
--- Job d_solveoneyear.gms Stop 06/17/23 20:20:29 elapsed 0:00:14.955
C:\Users\Saahil2\Documents\GitHub\ReEDS_OpenAccess_manual\runs\v20230617_201419_ercot_seq\ReEDS_Augur\utility\hourlyprofiles.py:634: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().
vre_gen
Preparing data for Osprey...
marg_vre_mw_curt set to 1000.0
marg_vre_mw_cc set to 4047
Adjusting net load by Canadian exports given Sw_Canada = 1

The outputs folder only contains one folder called the tc_phaseout_data

If I run the model till 2018, everything runs fine and I get the output graphs in that run.

Could you please help me with this

Thank you very much,
Saahil

[wesleyjcole]

This looks like an error with the gdxpds package, which is the package used to translate gdx files into Pandas and vice versa. This package gets used by the Augur module, which doesn't start to get used until the 2020 solve year, which is why you can solve to 2018 with no issue. I suspect that you are using versions of GAMS and Python that are incompatible with each other when interacting via gdxpds, but I don't know. If you use GAMS 34.3 and the environment.yml file in the repo, does that resolve the issue?

[sg19dev]

Thank you very much for your response,

I have been using GAMS version 25.1 and Python 3.9.7. I will try to get access to GAMS 34.3 and see if that works. In the meantime, are there any other steps/solutions I can try that may be helpful.

[wesleyjcole]

The only workaround that I can think of is to dump the gdx outputs to csv files (see e_report_dump.gms for an example), and then read in those csv files into the Augur module rather than the gdx files. That would take some work, but I don't have any more straightforward solutions.

It might be possible to find versions of gdxpds, GAMS, and Python that are compatible with one another, but I don't have any insight for doing that.

[sg19dev]

Thanks for the suggestions. I will try them out and see how it goes.

[sg19dev]

Hello, thanks for your help

I changed the version of gdxpds and I am able to run the test and small cases now.

I have another question- I am trying to input Canadian regions in the GSW region and region level to get some results for Canada. The run stops at LDC_prep.py. Here is the error log:

plantcostprep.py completed successfully
Starting climateprep.py
All climate switches are off. We stopped climate change!
Finished climateprep.py
Starting LDC_prep.py
Traceback (most recent call last):
File "C:\Users\Saahil2\Documents\GitHub\ReEDS_OpenAccess_manual2\input_processing\LDC_prep.py", line 652, in
hourly_prep(basedir, inputs_case)
File "C:\Users\Saahil2\Documents\GitHub\ReEDS_OpenAccess_manual2\input_processing\LDC_prep.py", line 171, in hourly_prep
pd.read_csv(os.path.join(inputs_case, 'valid_ba_list.csv'), squeeze=True, header=None)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\util_decorators.py", line 311, in wrapper
return func(*args, **kwargs)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\io\parsers\readers.py", line 586, in read_csv
return _read(filepath_or_buffer, kwds)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\io\parsers\readers.py", line 482, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\io\parsers\readers.py", line 811, in init
self._engine = self._make_engine(self.engine)
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\io\parsers\readers.py", line 1040, in _make_engine
return mapping[engine](self.f, **self.options) # type: ignore[call-arg]
File "C:\Users\Saahil2\anaconda3\lib\site-packages\pandas\io\parsers\c_parser_wrapper.py", line 69, in init
self._reader = parsers.TextReader(self.handles.handle, **kwds)
File "pandas_libs\parsers.pyx", line 549, in pandas._libs.parsers.TextReader.cinit
pandas.errors.EmptyDataError: No columns to parse from file

What steps could I try to correct this and are there other switches that need to be changed for the Canadian context.

Thank you very much,
Regards

[stuartcohen8]

While there are some pieces of code in the open access version of ReEDS that are meant to enable the model to run for Canada, that feature is not fully functional. There are missing data pieces for one, most importantly a power plant database, because the one used for the NARIS study was a proprietary dataset. It would take substantial data and code development to get this feature working again. The good news is that we are working internally to rebuild this functionality with open data so that we can make it available to the public. I can't give you an exact timeline, but my hope would be for a release of that feature in 2024.

[sg19dev]

Thank you for letting me know. Looking forward to the new version.

Regards