Sub-daily budget failed

[Kutcha7]

Hi everyone,
I tried to start a simulation and got the error sub daily budget failed. I checked the log file for NaNs or anything that could lead me to the error but I can't really find the source. Note that I only get this error when I use the opt executable. When I use the dbg it stops after initialising the met cycle and the error I get is related to "call_sdh5_open_f".

I've attached my logfile and the met header. I appreciate any feedback and happy to share more files!

1
/sci/labs/efratshef/irobk/Yatir/Future/gfdl-esm4/Weather/ISIMP_Israel_gfdl-esm4_ssp585_
2 2 0.5 0.5 35.065668 31.342226
13
'hgt' 'tmp' 'pres' 'sh' 'ugrd' 'vgrd' 'prate' 'dlwrf' 'nbdsf' 'nddsf' 'vbdsf' 'vddsf' 'co2'
27 3600 3600 3600 3600 3600 3600 3600 3600 3600 3600 3600 3600
4 1 1 1 1 1 0 1 1 1 1 1 1

Thanks!
Katja
output.txt

[mdietze]

Personally, I'd prioritize the error that shows up under debug model. What's the full trace of the error? I don't know what call_sdh5_open_f is but I'm guessing a low level HDF5 function (i.e. not our call) so you need to know where it's being called from and whether the file being open is present, valid, complete, and has the correct units. Misread met data could easily be the source of your downstream budget issue.

[mpaiao]

Most met driver issues typically pop up during sunrise and sunset, because this one is happening at midnight on the second day, so my hunch is that this is somehow a bug related to phenology. I see you started the run in January, which would be winter at your site, as a suggestion, maybe try setting the initial time in May or June. If the model goes through and keeps running, it may be indication of an issue with phenology.

[mpaiao]

But to @mdietze's point, met drivers are often the source of crashes. The main cause of problems is when IMETAVG is not properly set for the site, so do double check that variable too. It's on my to do list, but eventually I want to migrate IMETAVG to the met driver header (so all met driver info is in a single place), and after that I want to implement a refraction correction for sunset/twilight that should reduce/eliminate the problems, I have a prototype of it in R, just need to migrate it to ED2.

[Kutcha7]

Hi Mike and Marcos,
thanks for the feedback so far! This problem only occurred when I try to ran an initialised vegetation simulation with future climate data.

• I checked if it makes a difference to start the simulation in May -> I get the same error
• I checked the met forcing again for anything unusual and compared them with the historical data that I used for the potential vegetation simulation -> I couldn't find any NA's or unusual values (I implemented many checks when I created the data so it shouldn't have any faulty values)
• I tried to run the initialised vegetation simulation with the historical met forcings that worked for the potential vegetation simulation, and I got the same error

So my hunch is that the error is related to the .pss .css file. I'm not really sure what to look for. Do you have any suggestions?

Thanks again!

[Kutcha7]

But to @mdietze's point, met drivers are often the source of crashes. The main cause of problems is when IMETAVG is not properly set for the site, so do double check that variable too. It's on my to do list, but eventually I want to migrate IMETAVG to the met driver header (so all met driver info is in a single place), and after that I want to implement a refraction correction for sunset/twilight that should reduce/eliminate the problems, I have a prototype of it in R, just need to migrate it to ED2.

I set IMETAVG to 1 so it should be fine right?

[Kutcha7]

I still haven't been able to run it and get a sub-daily budget fail related to carbon budget.

So far I have:

• replaced/played around with some of the inputs (vegetation parameters, met forcings, .pss file)
• changed the disturbance type (trk)
• made the model start at different times of the year
• printed out all carbon related outputs in the log file

but I can't find the source....

Does anyone have any suggestions what else I could try?

Attached my log file, pss and css (as csv files).

Any help is appreciated!

output_StartMay15_withCpools.txt
Yat.lat31.342226lon35.065668css.csv
Yat.lat31.342226lon35.065668pss.csv

[mpaiao]

@Kutcha7 regarding the IMETAVG, it depends on what the time stamp means on your particular meteorological driver. If the time stamp corresponds to the end of the averaging period, then yes, it should be 1.

Your error is unlikely to be caused by IMETAVG, though. I see that you are only using PFT 7, which is an evergreen PFT, so I doubt the issue is phenology. Because the crash is happening on the second day, I think it may be a lingering problem with the initial conditions that is only checked on the second day (if I remember correctly, the CARBON block is only checked at daily time steps).

I noticed in your css file that the "cohort" column has repeated letters for multiple cohorts in the same patch. I would try to keep them unique within each patch, but I don't remember if this would cause problems in the initialisation. If that doesn't solve the problem, try running the model with IED_INIT_MODE=0 (near bare ground). This will ignore the pss/css files and start with a minimum number of plants. If the model runs fine, then I think it's safe to say that the issue is indeed in the pss/css files.

[Kutcha7]

I made several adjustments to n and let the model create cohorts itself. First it just resulted in the model crashing a few years later but I adjusted it further which seems to have fixed the problem. I have a full model run now! Thank you so much!

[mpaiao]

Great to hear it worked at last. Out of curiosity, which adjustments did you do? I am asking because it could be useful for future users. Thanks!

[mpaiao]

By the way, I never replied your question on trees that are barely alive. Currently ED does not have an explicit way of telling which trees are healthy and which ones are badly damaged or otherwise in poor health, though it would be a very nice addition to the model for anyone interested in implementing it ;).

If you do add them, just beware that they will be just like the healthy ones.

[Kutcha7]

Agree I also think it would be a nice addition. For now I have other priorities (adapting ED to drylands) but I will keep it in mind :)

After you asked me about the sample design, I asked the people who did the survey again and they told me that they had subplots of 8m^2. So I adjust n in the .css homogeneously to 1/8. It still resulted in a C budget fail.. so I assumed there might have been a size dependent sample effort and implemented that all trees with dbh < 10 -> n = 1/4. So here, I only adjust n and did not change the number of patches. With these settings I got a whole simulation run without problems.
However, then I let the same settings run with climate projections from a different climate model (fairly similar but more rain), and it resulted in a C budget fail after 4 years of simulation. So this time, I implemented more patches and changed n back to 1/8 for all entries. And I got a full run again.. Not sure what exactly causes this bug but it's definitely a strange one (and not intuitive to solve).

[Kutcha7]

Actually depending on climate projection it still keeps failing after a few years of simulation.. Whenever I decrease the density or include more patches, it runs for a little bit longer. I'm not sure what the best practice here is. Or what really causes the carbon budget fail.
Could it be related to sth else, such as decrease in LAI, critical height? The PFT is not properly calibrated yet so many individuals suffer under the harsh climate conditions there.