-
Notifications
You must be signed in to change notification settings - Fork 172
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create links for GOCART output instead of copying after forecast #1934
Comments
Was a UFS and/or GOCART issue created that can be linked here? Barring information to the contrary from the developers, it seems like the issue needs to be solved there. |
There was an older GOCART Issues that hopefully @lipan-NOAA or @rmontuoro can point you to - apologies Raffaele I lost the issue that you previously sent. |
UFS issue ufs-community/ufs-weather-model#1955 was created |
@JessicaMeixner-NOAA I just want to make sure I am understanding what you have said above. Are you asking for the aerosol history files from the forecast run (i.e., those in If I am misunderstanding and you mean that the previous forecast cycle aerosol history files be linked to the current forecast run, then that is straightforward and minor modification. Can you please provide me a bit more context and clarity? Thank you in advance. |
@HenryWinterbottom-NOAA here's some extra details which hopefully will be helpful: Currently what happens:
The desired path forward is instead:
The desired path forward for whatever reason did not work before. One reason it did not work was that there were some model updates and model settings that need to be set. @lipan-NOAA @bbakernoaa and @rmontuoro are aerosol experts to help with details. Tagging @WalterKolczynski-NOAA as well as I know we both looked at this before. He might be able to help fill in some gaps that I didn't. |
This can be fixed by adding in the flag |
@JessicaMeixner-NOAA I am not certain that request raised in this issue can be performed based on the current capabilities of the UFS weather model. As is currently done, the
In addition to the hanging symlink test above, I created empty The relevant contents of my
@bbakernoaa Are my conclusions what you'd expect or am I missing something else? |
@HenryWinterbottom-NOAA /s/N/g/R/g/global-workflow [develop ✚.1 …6] ❯❯❯ git diff ush/
diff --git i/ush/forecast_postdet.sh w/ush/forecast_postdet.sh
index 644a2180..c9eb9b2d 100755
--- i/ush/forecast_postdet.sh
+++ w/ush/forecast_postdet.sh
@@ -929,33 +929,19 @@ GOCART_rc() {
GOCART_postdet() {
echo "SUB ${FUNCNAME[0]}: Linking output data for GOCART"
+ local fhr vdate
for fhr in ${FV3_OUTPUT_FH}; do
- local vdate=$(date --utc -d "${current_cycle:0:8} ${current_cycle:8:2} + ${fhr} hours" +%Y%m%d%H)
-
- # Temporarily delete existing files due to noclobber in GOCART
- if [[ -e "${COM_CHEM_HISTORY}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4" ]]; then
- rm -f "${COM_CHEM_HISTORY}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4"
- fi
+ if (( fhr == 0 )); then continue; fi
+ vdate=$(date --utc -d "${current_cycle:0:8} ${current_cycle:8:2} + ${fhr} hours" +%Y%m%d%H)
- #To Do: Temporarily removing this as this will crash gocart, adding copy statement at the end
- #${NLN} "${COM_CHEM_HISTORY}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4" \
- # "${DATA}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4"
+ ${NLN} "${COM_CHEM_HISTORY}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4" \
+ "${DATA}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4"
done
}
GOCART_out() {
echo "SUB ${FUNCNAME[0]}: Copying output data for GOCART"
- # Copy gocart.inst_aod after the forecast is run (and successfull)
- # TO DO: this should be linked but there were issues where gocart was crashing if it was linked
- local fhr
- local vdate
- for fhr in ${FV3_OUTPUT_FH}; do
- if (( fhr == 0 )); then continue; fi
- vdate=$(date --utc -d "${current_cycle:0:8} ${current_cycle:8:2} + ${fhr} hours" +%Y%m%d%H)
- ${NCP} "${DATA}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4" \
- "${COM_CHEM_HISTORY}/gocart.inst_aod.${vdate:0:8}_${vdate:8:2}00z.nc4"
- done
}
CMEPS_postdet() {
This indicates to me that the GOCART component is unable to write via links. The component will have to resolve this issue for the workflow to be able to do what is required in this issue. |
@aerorahul Should we icebox or close this for now? |
icebox till the GOCART group can confirm this is resolved. The fix is very simple as indicated in the diff in this comment |
@junwang-noaa I'm running into this issues as well. Do you remember where to put this in the configuration? |
@bbakernoaa We used to add "Allow_Overwrite: .true." to AERO_HISTORY.rc to resolve the file clobbering issue. But this one looks different to me as it is the symbolic link that does not work. |
@lipan-NOAA could you help look into this? |
@JessicaMeixner-NOAA @bbakernoaa |
@aerorahul It was my understanding that this was already fixed in ufs-waether-model/GOCART, but @bbakernoaa and/or @lipan-NOAA would know more. |
Is that demonstrated? I have been unable to replicate the successes of @bbakernoaa or @lipan-NOAA |
@junwang-noaa found that by modifying AERO_HISTORY.rc, the problem was solved Create History List for Output####################################################################### VERSION: 1 To EXPID: ../$(gocart_output_directory)/gocart |
@lipan-NOAA |
I want to confirm that the symbolic link does not work and it will not be fixed as the fix will break GOES run. So the proposed method is to write to COM directory directly. |
For AQMv7 implementation, NCO asked to have all symbolic links removed. Forecast data must be written under the ./DATA (running) directory. At the end of forecast, a separate job was added to move the data to ./COM. NCO is open to have more discussion about this "new" requirement. @aerorahul EMC needs to discuss with NCO to reach an agreement. It might be impossible to follow this new requirement for large systems like RRFS and GFS/GEFS. |
tested it and Jun's modification works |
What new functionality do you need?
Would like to be able to link gocart diagnostic files to COM directory instead of copying them at the end like #1933 implements.
What are the requirements for the new functionality?
The forecast for S2SWA runs even if you link the gocart*.nc files instead of copying at the end.
Acceptance Criteria
The aerosol team has the diagnostic output they need.
Suggest a solution (optional)
I tried to implement suggestions from @rmontuoro and @bbakernoaa here: https://github.com/jessicameixner-noaa/global-workflow/tree/trygocartfix but the forecast still crashed. The setting updates are in this commit: JessicaMeixner-NOAA@f766fd9
The text was updated successfully, but these errors were encountered: