Handling errors on large zstash runs

[forsyth2]

Questions/comments from @darincomeau:

• zstash create failed to transfer a tar from LCRC to NERSC HPSS due to a failed check.
• Listing file contents doesn't return anything (zstash ls -l --hpss=globus://nersc/~/amoc/v1.HR.piControl.advection-bug-fix 00035f.tar -- the tar with the error, zstash ls -l --hpss=globus://nersc/~/amoc/v1.HR.piControl.advection-bug-fix 00035e.tar -- a successful tar)
• Can zstash update or zstash create be used to redo just the one failed tar?

[darincomeau]

Thanks @forsyth2 for opening this up.

I've check on NERSC HPSS and this tar file is indeed empty over there:

A:.../v1.HR.piControl.advectiodu 00035f.tar
0	00035f.tar

and a non-problematic tar file, for which I still can't list files as noted above, does seem to be the right size

A:.../v1.HR.piControl.advectiodu 00035e.tar
536616280	00035e.tar
-----------------------
536616280	total 512-byte blocks, 1 Files (274,747,535,360 bytes)

[forsyth2]

@darincomeau (also tagging @golaz in case he has any insights here).

zstash create failed to transfer a tar from LCRC to NERSC HPSS due to a failed check.

With #218 (which will be available in the next zstash release (and in the next E3SM Unified release), zstash will automatically retry zstash extract if a tar transfer is unsuccessful. However, the same automatic-retry functionality does not exist for zstash create.

Listing file contents doesn't return anything (zstash ls -l --hpss=globus://nersc/~/amoc/v1.HR.piControl.advection-bug-fix 00035f.tar -- the tar with the error, zstash ls -l --hpss=globus://nersc/~/amoc/v1.HR.piControl.advection-bug-fix 00035e.tar -- a successful tar)

I'm not sure what's going on here. Where are you running the commands from? https://e3sm-project.github.io/zstash/_build/html/master/usage.html#list does note "Running zstash ls outside the source directory (the directory you’re archiving) is not advised. zstash will only retrieve index.db from the HPSS archive if a local archive (cache) is not present."

Can zstash update or zstash create be used to redo just the one failed tar?

zstash update will only add changes, but I'm not sure if it will recognize failed-to-transfer files as changes to update. I need to look into this more.

[darincomeau]

Where are you running the commands from? https://e3sm-project.github.io/zstash/_build/html/master/usage.html#list does note "Running zstash ls outside the source directory (the directory you’re archiving) is not advised.

Ah that was it - I was running it in the zstash directory where the local copy of index.db was (but was still not being read, hence me adding the hpss path). Moving up a directory, I can now see the files that are supposed to be in the corrupted tar file:

(e3sm_unified_1.7.1_nompi) [ac.dcomeau@chrlogin1 v1.HR.piControl.advection-bug-fix]$ zstash ls -l --hpss=globus://nersc/~/amoc/v1.HR.piControl.advection-bug-fix 00035f.tar
id	name	size	mtime	md5	tar	offset
70249	zupdate3.txt	5501173	2022-10-20 13:56:29.484261	5233fb29ccf2635cebfd47a771c4329a	00035f.tar	0
70250	zupdate4.txt	141	2022-10-20 13:56:44.060863	d8cfbc1f8ce849dc0556e6045f60a0ed	00035f.tar	5502976
70251	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0103-11-01_00.00.00.nc	14566140880	2022-10-10 18:15:27.503590	22f2c03f94f071fcd15d327c1424ee48	00035f.tar	5505024
70252	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0103-12-01_00.00.00.nc	15570698224	2022-10-10 20:26:19.842366	c9687252543647e18398a326a3ac1d97	00035f.tar	14571647488
70253	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-01-01_00.00.00.nc	15570698224	2022-10-10 22:37:22.016189	e5badc8fec8aa5a3da9ec46c9a2b1b19	00035f.tar	30142347264
70254	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-02-01_00.00.00.nc	14063862208	2022-10-11 00:35:00.093848	c3180c62e6bb5fb0ba201aa4f8ce8e1e	00035f.tar	45713047040
70255	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-03-01_00.00.00.nc	15570698224	2022-10-11 02:44:58.662073	d8e8de4215ac89822f0b2195f4beb9e3	00035f.tar	59776910848
70256	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-04-01_00.00.00.nc	15068419552	2022-10-11 04:49:51.664170	297f667de32934e1f88f23a12df2de49	00035f.tar	75347610624
70257	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-05-01_00.00.00.nc	15068419552	2022-10-11 16:34:39.288761	8fe276a6942eda755528867bcf81bc72	00035f.tar	90416031744
70258	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-06-01_00.00.00.nc	15068419552	2022-10-11 18:35:26.240340	b02b4fe2714f920e06a81a57cade07dc	00035f.tar	105484452864
70259	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-07-01_00.00.00.nc	15068419552	2022-10-12 00:40:07.060476	77baf7640fdf44d03c5ea8f1d30e8756	00035f.tar	120552873984
70260	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-08-01_00.00.00.nc	15570698224	2022-10-12 02:47:14.789903	ab693ff4c3e624bea5ab35d07cde1549	00035f.tar	135621295104
70261	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-09-01_00.00.00.nc	15068419552	2022-10-12 04:51:23.682520	dd5fffd964fc31e98f182eb6b947feb6	00035f.tar	151191994880
70262	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-10-01_00.00.00.nc	15570698224	2022-10-12 07:02:14.666140	c1dfa97ee7585acc9e510fe480b4e8f3	00035f.tar	166260416000
70263	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-11-01_00.00.00.nc	15068419552	2022-10-12 09:11:23.839518	7b5ea026b40a46a16f5d5e53d31c284b	00035f.tar	181831115776
70264	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0104-12-01_00.00.00.nc	15570698224	2022-10-12 11:24:49.597480	52c787de340e4dd131e08da75f9a30ce	00035f.tar	196899536896
70265	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0105-01-01_00.00.00.nc	15570698224	2022-10-12 13:37:12.470932	cb95a5abbb8c5bf36c9c1804f658be1f	00035f.tar	212470236672
70266	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0105-02-01_00.00.00.nc	14063862208	2022-10-12 15:36:18.351678	c0df430141f4ad006c082923b879dd01	00035f.tar	228040936448
70267	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0105-03-01_00.00.00.nc	15570698224	2022-10-12 17:46:19.399329	1e86e4f15e7c9bb9d42f456e8d2d65d1	00035f.tar	242104800256
70268	archive/ocn/hist/mpaso.hist.am.highFrequencyOutput.0105-04-01_00.00.00.nc	15068419552	2022-10-12 19:51:57.912974	ff860ee875a9771484048e2e59bbc2b2	00035f.tar	257675500032

So it's about 1.5 years of the ocean high frequency output files. I suppose zstash update goes by file names only, and doesn't look at datestamps? I say this because I could copy these files out, and back in to update the datestamp, to trick zstash update into picking them up on the next update?

But now that I know the files, I can at least tar these up myself and transfer over as a backup plan. zstash extract won't pick them up, but I can at least make a note on the simulation page about these files if someone is looking for them.

[forsyth2]

I suppose zstash update goes by file names only, and doesn't look at datestamps?

zstash update does indeed look at datestamps (https://github.com/E3SM-Project/zstash/blob/main/zstash/update.py#L161, https://github.com/E3SM-Project/zstash/blob/main/zstash/update.py#L186).

So, I think it would be fine to try zstash update.

[darincomeau]

@forsyth2 changing the timestep indeed worked - zstash is picking those files up now. I'm all set with this - thanks for the help here!

[forsyth2]

Great! Glad to help!