Zstash error updating index.db

[wagmanbe]

Question criteria

• I added a descriptive title here.
• I searched the zstash GitHub Discussions to find a similar question and didn't find it.
• I searched the zstash documentation.

What is the deadline?

None

Describe your question

I am trying to archive a large ensemble of over 700 E3SM simulations, which is probably an unusual use for zstash. Zstash runs for a few days, creating and transferring dozens of tar files, but then throws an error UnicodeEncodeError: 'utf-8' codec can't encode characters in position 65-66: surrogates not allowed. Occurs for both zstash create and zstash update. The error originates from
site-packages/zstash/hpss_utils.py. Log files are available in /pscratch/sd/w/wagmanbe/E3SM_simulations/dakota/v3_pmcpu/ne30_F2010
Starting a discussion instead of creating an "issue" because this problem may be due to my unconventional use of zstash.

Are there are any possible answers you came across?

No response

What machine were you running on?

Perlmutter

Environment

zstash v1.3.0

Minimal Complete Verifiable Example (MCVE)

#########################################################################
ROOT=/pscratch/sd/w/wagmanbe/E3SM_simulations/dakota/v3_pmcpu/ne30_F2010/
#########################################################################

cd $ROOT
zstash create --hpss=/home/w/wagmanbe/E3SMv3/dakota/v3_pmcpu/ne30_F2010  --maxsize 128 . 2>&1 | tee zstash_create_20231107.log # Created 20231107

Relevant log output

INFO: Archiving ens/workdir.232/20230802.v3alpha02.F2010.pmcpu.intel.8N/archive/atm/hist/20230802.v3alpha02.F2010.pmcpu.intel.8N.eam.h0.0006-04.nc
INFO: Archiving ens/workdir.232/20230802.v3alpha02.F2010.pmcpu.intel.8N/archive/atm/hist/20230802.v3alpha02.F2010.pmcpu.intel.8N.eam.h0.0006-05.nc
INFO: tar name=0000a5.tar, tar size=136846192640, tar md5=c1621d26c83b996b52de13c90192d494
INFO: Transferring file to HPSS: zstash/0000a5.tar
For help, please see https://e3sm-project.github.io/zstash. Ask questions at https://github.com/E3SM-Project/zstash/discussions/categories/q-a.
Traceback (most recent call last):
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.1_login/bin/zstash", line 10, in <module>
    sys.exit(main())
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.1_login/lib/python3.10/site-packages/zstash/main.py", line 64, in main
    create()
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.1_login/lib/python3.10/site-packages/zstash/create.py", line 88, in create
    failures: List[str] = create_database(cache, args)
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.1_login/lib/python3.10/site-packages/zstash/create.py", line 263, in create_database
    failures = add_files(
  File "/global/common/software/e3sm/anaconda_envs/base/envs/e3sm_unified_1.9.1_login/lib/python3.10/site-packages/zstash/hpss_utils.py", line 165, in add_files
    cur.executemany("insert into files values (NULL,?,?,?,?,?,?)", archived)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 65-66: surrogates not allowed

Anything else we need to know?

I am running zstash from inside a screen session.

[forsyth2]

Hmm I've never seen anything like this before. https://stackoverflow.com/questions/27366479/python-3-os-walk-file-paths-unicodeencodeerror-utf-8-codec-cant-encode-s implies there are bad characters in the files. I'm not quite sure what would cause that.

@golaz @TonyB9000 Have you ever encountered anything like this?

[TonyB9000]

@forsyth2 I have not. But is the encoding issue with the file name, not the file content? It is possible to place all sorts of junk into the "filename" entry in a unix/linux file system inode. Back circa 1995, Lab investigators found a huge pron stash an employee had been hiding on a server, giving the image files names that you could not list or type at the console. I had to write C code that did "direct/dirent" walking of the inodes manually, expressing the bytes of the filenames as hex-chars to make them accessible. And that was before Unicode.

[wagmanbe]

There are some unusual file names in one of the directories that triggered the error. For example a file named,
^P��^O
I'll try identifying and removing such files. They seem to have been created by the machine after a seg fault.