Watcher get 401 unauthorized error from jobs endpoint

[mbthornton-lbl]

2024-12-03 15:59:34,394 DEBUG: Starting new HTTPS connection (1): api-dev.microbiomedata.org:443
2024-12-03 15:59:34,635 DEBUG: https://api-dev.microbiomedata.org:443 "GET /jobs?max_page_size=100&filter=%7B%22workflow.id%22:%20%7B%22$in%22:%20%5B%22Sequencing%20Noninterleaved:%20%22,%20%22Sequencin
g%20Interleaved:%20%22,%20%22Reads%20QC:%20v1.0.13%22,%20%22Reads%20QC%20Interleave:%20v1.0.12%22,%20%22Metagenome%20Assembly:%20v1.0.7%22,%20%22Metagenome%20Annotation:%20v1.1.0%22,%20%22MAGs:%20v1.3.1
2%22,%20%22Readbased%20Analysis:%20v1.0.8%22%5D%7D,%20%22claims%22:%20%7B%22$size%22:%200%7D%7D HTTP/11" 401 43
2024-12-03 15:59:34,635 WARNING: {'detail': 'Could not validate credentials'}

[mbthornton-lbl]

Getting a different error:

ERROR:nmdc_automation.workflow_automation.watch_nmdc:Error occurred during cycle: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/global/cfs/cdirs/m3408/nmdc_automation/dev/nmdc_automation/.venv/lib/python3.11/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.9.0/python-3.11/nersc-python/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.9.0/python-3.11/nersc-python/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.9.0/python-3.11/nersc-python/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/workflow_automation/watch_nmdc.py", line 394, in watch
    self.cycle()
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/workflow_automation/watch_nmdc.py", line 344, in cycle
    unclaimed_jobs = self.runtime_api_handler.get_unclaimed_jobs(self.config.allowed_workflows)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/workflow_automation/watch_nmdc.py", line 303, in get_unclaimed_jobs
    job_records = self.runtime_api.list_jobs(filt=filt)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/api/nmdcapi.py", line 71, in _get_token
    self.get_token()
  File "/global/homes/n/nmdcda/nmdc_automation/dev/nmdc_automation/nmdc_automation/api/nmdcapi.py", line 90, in get_token
    resp = requests.post(url, headers=h, data=data).json()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/cfs/cdirs/m3408/nmdc_automation/dev/nmdc_automation/.venv/lib/python3.11/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)