PLAT-341: debug _DJConnector._fetch_records for filepath attributes

[ethho]

https://datajoint.atlassian.net/browse/PLAT-341

[ethho]

Able to reproduce the error on commit c6ad4ba. We can't fetch the filepath attributes because we force filepath download in datajoint.fetch.py here, when the store is not configured on the works-api host:

https://github.com/datajoint/datajoint-python/blob/4c1540f533844c9d58ee0d7f9265197e6586a025/datajoint/fetch.py#L63-L64

    if attr.is_filepath:
        return adapt(extern.download_filepath(uuid.UUID(bytes=data))[0])

Ideally, we'd have a kwarg so that we could fetch with fetch(..., skip_download_external=True). As a workaround, we could try catching the DataJointError thrown.

pytest logs:

❯ PY_VER=3.9 IMAGE=djtest DISTRO=alpine AS_SCRIPT=TRUE PHARUS_VERSION=$(cat pharus/version.py | grep -oP '\d+\.\d+\.\d+') HOST_UID=$(id -u) \
      docker compose -f docker-compose-test.yaml \
      up --exit-code-from pharus
# ...
pharus-1  | ------ SYNTAX TESTS ------
pharus-1  | 0
pharus-1  | 0
pharus-1  | ------ UNIT TESTS ------
pharus-1  | [2024-09-08 19:57:11,150][INFO]: DataJoint verified plugin `datajoint_connection_hub` detected.
pharus-1  | USING STANDARD TYPE_MAP
pharus-1  | 'FLASK_ENV' is deprecated and will not be used in Flask 2.3. Use 'FLASK_DEBUG' instead.
pharus-1  | ============================= test session starts ==============================
pharus-1  | platform linux -- Python 3.9.15, pytest-8.3.2, pluggy-1.5.0 -- /opt/conda/bin/python
pharus-1  | cachedir: .pytest_cache
pharus-1  | rootdir: /main
pharus-1  | plugins: Faker-28.4.1, cov-5.0.0
pharus-1  | collecting ... [2024-09-08 19:57:12,829][INFO]: Connecting [email protected]:3306
pharus-1  | [2024-09-08 19:57:13,152][INFO]: Connected [email protected]:3306
collected 1 item
pharus-1  |
pharus-1  | tests/test_interface.py::TestDJConnector::test_can_fetch_filepath_attrs [2024-09-08 19:57:13,704][INFO]: Connecting [email protected]:3306
pharus-1  | [2024-09-08 19:57:14,031][INFO]: Connected [email protected]:3306
pharus-1  | FAILED
pharus-1  |
pharus-1  | =================================== FAILURES ===================================
pharus-1  | ________________ TestDJConnector.test_can_fetch_filepath_attrs _________________
pharus-1  |
pharus-1  | self = <tests.test_interface.TestDJConnector object at 0x7f7f49b0ceb0>
pharus-1  | nei_nienborg_model_labeledvideo_file = <class 'datajoint.schemas.File'>
pharus-1  | connection = DataJoint connection (connected) [email protected]:3306
pharus-1  |
pharus-1  |     @pytest.mark.skipif(
pharus-1  |         (get_schema_as_vm('nei_nienborg_model', dj.conn(**get_db_creds())) is None),
pharus-1  |         reason="Cannot access schema 'iub_lulab_devo_model' with these credentials"
pharus-1  |     )
pharus-1  |     def test_can_fetch_filepath_attrs(self, nei_nienborg_model_labeledvideo_file: dj.Table, connection):
pharus-1  |         """
pharus-1  |         Tests _DJConnector._fetch_records for a table with a filepath attribute
pharus-1  |
pharus-1  |         https://datajoint.atlassian.net/browse/PLAT-341
pharus-1  |         """
pharus-1  |         table = nei_nienborg_model_labeledvideo_file
pharus-1  |         assert os.environ.get('DJ_SUPPORT_FILEPATH_MANAGEMENT').upper() == 'TRUE'
pharus-1  |         assert table.fetch('KEY', limit=1, download_path=None)
pharus-1  | >       assert table.fetch(limit=1)
pharus-1  |
pharus-1  | tests/test_interface.py:36:
pharus-1  | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pharus-1  | /opt/conda/lib/python3.9/site-packages/datajoint/fetch.py:291: in __call__
pharus-1  |     ret[name] = list(map(partial(get, heading[name]), ret[name]))
pharus-1  | /opt/conda/lib/python3.9/site-packages/datajoint/fetch.py:55: in _get
pharus-1  |     connection.schemas[attr.database].external[attr.store]
pharus-1  | /opt/conda/lib/python3.9/site-packages/datajoint/external.py:497: in __getitem__
pharus-1  |     self._tables[store] = ExternalTable(
pharus-1  | /opt/conda/lib/python3.9/site-packages/datajoint/external.py:42: in __init__
pharus-1  |     self.spec = config.get_store_spec(store)
pharus-1  | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pharus-1  |
pharus-1  | self = {   'connection.charset': '',
pharus-1  |     'connection.init_function': None,
pharus-1  |     'database.host': 'localhost',
pharus-1  |     'database.pa...,
pharus-1  |     'fetch_format': 'array',
pharus-1  |     'filepath_checksum_size_limit': None,
pharus-1  |     'loglevel': 'INFO',
pharus-1  |     'safemode': False}
pharus-1  | store = 'dlc-processed'
pharus-1  |
pharus-1  |     def get_store_spec(self, store):
pharus-1  |         """
pharus-1  |         find configuration of external stores for blobs and attachments
pharus-1  |         """
pharus-1  |         try:
pharus-1  |             spec = self["stores"][store]
pharus-1  |         except KeyError:
pharus-1  | >           raise DataJointError(
pharus-1  |                 "Storage {store} is requested but not configured".format(store=store)
pharus-1  |             )
pharus-1  | E           datajoint.errors.DataJointError: Storage dlc-processed is requested but not configured
pharus-1  |
pharus-1  | /opt/conda/lib/python3.9/site-packages/datajoint/settings.py:139: DataJointError
pharus-1  | ------------------------------ Captured log setup ------------------------------
pharus-1  | INFO     datajoint:connection.py:190 Connecting [email protected]:3306
pharus-1  | INFO     datajoint:connection.py:195 Connected [email protected]:3306
pharus-1  | =============================== warnings summary ===============================
pharus-1  | pharus/__init__.py:7
pharus-1  | pharus/__init__.py:7
pharus-1  | pharus/__init__.py:7
pharus-1  | pharus/__init__.py:7
pharus-1  | pharus/__init__.py:7
pharus-1  |   /main/pharus/__init__.py:7: DeprecationWarning: table component to be Deprecated in next major release, please use antd-table
pharus-1  |     dynamic_api_gen.populate_api()
pharus-1  |
pharus-1  | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
pharus-1  |
pharus-1  | ---------- coverage: platform linux, python 3.9.15-final-0 -----------
pharus-1  | Name                            Stmts   Miss  Cover   Missing
pharus-1  | -------------------------------------------------------------
pharus-1  | pharus/__init__.py                 13      4    69%   8-9, 13-14
pharus-1  | pharus/component_interface.py     203    136    33%   41-49, 53, 65-79, 86-96, 107, 113, 132-138, 155, 158-183, 203, 206, 215-284, 295, 298-311, 314-344, 389-442, 454-459, 496-510, 523-526, 538-568, 581-583, 614-619, 634-639, 653-654, 667-672, 677-684
pharus-1  | pharus/dynamic_api.py             410    264    36%   22-31, 38-47, 54-63, 70-79, 86-95, 102-111, 118-127, 134-143, 150-159, 166-175, 182-191, 198-207, 214-223, 230-239, 246-255, 262-271, 278-287, 294-303, 310-319, 326-335, 342-351, 358-367, 374-383, 390-399, 406-415, 422-431, 438-447, 454-463, 470-479, 486-495, 502-511, 518-527, 534-543, 550-559, 566-575, 582-591, 598-607, 614-623, 630-639, 646-655, 662-671, 678-687, 694-703, 710-719
pharus-1  | pharus/dynamic_api_gen.py          54      5    91%   79-82, 92-93, 108
pharus-1  | pharus/error.py                     8      0   100%
pharus-1  | pharus/interface.py               136    100    26%   41, 72-102, 135-223, 245-284, 307-311, 332-335, 361-390, 410-414, 441-460, 479-489, 506-528
pharus-1  | pharus/server.py                  184    116    37%   74, 102-134, 174-175, 254-332, 396-402, 482-487, 811-883, 966-973, 1141-1159, 1171-1185, 1282-1297, 1304, 1308
pharus-1  | pharus/version.py                   1      0   100%
pharus-1  | -------------------------------------------------------------
pharus-1  | TOTAL                            1009    625    38%
pharus-1  |
pharus-1  | =========================== short test summary info ============================
pharus-1  | FAILED tests/test_interface.py::TestDJConnector::test_can_fetch_filepath_attrs - datajoint.errors.DataJointError: Storage dlc-processed is requested but not...
pharus-1  | ======================== 1 failed, 5 warnings in 2.31s =========================