Adapting SpatialData to naming restrictions

[aeisenbarth]

Since the introduction of naming restrictions in 137e1e0 and #703, the issue can occur that previously created SpatialData files have become invalid in the current SpatialData version. For being able to use them in newer versions, they need to be adapted to the new naming rules.

The rules are as follows:

• ✅ Only alphanumeric characters and underscores _, hyphens - and dots ..
  Alphanumeric includes upper- and lowercase letters of various alphabets (µ, 福), and number-like (²). Whitespace and symbols are excluded (especially those disallowed by file systems like /,\).
• ❌ Full names ., .., which would have special meanings as file paths.
• ❌ Names prefixed with double underscores __.
• ❌ Names only differing in character case, to avoid name collisions on case-insensitive systems. abc, Abc, ABC cannot occur together, but a single one of either case is allowed.
• ❌ For var and obs dataframes: _index is a reserved name.

With the following code samples, I show how to identify invalid names and how to rename them. The examples work without parsing the files with SpatialData, since it would reject them.

Approach

1. Iterate over all elements and check names against the validation functions.
   Collect paths of invalid names. I chose paths because they contain the nesting level and make renaming easier.
2. Transform the names into valid names with a transformation function that you define.
3. Rename the paths.

It is important to know:

• The name of SpatialData elements is only encoded in the folder paths.
• However, labels, points and shapes may be referenced by the region (or whatever REGION_KEY is) in tables. This column is stored in binary Zarr files and cannot be edited in a text editor. If that is the case, you need to read the valid SpatialData afterwards and correct the entries in the table.
• The names of obs and var columns are also referenced in the obs/.zattrs file (or var/.zattrs) in the "column-order" list. You must also rename the entries in this list.

Code

Assume spatialdata_path is the file path to the SpatialData .zarr file.

1. Iterate over all elements and check names against the validation functions

import json
import re
from collections import Counter
from itertools import groupby
from pathlib import Path
from spatialdata._core.validation import check_valid_name, check_all_keys_case_insensitively_unique

all_element_paths = []
for element_types in ("images", "labels", "points", "shapes", "tables"):
    if not (spatialdata_path / element_types).exists():
        continue
    for element_path in (spatialdata_path / element_types).iterdir():
        if not element_path.is_dir():
            continue
        all_element_paths.append(element_path)

Collect SpatialData elements whose names contain forbidden characters:

invalid_element_paths = []
for element_path in all_element_paths:
    try:
        check_valid_name(element_path.name)
    except ValueError:
        invalid_element_paths.append(element_path)

Collect SpatialData elements whose names are not unique on case-insensitive file systems:

case_collisions = {k for k, v in Counter([p.name.lower() for p in all_element_paths]).items() if v > 1}
duplicate_element_paths = [p for p in all_element_paths if p.name.lower() in case_collisions]

Find table columns with forbidden characters, and table columns with colliding names:

invalid_table_columns = []
duplicate_table_columns = []
for table_path in (spatialdata_path / "tables").iterdir():
    if not table_path.is_dir():
        continue
    for attr in ("obs", "obsm", "obsp", "var", "varm", "varp", "uns", "layers"):
        col_paths = [p for p in (table_path / attr).iterdir() if p.is_dir()]
        for col_path in col_paths:
            try:
                check_valid_name(col_path.name)
            except ValueError:
                invalid_table_columns.append(col_path)
        case_collisions = {k for k, v in Counter([p.name.lower() for p in col_paths]).items() if v > 1}
        duplicate_element_paths += [p for p in col_paths if p.name.lower() in case_collisions]

Now that you have collected paths that you need to rename, print them and inspect them.

2. Transform the names into valid names with a transformation function

For simplicity, we continue with this primitive replacement function.

def transform_name(old_name: str) -> str:
    return re.sub(r"[^\w\._-]", "_", old_name)

Note that this can cause new name collisions if different forbidden character are replaced by the same. With a more concrete use case, you may rather write them to a CSV file, assign new names and read them back to a dictionary.

3. Rename the paths

For elements:

for path in invalid_element_paths:
    path.rename(path.parent / transform_name(path.name))

And for table columns:

for path in invalid_table_columns:
    path.rename(path.parent / transform_name(path.name))
for group, column_paths in groupby(invalid_table_columns, lambda p: p.parent):
    to_rename = {p.name for p in column_paths}
    zattrs_path = group / ".zattrs"
    json_dct = json.loads(zattrs_path.read_text())
    json_dct["column-order"] = [transform_name(name) if name in to_rename else name for name in json_dct["column-order"]]
    zattrs_path.write_text(json.dumps(json_dct, indent=4))

Proceed analogously with duplicate_element_paths, duplicate_table_columns.