diff --git a/CHANGES b/CHANGES index c954a63..7db4916 100644 --- a/CHANGES +++ b/CHANGES @@ -1,5 +1,11 @@ main ==== +Support for TIMEUUID and COUNTER columns: + - enlarged ColumnType enum (used by the API to describe non-creatable columns) + - readable through find/find_one operations (now returned by the API when reading) +shortened string representation of table column descriptors for improved readability +added 'filter' to the `TableAPISupportDescriptor` structure (now returned by the Data API) +added optional `api_support` member to all column descriptor (as the Data API returns it for various columns) restore support for Python 3.8, 3.9 maintenance: full restructuring of tests and CI (tables+collections on same footing+other) diff --git a/astrapy/data/info/table_descriptor/table_columns.py b/astrapy/data/info/table_descriptor/table_columns.py index a15644b..b239164 100644 --- a/astrapy/data/info/table_descriptor/table_columns.py +++ b/astrapy/data/info/table_descriptor/table_columns.py @@ -29,6 +29,78 @@ from astrapy.utils.parsing import _warn_residual_keys +@dataclass +class TableAPISupportDescriptor: + """ + Represents the additional support information returned by the Data API when + describing columns of a table. Some columns indeed require a detailed description + of what operations are supported on them - this includes, but is not limited to, + columns created by means other than the Data API (e.g. CQL direct interaction + with the database). + + When the Data API reports these columns (in listing the tables and their metadata), + it provides the information marshaled in this object to detail which level + of support the column has (for instance, it can be a partial support whereby the + column is readable by the API but not writable). + + Attributes: + cql_definition: a free-form string containing the CQL definition for the column. + create_table: whether a column of this nature can be used in API table creation. + insert: whether a column of this nature can be written through the API. + filter: whether a column of this nature can be used for filtering with API find. + read: whether a column of this nature can be read through the API. + """ + + cql_definition: str + create_table: bool + insert: bool + filter: bool + read: bool + + def __repr__(self) -> str: + desc = ", ".join( + [ + f'"{self.cql_definition}"', + f"create_table={self.create_table}", + f"insert={self.insert}", + f"filter={self.filter}", + f"read={self.read}", + ] + ) + return f"{self.__class__.__name__}({desc})" + + def as_dict(self) -> dict[str, Any]: + """Recast this object into a dictionary.""" + + return { + "cqlDefinition": self.cql_definition, + "createTable": self.create_table, + "insert": self.insert, + "filter": self.filter, + "read": self.read, + } + + @classmethod + def _from_dict(cls, raw_dict: dict[str, Any]) -> TableAPISupportDescriptor: + """ + Create an instance of TableAPISupportDescriptor from a dictionary + such as one from the Data API. + """ + + _warn_residual_keys( + cls, + raw_dict, + {"cqlDefinition", "createTable", "insert", "filter", "read"}, + ) + return TableAPISupportDescriptor( + cql_definition=raw_dict["cqlDefinition"], + create_table=raw_dict["createTable"], + insert=raw_dict["insert"], + filter=raw_dict["filter"], + read=raw_dict["read"], + ) + + @dataclass class TableColumnTypeDescriptor(ABC): """ @@ -46,6 +118,7 @@ class TableColumnTypeDescriptor(ABC): `column_type` attributes. For example the `column_type` of `TableValuedColumnTypeDescriptor` is a `TableValuedColumnType`. + api_support: a `TableAPISupportDescriptor` object giving more details. """ column_type: ( @@ -55,6 +128,7 @@ class TableColumnTypeDescriptor(ABC): | TableVectorColumnType | TableUnsupportedColumnType ) + api_support: TableAPISupportDescriptor | None @abstractmethod def as_dict(self) -> dict[str, Any]: ... @@ -77,10 +151,7 @@ def _from_dict(cls, raw_dict: dict[str, Any]) -> TableColumnTypeDescriptor: elif raw_dict["type"] == "UNSUPPORTED": return TableUnsupportedColumnTypeDescriptor._from_dict(raw_dict) else: - _warn_residual_keys(cls, raw_dict, {"type"}) - return TableScalarColumnTypeDescriptor( - column_type=raw_dict["type"], - ) + return TableScalarColumnTypeDescriptor._from_dict(raw_dict) @classmethod def coerce( @@ -108,21 +179,34 @@ class TableScalarColumnTypeDescriptor(TableColumnTypeDescriptor): Attributes: column_type: a `ColumnType` value. When creating the object, simple strings such as "TEXT" or "UUID" are also accepted. + api_support: a `TableAPISupportDescriptor` object giving more details. """ column_type: ColumnType - def __init__(self, column_type: str | ColumnType) -> None: - self.column_type = ColumnType.coerce(column_type) + def __init__( + self, + column_type: str | ColumnType, + api_support: TableAPISupportDescriptor | None = None, + ) -> None: + super().__init__( + column_type=ColumnType.coerce(column_type), + api_support=api_support, + ) def __repr__(self) -> str: - return f"{self.__class__.__name__}({self.column_type})" + return f"{self.__class__.__name__}({self.column_type.value})" def as_dict(self) -> dict[str, Any]: """Recast this object into a dictionary.""" return { - "type": self.column_type.value, + k: v + for k, v in { + "type": self.column_type.value, + "apiSupport": self.api_support.as_dict() if self.api_support else None, + }.items() + if v is not None } @classmethod @@ -132,9 +216,12 @@ def _from_dict(cls, raw_dict: dict[str, Any]) -> TableScalarColumnTypeDescriptor such as one from the Data API. """ - _warn_residual_keys(cls, raw_dict, {"type"}) + _warn_residual_keys(cls, raw_dict, {"type", "apiSupport"}) return TableScalarColumnTypeDescriptor( column_type=raw_dict["type"], + api_support=TableAPISupportDescriptor._from_dict(raw_dict["apiSupport"]) + if raw_dict.get("apiSupport") + else None, ) @@ -151,6 +238,7 @@ class TableVectorColumnTypeDescriptor(TableColumnTypeDescriptor): This can be left unspecified in some cases of vectorize-enabled columns. service: an optional `VectorServiceOptions` object defining the vectorize settings (i.e. server-side embedding computation) for the column. + api_support: a `TableAPISupportDescriptor` object giving more details. """ column_type: TableVectorColumnType @@ -163,10 +251,14 @@ def __init__( column_type: str | TableVectorColumnType = TableVectorColumnType.VECTOR, dimension: int | None, service: VectorServiceOptions | None = None, + api_support: TableAPISupportDescriptor | None = None, ) -> None: self.dimension = dimension self.service = service - super().__init__(column_type=TableVectorColumnType.coerce(column_type)) + super().__init__( + column_type=TableVectorColumnType.coerce(column_type), + api_support=api_support, + ) def __repr__(self) -> str: not_null_pieces = [ @@ -179,7 +271,7 @@ def __repr__(self) -> str: ] inner_desc = ", ".join(not_null_pieces) - return f"{self.__class__.__name__}({self.column_type}[{inner_desc}])" + return f"{self.__class__.__name__}({self.column_type.value}[{inner_desc}])" def as_dict(self) -> dict[str, Any]: """Recast this object into a dictionary.""" @@ -190,6 +282,7 @@ def as_dict(self) -> dict[str, Any]: "type": self.column_type.value, "dimension": self.dimension, "service": None if self.service is None else self.service.as_dict(), + "apiSupport": self.api_support.as_dict() if self.api_support else None, }.items() if v is not None } @@ -201,11 +294,18 @@ def _from_dict(cls, raw_dict: dict[str, Any]) -> TableVectorColumnTypeDescriptor such as one from the Data API. """ - _warn_residual_keys(cls, raw_dict, {"type", "dimension", "service"}) + _warn_residual_keys( + cls, + raw_dict, + {"type", "dimension", "service", "apiSupport"}, + ) return TableVectorColumnTypeDescriptor( column_type=raw_dict["type"], dimension=raw_dict.get("dimension"), service=VectorServiceOptions.coerce(raw_dict.get("service")), + api_support=TableAPISupportDescriptor._from_dict(raw_dict["apiSupport"]) + if raw_dict.get("apiSupport") + else None, ) @@ -221,6 +321,7 @@ class TableValuedColumnTypeDescriptor(TableColumnTypeDescriptor): value_type: the type of the individual items stored in the column. This is a `ColumnType`, but when creating the object, strings such as "TEXT" or "UUID" are also accepted. + api_support: a `TableAPISupportDescriptor` object giving more details. """ column_type: TableValuedColumnType @@ -231,19 +332,31 @@ def __init__( *, column_type: str | TableValuedColumnType, value_type: str | ColumnType, + api_support: TableAPISupportDescriptor | None = None, ) -> None: self.value_type = ColumnType.coerce(value_type) - super().__init__(column_type=TableValuedColumnType.coerce(column_type)) + super().__init__( + column_type=TableValuedColumnType.coerce(column_type), + api_support=api_support, + ) def __repr__(self) -> str: - return f"{self.__class__.__name__}({self.column_type}<{self.value_type}>)" + return ( + f"{self.__class__.__name__}({self.column_type.value}" + f"<{self.value_type.value}>)" + ) def as_dict(self) -> dict[str, Any]: """Recast this object into a dictionary.""" return { - "type": self.column_type.value, - "valueType": self.value_type.value, + k: v + for k, v in { + "type": self.column_type.value, + "valueType": self.value_type.value, + "apiSupport": self.api_support.as_dict() if self.api_support else None, + }.items() + if v is not None } @classmethod @@ -253,10 +366,13 @@ def _from_dict(cls, raw_dict: dict[str, Any]) -> TableValuedColumnTypeDescriptor such as one from the Data API. """ - _warn_residual_keys(cls, raw_dict, {"type", "valueType"}) + _warn_residual_keys(cls, raw_dict, {"type", "valueType", "apiSupport"}) return TableValuedColumnTypeDescriptor( column_type=raw_dict["type"], value_type=raw_dict["valueType"], + api_support=TableAPISupportDescriptor._from_dict(raw_dict["apiSupport"]) + if raw_dict.get("apiSupport") + else None, ) @@ -276,6 +392,7 @@ class TableKeyValuedColumnTypeDescriptor(TableColumnTypeDescriptor): value_type: the type of the individual values stored in the map for a single key. This is a `ColumnType`, but when creating the object, strings such as "TEXT" or "UUID" are also accepted. + api_support: a `TableAPISupportDescriptor` object giving more details. """ column_type: TableKeyValuedColumnType @@ -288,21 +405,33 @@ def __init__( column_type: str | TableKeyValuedColumnType, value_type: str | ColumnType, key_type: str | ColumnType, + api_support: TableAPISupportDescriptor | None = None, ) -> None: self.key_type = ColumnType.coerce(key_type) self.value_type = ColumnType.coerce(value_type) - super().__init__(column_type=TableKeyValuedColumnType.coerce(column_type)) + super().__init__( + column_type=TableKeyValuedColumnType.coerce(column_type), + api_support=api_support, + ) def __repr__(self) -> str: - return f"{self.__class__.__name__}({self.column_type}<{self.key_type},{self.value_type}>)" + return ( + f"{self.__class__.__name__}({self.column_type.value}" + f"<{self.key_type.value},{self.value_type.value}>)" + ) def as_dict(self) -> dict[str, Any]: """Recast this object into a dictionary.""" return { - "type": self.column_type.value, - "keyType": self.key_type.value, - "valueType": self.value_type.value, + k: v + for k, v in { + "type": self.column_type.value, + "keyType": self.key_type.value, + "valueType": self.value_type.value, + "apiSupport": self.api_support.as_dict() if self.api_support else None, + }.items() + if v is not None } @classmethod @@ -312,76 +441,16 @@ def _from_dict(cls, raw_dict: dict[str, Any]) -> TableKeyValuedColumnTypeDescrip such as one from the Data API. """ - _warn_residual_keys(cls, raw_dict, {"type", "keyType", "valueType"}) + _warn_residual_keys( + cls, raw_dict, {"type", "keyType", "valueType", "apiSupport"} + ) return TableKeyValuedColumnTypeDescriptor( column_type=raw_dict["type"], key_type=raw_dict["keyType"], value_type=raw_dict["valueType"], - ) - - -@dataclass -class TableAPISupportDescriptor: - """ - Represents the additional information returned by the Data API when describing - a table with unsupported columns. Unsupported columns may have been created by - means other than the Data API (e.g. CQL direct interaction with the database). - - The Data API reports these columns when listing the tables and their metadata, - and provides the information marshaled in this object to detail which level - of support the column has (for instance, it can be a partial support where the - column is readable by the API but not writable). - - Attributes: - cql_definition: a free-form string containing the CQL definition for the column. - create_table: whether a column of this nature can be used in API table creation. - insert: whether a column of this nature can be written through the API. - read: whether a column of this nature can be read through the API. - """ - - cql_definition: str - create_table: bool - insert: bool - read: bool - - def __repr__(self) -> str: - desc = ", ".join( - [ - f'"{self.cql_definition}"', - f"create_table={self.create_table}", - f"insert={self.insert}", - f"read={self.read}", - ] - ) - return f"{self.__class__.__name__}({desc})" - - def as_dict(self) -> dict[str, Any]: - """Recast this object into a dictionary.""" - - return { - "cqlDefinition": self.cql_definition, - "createTable": self.create_table, - "insert": self.insert, - "read": self.read, - } - - @classmethod - def _from_dict(cls, raw_dict: dict[str, Any]) -> TableAPISupportDescriptor: - """ - Create an instance of TableAPISupportDescriptor from a dictionary - such as one from the Data API. - """ - - _warn_residual_keys( - cls, - raw_dict, - {"cqlDefinition", "createTable", "insert", "read"}, - ) - return TableAPISupportDescriptor( - cql_definition=raw_dict["cqlDefinition"], - create_table=raw_dict["createTable"], - insert=raw_dict["insert"], - read=raw_dict["read"], + api_support=TableAPISupportDescriptor._from_dict(raw_dict["apiSupport"]) + if raw_dict.get("apiSupport") + else None, ) @@ -410,8 +479,10 @@ def __init__( column_type: TableUnsupportedColumnType | str, api_support: TableAPISupportDescriptor, ) -> None: - self.api_support = api_support - super().__init__(column_type=TableUnsupportedColumnType.coerce(column_type)) + super().__init__( + column_type=TableUnsupportedColumnType.coerce(column_type), + api_support=api_support, + ) def __repr__(self) -> str: return f"{self.__class__.__name__}({self.api_support.cql_definition})" @@ -420,8 +491,12 @@ def as_dict(self) -> dict[str, Any]: """Recast this object into a dictionary.""" return { - "type": self.column_type.value, - "apiSupport": self.api_support.as_dict(), + k: v + for k, v in { + "type": self.column_type.value, + "apiSupport": self.api_support.as_dict(), + }.items() + if v is not None } @classmethod diff --git a/astrapy/data/utils/table_converters.py b/astrapy/data/utils/table_converters.py index cc4c0ae..412cf7b 100644 --- a/astrapy/data/utils/table_converters.py +++ b/astrapy/data/utils/table_converters.py @@ -93,6 +93,7 @@ def _tpostprocessor_bool(raw_value: Any) -> bool | None: ColumnType.BIGINT, ColumnType.SMALLINT, ColumnType.TINYINT, + ColumnType.COUNTER, }: def _tpostprocessor_int(raw_value: Any) -> int | None: @@ -136,7 +137,7 @@ def _tpostprocessor_bytes(raw_value: Any) -> bytes | None: return _tpostprocessor_bytes - elif column_type == ColumnType.UUID: + elif column_type in {ColumnType.UUID, ColumnType.TIMEUUID}: def _tpostprocessor_uuid(raw_value: Any) -> UUID | None: if raw_value is None: diff --git a/astrapy/data/utils/table_types.py b/astrapy/data/utils/table_types.py index bb0bc9e..05990b1 100644 --- a/astrapy/data/utils/table_types.py +++ b/astrapy/data/utils/table_types.py @@ -29,6 +29,7 @@ class ColumnType(StrEnum): BIGINT = "bigint" BLOB = "blob" BOOLEAN = "boolean" + COUNTER = "counter" DATE = "date" DECIMAL = "decimal" DOUBLE = "double" @@ -40,6 +41,7 @@ class ColumnType(StrEnum): TEXT = "text" TIME = "time" TIMESTAMP = "timestamp" + TIMEUUID = "timeuuid" TINYINT = "tinyint" UUID = "uuid" VARINT = "varint" diff --git a/tests/base/integration/tables/test_table_lifecycle_async.py b/tests/base/integration/tables/test_table_lifecycle_async.py index 861f336..0511c56 100644 --- a/tests/base/integration/tables/test_table_lifecycle_async.py +++ b/tests/base/integration/tables/test_table_lifecycle_async.py @@ -14,7 +14,7 @@ from __future__ import annotations -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any import pytest @@ -43,6 +43,21 @@ from astrapy import AsyncDatabase +def _remove_apisupport(def_dict: dict[str, Any]) -> dict[str, Any]: + """ + Strip apiSupport keys from columns since its presence + is != between sending and receiving. + """ + + def _clean_col(col_dict: dict[str, Any]) -> dict[str, Any]: + return {k: v for k, v in col_dict.items() if k != "apiSupport"} + + return { + k: v if k != "columns" else {colk: _clean_col(colv) for colk, colv in v.items()} + for k, v in def_dict.items() + } + + class TestTableLifecycle: @pytest.mark.describe("test of create/verify/delete tables, async") async def test_table_basic_crd_async( @@ -184,9 +199,11 @@ async def test_table_basic_crd_async( ) # definition and info + atf_def = await atable_fluent.definition() assert ( - await atable_fluent.definition() - ).as_dict() == table_whole_obj_definition.as_dict() + _remove_apisupport(atf_def.as_dict()) + == table_whole_obj_definition.as_dict() + ) if IS_ASTRA_DB: fl_info = await atable_fluent.info() assert fl_info.name == "table_fluent" @@ -382,6 +399,7 @@ async def test_alter_tableindex_async( operation=AlterTableDropVectorize.coerce({"columns": ["p_vector"]}) ) # back to the original table: - assert (await atable.definition()).as_dict() == orig_table_def.as_dict() + adef = await atable.definition() + assert _remove_apisupport(adef.as_dict()) == orig_table_def.as_dict() finally: await atable.drop() diff --git a/tests/base/integration/tables/test_table_lifecycle_sync.py b/tests/base/integration/tables/test_table_lifecycle_sync.py index a734dba..cbc6f74 100644 --- a/tests/base/integration/tables/test_table_lifecycle_sync.py +++ b/tests/base/integration/tables/test_table_lifecycle_sync.py @@ -14,7 +14,7 @@ from __future__ import annotations -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any import pytest @@ -43,6 +43,21 @@ from astrapy import Database +def _remove_apisupport(def_dict: dict[str, Any]) -> dict[str, Any]: + """ + Strip apiSupport keys from columns since its presence + is != between sending and receiving. + """ + + def _clean_col(col_dict: dict[str, Any]) -> dict[str, Any]: + return {k: v for k, v in col_dict.items() if k != "apiSupport"} + + return { + k: v if k != "columns" else {colk: _clean_col(colv) for colk, colv in v.items()} + for k, v in def_dict.items() + } + + class TestTableLifecycle: @pytest.mark.describe("test of create/verify/delete tables, sync") def test_table_basic_crd_sync( @@ -185,7 +200,8 @@ def test_table_basic_crd_sync( # definition and info assert ( - table_fluent.definition().as_dict() == table_whole_obj_definition.as_dict() + _remove_apisupport(table_fluent.definition().as_dict()) + == table_whole_obj_definition.as_dict() ) if IS_ASTRA_DB: fl_info = table_fluent.info() @@ -382,6 +398,9 @@ def test_alter_tableindex_sync( operation=AlterTableDropVectorize.coerce({"columns": ["p_vector"]}) ) # back to the original table: - assert table.definition().as_dict() == orig_table_def.as_dict() + assert ( + _remove_apisupport(table.definition().as_dict()) + == orig_table_def.as_dict() + ) finally: table.drop() diff --git a/tests/base/unit/test_tabledescriptors.py b/tests/base/unit/test_tabledescriptors.py index 07abf2c..1be5246 100644 --- a/tests/base/unit/test_tabledescriptors.py +++ b/tests/base/unit/test_tabledescriptors.py @@ -85,23 +85,149 @@ }, }, { - "name": "uns_counter", + "name": "with_uncreatable_columns", "definition": { "columns": { "p_text": {"type": "text"}, "p_counter": { + "type": "counter", + "apiSupport": { + "createTable": False, + "insert": False, + "read": True, + "filter": True, + "cqlDefinition": "counter", + }, + }, + "p_timeuuid": { + "type": "timeuuid", + "apiSupport": { + "createTable": False, + "insert": True, + "read": True, + "filter": True, + "cqlDefinition": "timeuuid", + }, + }, + }, + "primaryKey": {"partitionBy": ["p_text"], "partitionSort": {}}, + }, + }, + { + "name": "with_unsupported", + "definition": { + "columns": { + "p_text": {"type": "text"}, + "my_nested_frozen": { "type": "UNSUPPORTED", "apiSupport": { "createTable": False, "insert": False, "read": False, - "cqlDefinition": "counter", + "filter": False, + "cqlDefinition": "map>>", }, }, }, "primaryKey": {"partitionBy": ["p_text"], "partitionSort": {}}, }, }, + { + "name": "without_apisupports", + "definition": { + "columns": { + "p_scalar": { + "type": "text", + }, + "p_list": { + "type": "list", + "valueType": "text", + }, + "p_map": { + "type": "map", + "keyType": "text", + "valueType": "text", + }, + "p_vector": { + "type": "vector", + "dimension": 999, + }, + "p_unsupported": { + "type": "UNSUPPORTED", + "apiSupport": { + "createTable": False, + "insert": False, + "read": False, + "filter": False, + "cqlDefinition": "real unsupported cannot omit apiSupport!", + }, + }, + }, + "primaryKey": {"partitionBy": ["id"], "partitionSort": {}}, + }, + }, + { + "name": "with_all_apisupports", + "definition": { + "columns": { + "p_scalar": { + "type": "text", + "apiSupport": { + "createTable": True, + "insert": True, + "read": True, + "filter": True, + "cqlDefinition": "for scalar, not seen in API yet", + }, + }, + "p_list": { + "type": "list", + "valueType": "text", + "apiSupport": { + "createTable": True, + "insert": True, + "read": True, + "filter": True, + "cqlDefinition": "the API returns apisupport for this type", + }, + }, + "p_map": { + "type": "map", + "keyType": "text", + "valueType": "text", + "apiSupport": { + "createTable": True, + "insert": True, + "read": True, + "filter": True, + "cqlDefinition": "the API returns apisupport for this type", + }, + }, + "p_vector": { + "type": "vector", + "dimension": 999, + "apiSupport": { + "createTable": True, + "insert": True, + "read": True, + "filter": True, + "cqlDefinition": "the API returns apisupport for this type", + }, + }, + "p_unsupported": { + "type": "UNSUPPORTED", + "apiSupport": { + "createTable": False, + "insert": False, + "read": False, + "filter": False, + "cqlDefinition": "real unsupported cannot omit apiSupport!", + }, + }, + }, + "primaryKey": {"partitionBy": ["id"], "partitionSort": {}}, + }, + }, ] DICT_DEFINITION = { diff --git a/tests/base/unit/test_tpostprocessors.py b/tests/base/unit/test_tpostprocessors.py index 61dfcbe..d05f795 100644 --- a/tests/base/unit/test_tpostprocessors.py +++ b/tests/base/unit/test_tpostprocessors.py @@ -94,6 +94,7 @@ "apiSupport": { "createTable": False, "insert": False, + "filter": True, "read": False, "cqlDefinition": "counter", }, @@ -103,6 +104,7 @@ "apiSupport": { "createTable": False, "insert": False, + "filter": True, "read": False, "cqlDefinition": "varchar", }, @@ -112,6 +114,7 @@ "apiSupport": { "createTable": False, "insert": False, + "filter": True, "read": False, "cqlDefinition": "timeuuid", }, @@ -493,7 +496,7 @@ def test_pk_postprocessors_from_schema_nocustom(self) -> None: expected_primary_key ) - @pytest.mark.descripte("test of type-based row preprocessor") + @pytest.mark.describe("test of type-based row preprocessor") def test_row_preprocessors_from_types(self) -> None: ptp_opts = FullSerdesOptions( binary_encode_vectors=True,