diff --git a/lib/dl_connector_oracle/LICENSE b/lib/dl_connector_oracle/LICENSE new file mode 100644 index 000000000..74ba5f6c7 --- /dev/null +++ b/lib/dl_connector_oracle/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2023 YANDEX LLC + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/lib/dl_connector_oracle/README.md b/lib/dl_connector_oracle/README.md new file mode 100644 index 000000000..aa34b44b1 --- /dev/null +++ b/lib/dl_connector_oracle/README.md @@ -0,0 +1 @@ +# dl_connector_oracle diff --git a/lib/dl_connector_oracle/dl_connector_oracle/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/__init__.py new file mode 100644 index 000000000..c6c1ecb8c --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/__init__.py @@ -0,0 +1,7 @@ +import sys + +import oracledb + + +oracledb.version = "8.3.0" +sys.modules["cx_Oracle"] = oracledb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/api/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/api_schema/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/api/api_schema/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/api_schema/connection.py b/lib/dl_connector_oracle/dl_connector_oracle/api/api_schema/connection.py new file mode 100644 index 000000000..56e441648 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/api/api_schema/connection.py @@ -0,0 +1,28 @@ +from __future__ import annotations + +from marshmallow import fields as ma_fields + +from dl_api_connector.api_schema.connection_base import ConnectionMetaMixin +from dl_api_connector.api_schema.connection_mixins import ( + DataExportForbiddenMixin, + RawSQLLevelMixin, +) +from dl_api_connector.api_schema.connection_sql import ClassicSQLConnectionSchema +from dl_api_connector.api_schema.extras import FieldExtra + +from dl_connector_oracle.core.constants import OracleDbNameType +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle + + +class OracleConnectionSchema( + ConnectionMetaMixin, RawSQLLevelMixin, DataExportForbiddenMixin, ClassicSQLConnectionSchema +): + TARGET_CLS = ConnectionSQLOracle + ALLOW_MULTI_HOST = True + + db_connect_method = ma_fields.Enum( + OracleDbNameType, + attribute="data.db_name_type", + required=True, + bi_extra=FieldExtra(editable=True), + ) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/connection_form/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/api/connection_form/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/connection_form/form_config.py b/lib/dl_connector_oracle/dl_connector_oracle/api/connection_form/form_config.py new file mode 100644 index 000000000..5a284b6e2 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/api/connection_form/form_config.py @@ -0,0 +1,113 @@ +from __future__ import annotations + +from enum import unique +from typing import Optional + +from dl_api_commons.base_models import TenantDef +from dl_api_connector.form_config.models.api_schema import ( + FormActionApiSchema, + FormApiSchema, + FormFieldApiSchema, +) +from dl_api_connector.form_config.models.base import ( + ConnectionForm, + ConnectionFormFactory, + ConnectionFormMode, +) +from dl_api_connector.form_config.models.common import ( + CommonFieldName, + FormFieldName, +) +import dl_api_connector.form_config.models.rows as C +from dl_api_connector.form_config.models.shortcuts.rows import RowConstructor +from dl_configs.connectors_settings import ConnectorSettingsBase + +from dl_connector_oracle.api.connection_info import OracleConnectionInfoProvider +from dl_connector_oracle.api.i18n.localizer import Translatable +from dl_connector_oracle.core.constants import OracleDbNameType + + +@unique +class OracleFieldName(FormFieldName): + db_connect_method = "db_connect_method" + + +class OracleConnectionFormFactory(ConnectionFormFactory): + def get_form_config( + self, + connector_settings: Optional[ConnectorSettingsBase], + tenant: Optional[TenantDef], + ) -> ConnectionForm: + rc = RowConstructor(self._localizer) + + common_api_schema_items: list[FormFieldApiSchema] = [ + FormFieldApiSchema(name=CommonFieldName.host, required=True), + FormFieldApiSchema(name=CommonFieldName.port, required=True), + FormFieldApiSchema(name=OracleFieldName.db_connect_method, required=True), + FormFieldApiSchema(name=CommonFieldName.db_name, required=True), + FormFieldApiSchema(name=CommonFieldName.username, required=True), + FormFieldApiSchema(name=CommonFieldName.password, required=self.mode == ConnectionFormMode.create), + ] + + edit_api_schema = FormActionApiSchema( + items=[ + *common_api_schema_items, + FormFieldApiSchema(name=CommonFieldName.cache_ttl_sec, nullable=True), + FormFieldApiSchema(name=CommonFieldName.raw_sql_level), + FormFieldApiSchema(name=CommonFieldName.data_export_forbidden), + ] + ) + + create_api_schema = FormActionApiSchema( + items=[ + *edit_api_schema.items, + *self._get_top_level_create_api_schema_items(), + ] + ) + + check_api_schema = FormActionApiSchema( + items=[ + *common_api_schema_items, + *self._get_top_level_check_api_schema_items(), + ] + ) + + db_name_row = C.CustomizableRow( + items=[ + *rc.db_name_row().items, + C.RadioButtonRowItem( + name=OracleFieldName.db_connect_method, + options=[ + C.SelectableOption( + text=self._localizer.translate(Translatable("value_db-connect-method-service-name")), + value=OracleDbNameType.service_name.value, + ), + C.SelectableOption( + text=self._localizer.translate(Translatable("value_db-connect-method-sid")), + value=OracleDbNameType.sid.value, + ), + ], + default_value=OracleDbNameType.service_name.value, + ), + ] + ) + + return ConnectionForm( + title=OracleConnectionInfoProvider.get_title(self._localizer), + rows=[ + rc.host_row(), + rc.port_row(default_value="1521"), + db_name_row, + rc.username_row(), + rc.password_row(self.mode), + C.CacheTTLRow(name=CommonFieldName.cache_ttl_sec), + rc.raw_sql_level_row(), + rc.collapse_advanced_settings_row(), + rc.data_export_forbidden_row(), + ], + api_schema=FormApiSchema( + create=create_api_schema if self.mode == ConnectionFormMode.create else None, + edit=edit_api_schema if self.mode == ConnectionFormMode.edit else None, + check=check_api_schema, + ), + ) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/connection_info.py b/lib/dl_connector_oracle/dl_connector_oracle/api/connection_info.py new file mode 100644 index 000000000..656697b25 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/api/connection_info.py @@ -0,0 +1,7 @@ +from dl_api_connector.connection_info import ConnectionInfoProvider + +from dl_connector_oracle.api.i18n.localizer import Translatable + + +class OracleConnectionInfoProvider(ConnectionInfoProvider): + title_translatable = Translatable("label_connector-oracle") diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/connector.py b/lib/dl_connector_oracle/dl_connector_oracle/api/connector.py new file mode 100644 index 000000000..4934afc81 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/api/connector.py @@ -0,0 +1,55 @@ +from dl_api_connector.api_schema.source_base import ( + SchematizedSQLDataSourceSchema, + SchematizedSQLDataSourceTemplateSchema, + SubselectDataSourceSchema, + SubselectDataSourceTemplateSchema, +) +from dl_api_connector.connector import ( + ApiConnectionDefinition, + ApiConnector, + ApiSourceDefinition, +) + +from dl_connector_oracle.api.api_schema.connection import OracleConnectionSchema +from dl_connector_oracle.api.connection_form.form_config import OracleConnectionFormFactory +from dl_connector_oracle.api.connection_info import OracleConnectionInfoProvider +from dl_connector_oracle.api.dashsql import OracleDashSQLParamLiteralizer +from dl_connector_oracle.api.i18n.localizer import CONFIGS +from dl_connector_oracle.core.connector import ( + OracleCoreConnectionDefinition, + OracleCoreConnector, + OracleSubselectCoreSourceDefinition, + OracleTableCoreSourceDefinition, +) +from dl_connector_oracle.formula.constants import DIALECT_NAME_ORACLE + + +class OracleApiTableSourceDefinition(ApiSourceDefinition): + core_source_def_cls = OracleTableCoreSourceDefinition + api_schema_cls = SchematizedSQLDataSourceSchema + template_api_schema_cls = SchematizedSQLDataSourceTemplateSchema + + +class OracleApiSubselectSourceDefinition(ApiSourceDefinition): + core_source_def_cls = OracleSubselectCoreSourceDefinition + api_schema_cls = SubselectDataSourceSchema + template_api_schema_cls = SubselectDataSourceTemplateSchema + + +class OracleApiConnectionDefinition(ApiConnectionDefinition): + core_conn_def_cls = OracleCoreConnectionDefinition + api_generic_schema_cls = OracleConnectionSchema + info_provider_cls = OracleConnectionInfoProvider + form_factory_cls = OracleConnectionFormFactory + + +class OracleApiConnector(ApiConnector): + core_connector_cls = OracleCoreConnector + connection_definitions = (OracleApiConnectionDefinition,) + source_definitions = ( + OracleApiTableSourceDefinition, + OracleApiSubselectSourceDefinition, + ) + formula_dialect_name = DIALECT_NAME_ORACLE + translation_configs = frozenset(CONFIGS) + dashsql_literalizer_cls = OracleDashSQLParamLiteralizer diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/dashsql.py b/lib/dl_connector_oracle/dl_connector_oracle/api/dashsql.py new file mode 100644 index 000000000..213b8aa0d --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/api/dashsql.py @@ -0,0 +1,24 @@ +import sqlalchemy as sa +from sqlalchemy.types import TypeEngine + +from dl_api_connector.dashsql import ( + DefaultDashSQLParamLiteralizer, + TValueBase, +) +from dl_constants.enums import UserDataType + + +class OracleDashSQLParamLiteralizer(DefaultDashSQLParamLiteralizer): + def get_sa_type(self, bi_type: UserDataType, value_base: TValueBase) -> TypeEngine: + if bi_type == UserDataType.string: + # See also: dl_formula/definitions/literals.py + value_lst = [value_base] if isinstance(value_base, str) else value_base + max_len = max(len(val) for val in value_lst) + try: + for val in value_lst: + val.encode("ascii") + except UnicodeEncodeError: + return sa.NCHAR(max_len) + return sa.CHAR(max_len) + + return super().get_sa_type(bi_type=bi_type, value_base=value_base) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/i18n/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/api/i18n/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/api/i18n/localizer.py b/lib/dl_connector_oracle/dl_connector_oracle/api/i18n/localizer.py new file mode 100644 index 000000000..3c32e9a15 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/api/i18n/localizer.py @@ -0,0 +1,28 @@ +import os + +import attr + +from dl_i18n.localizer_base import Translatable as BaseTranslatable +from dl_i18n.localizer_base import TranslationConfig + +import dl_connector_oracle as package + + +DOMAIN = f"{package.__name__}" +CONFIGS = [ + TranslationConfig( + path=os.path.relpath(os.path.join(os.path.dirname(__file__), "../../locales")), + domain=DOMAIN, + locale="en", + ), + TranslationConfig( + path=os.path.relpath(os.path.join(os.path.dirname(__file__), "../../locales")), + domain=DOMAIN, + locale="ru", + ), +] + + +@attr.s +class Translatable(BaseTranslatable): + domain: str = attr.ib(default=DOMAIN) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/core/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/adapters_oracle.py b/lib/dl_connector_oracle/dl_connector_oracle/core/adapters_oracle.py new file mode 100644 index 000000000..a4b2bde3b --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/adapters_oracle.py @@ -0,0 +1,212 @@ +from __future__ import annotations + +from typing import ( + Any, + ClassVar, + Optional, + Tuple, + Type, +) + +import oracledb # type: ignore # TODO: fix +import sqlalchemy as sa +import sqlalchemy.dialects.oracle.base as sa_ora # not all data types are imported in init in older SA versions +from sqlalchemy.sql.type_api import TypeEngine + +from dl_core.connection_executors.adapters.adapters_base_sa_classic import ( + BaseClassicAdapter, + ClassicSQLConnLineConstructor, +) +from dl_core.connection_executors.models.db_adapter_data import DBAdapterQuery +from dl_core.connection_models import ( + DBIdent, + SchemaIdent, + TableIdent, +) +from dl_core.db.native_type import SATypeSpec + +from dl_connector_oracle.core.constants import CONNECTION_TYPE_ORACLE +from dl_connector_oracle.core.target_dto import OracleConnTargetDTO + + +class OracleConnLineConstructor(ClassicSQLConnLineConstructor[OracleConnTargetDTO]): + def _get_dsn_params( + self, + safe_db_symbols: Tuple[str, ...] = (), + db_name: Optional[str] = None, + standard_auth: Optional[bool] = True, + ) -> dict: + return dict( + super()._get_dsn_params( + safe_db_symbols=safe_db_symbols, + db_name=db_name, + standard_auth=standard_auth, + ), + db_name_type=self._target_dto.db_name_type.value.upper(), + ) + + +class OracleDefaultAdapter(BaseClassicAdapter[OracleConnTargetDTO]): + conn_type = CONNECTION_TYPE_ORACLE + conn_line_constructor_type: ClassVar[Type[OracleConnLineConstructor]] = OracleConnLineConstructor + + dsn_template = ( + "{dialect}://{user}:{passwd}@(DESCRIPTION=" + "(ADDRESS=(PROTOCOL=TCP)(HOST={host})(PORT={port}))" + "(CONNECT_DATA=({db_name_type}={db_name})))" + ) + + def _test(self) -> None: + self.execute(DBAdapterQuery("SELECT 1 FROM DUAL")).get_all() + + def _get_db_version(self, db_ident: DBIdent) -> Optional[str]: + return self.execute(DBAdapterQuery("SELECT * FROM V$VERSION", db_name=db_ident.db_name)).get_all()[0][0] + + def normalize_sa_col_type(self, sa_col_type: TypeEngine) -> TypeEngine: + if isinstance(sa_col_type, sa_ora.NUMBER) and not sa_col_type.scale: + return sa.Integer() + + return super().normalize_sa_col_type(sa_col_type) + + def _is_table_exists(self, table_ident: TableIdent) -> bool: + sa_exists_result = super()._is_table_exists(table_ident) + + # Dialect does not check for views so we will check it manually + if not sa_exists_result: + assert "'" not in table_ident.table_name # FIXME: quote the name properly + + rows = self.execute( + DBAdapterQuery( + f"SELECT view_name FROM all_views WHERE UPPER(view_name) = '{table_ident.table_name.upper()}'" + ) + ).get_all() + + return len(rows) > 0 + + return sa_exists_result + + _type_code_to_sa = { + oracledb.NUMBER: sa_ora.NUMBER, + oracledb.STRING: sa_ora.VARCHAR, # e.g. 'VARCHAR2(44)' + oracledb.NATIVE_FLOAT: sa_ora.BINARY_FLOAT, # ... or `sa_ora.BINARY_FLOAT` + oracledb.FIXED_CHAR: sa_ora.CHAR, # e.g. 'CHAR(1)' + oracledb.FIXED_NCHAR: sa_ora.NCHAR, # e.g. 'NCHAR(1)' + oracledb.NCHAR: sa_ora.NVARCHAR, # e.g. 'NVARCHAR(5)' + oracledb.DATETIME: sa_ora.DATE, + oracledb.TIMESTAMP: sa_ora.TIMESTAMP, # e.g. 'TIMESTAMP(9)', 'TIMESTAMP(9) WITH TIME ZONE' + # # For newer versions of oracledb, just in case. + # # Listed are all types from 8.1.0, + # # Enabled are types listed in `dl_core.db.conversions`. + # getattr(oracledb, 'DB_TYPE_BFILE', None): sa_ora.NULL, + getattr(oracledb, "DB_TYPE_BINARY_DOUBLE", None): sa_ora.BINARY_DOUBLE, + getattr(oracledb, "DB_TYPE_BINARY_FLOAT", None): sa_ora.BINARY_FLOAT, + # getattr(oracledb, 'DB_TYPE_BINARY_INTEGER', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_BLOB', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_BOOLEAN', None): sa_ora.NULL, + getattr(oracledb, "DB_TYPE_CHAR", None): sa_ora.CHAR, + # getattr(oracledb, 'DB_TYPE_CLOB', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_CURSOR', None): sa_ora.NULL, + getattr(oracledb, "DB_TYPE_DATE", None): sa_ora.DATE, + # getattr(oracledb, 'DB_TYPE_INTERVAL_DS', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_INTERVAL_YM', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_JSON', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_LONG', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_LONG_RAW', None): sa_ora.NULL, + getattr(oracledb, "DB_TYPE_NCHAR", None): sa_ora.NCHAR, + # getattr(oracledb, 'DB_TYPE_NCLOB', None): sa_ora.NULL, + getattr(oracledb, "DB_TYPE_NUMBER", None): sa_ora.NUMBER, + getattr(oracledb, "DB_TYPE_NVARCHAR", None): sa_ora.NVARCHAR, + # getattr(oracledb, 'DB_TYPE_OBJECT', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_RAW', None): sa_ora.NULL, + # getattr(oracledb, 'DB_TYPE_ROWID', None): sa_ora.NULL, + getattr(oracledb, "DB_TYPE_TIMESTAMP", None): sa_ora.TIMESTAMP, + # getattr(oracledb, 'DB_TYPE_TIMESTAMP_LTZ', None): sa_ora.NULL, + # # NOTE: not `timestamptz` here; matches the view schema, somehow. + getattr(oracledb, "DB_TYPE_TIMESTAMP_TZ", None): sa_ora.TIMESTAMP, + getattr(oracledb, "DB_TYPE_VARCHAR", None): sa_ora.VARCHAR, + } + + def _cursor_column_to_name(self, cursor_col, dialect=None) -> str: # type: ignore # TODO: fix + assert dialect, "required in this case" + # To match the `get_columns` result. + # Generally just lowercases the name. + # Notably, column names seem to be case-insensitive in oracle. + return dialect.normalize_name(cursor_col[0]) + + def _cursor_column_to_sa(self, cursor_col, require: bool = True) -> Optional[SATypeSpec]: # type: ignore # TODO: fix + """ + cursor_col: + + name, type, display_size, internal_size, precision, scale, null_ok + + See also: + https://cx-oracle.readthedocs.io/en/latest/api_manual/cursor.html#Cursor.description + """ + type_code = cursor_col[1] + if type_code is None: # shouldn't really happen + if require: + raise ValueError(f"Cursor column has no type: {cursor_col!r}") + return None + sa_cls = self._type_code_to_sa.get(type_code) + if sa_cls is None: + if require: + raise ValueError(f"Unknown cursor type: {type_code!r}") + return None + + # It would be nice to distinguish timestamp/timestamptz here, but the + # only observed difference is `display_size=23` for tz-*naive* + # timestamp. + + if sa_cls is sa_ora.NUMBER: + # See also: `self.normalize_sa_col_type` + precision = cursor_col[4] + scale = cursor_col[5] + + # Going by the comparison with the 'create view' -> SA logic. + if scale == -127: + scale = 0 + + sa_type = sa_cls(precision, scale) + else: + sa_type = sa_cls + + return sa_type + + def _cursor_column_to_nullable(self, cursor_col) -> Optional[bool]: # type: ignore # TODO: fix + return bool(cursor_col[6]) + + def _make_cursor_info(self, cursor, db_session=None) -> dict: # type: ignore # TODO: fix + return dict( + super()._make_cursor_info(cursor, db_session=db_session), + cxoracle_types=[self._cursor_type_to_str(column[1]) for column in cursor.description], + ) + + ORACLE_LIST_SOURCES_ALL_SCHEMA_SQL = """ + SELECT OWNER, TABLE_NAME FROM ALL_TABLES + WHERE nvl(tablespace_name, 'no tablespace') + NOT IN ('SYSTEM', 'SYSAUX') + AND IOT_NAME IS NULL + AND DURATION IS NULL + """ + + def _get_tables(self, schema_ident: SchemaIdent) -> list[TableIdent]: + if schema_ident.schema_name is not None: + return super()._get_tables(schema_ident) + + db_name = schema_ident.db_name + db_engine = self.get_db_engine(db_name) + + query = self.ORACLE_LIST_SOURCES_ALL_SCHEMA_SQL + result = db_engine.execute(sa.text(query)) + return [ + TableIdent( + db_name=db_name, + schema_name=owner, + table_name=table_name, + ) + for owner, table_name in result + ] + + @staticmethod + def _cursor_type_to_str(value: Any) -> str: + return value.name.lower() diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/connection_executors.py b/lib/dl_connector_oracle/dl_connector_oracle/core/connection_executors.py new file mode 100644 index 000000000..079f6d0b4 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/connection_executors.py @@ -0,0 +1,36 @@ +from __future__ import annotations + +from typing import List + +import attr + +from dl_core.connection_executors.async_sa_executors import DefaultSqlAlchemyConnExecutor + +from dl_connector_oracle.core.adapters_oracle import OracleDefaultAdapter +from dl_connector_oracle.core.dto import OracleConnDTO +from dl_connector_oracle.core.target_dto import OracleConnTargetDTO + + +@attr.s(cmp=False, hash=False) +class OracleDefaultConnExecutor(DefaultSqlAlchemyConnExecutor[OracleDefaultAdapter]): + TARGET_ADAPTER_CLS = OracleDefaultAdapter + + _conn_dto: OracleConnDTO = attr.ib() + + async def _make_target_conn_dto_pool(self) -> List[OracleConnTargetDTO]: # type: ignore # TODO: fix + dto_pool = [] + for host in self._conn_hosts_pool: + dto_pool.append( + OracleConnTargetDTO( + conn_id=self._conn_dto.conn_id, + pass_db_messages_to_user=self._conn_options.pass_db_messages_to_user, + pass_db_query_to_user=self._conn_options.pass_db_query_to_user, + host=host, + port=self._conn_dto.port, + db_name=self._conn_dto.db_name, + db_name_type=self._conn_dto.db_name_type, + username=self._conn_dto.username, + password=self._conn_dto.password, + ) + ) + return dto_pool diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/connector.py b/lib/dl_connector_oracle/dl_connector_oracle/core/connector.py new file mode 100644 index 000000000..0d7ae59f9 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/connector.py @@ -0,0 +1,58 @@ +from dl_core.connectors.base.connector import ( + CoreConnectionDefinition, + CoreConnector, +) +from dl_core.connectors.sql_base.connector import ( + SQLSubselectCoreSourceDefinitionBase, + SQLTableCoreSourceDefinitionBase, +) + +from dl_connector_oracle.core.adapters_oracle import OracleDefaultAdapter +from dl_connector_oracle.core.connection_executors import OracleDefaultConnExecutor +from dl_connector_oracle.core.constants import ( + BACKEND_TYPE_ORACLE, + CONNECTION_TYPE_ORACLE, + SOURCE_TYPE_ORACLE_SUBSELECT, + SOURCE_TYPE_ORACLE_TABLE, +) +from dl_connector_oracle.core.data_source import ( + OracleDataSource, + OracleSubselectDataSource, +) +from dl_connector_oracle.core.data_source_migration import OracleDataSourceMigrator +from dl_connector_oracle.core.sa_types import SQLALCHEMY_ORACLE_TYPES +from dl_connector_oracle.core.storage_schemas.connection import ConnectionSQLOracleDataStorageSchema +from dl_connector_oracle.core.type_transformer import OracleServerTypeTransformer +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle + + +class OracleCoreConnectionDefinition(CoreConnectionDefinition): + conn_type = CONNECTION_TYPE_ORACLE + connection_cls = ConnectionSQLOracle + us_storage_schema_cls = ConnectionSQLOracleDataStorageSchema + type_transformer_cls = OracleServerTypeTransformer + sync_conn_executor_cls = OracleDefaultConnExecutor + async_conn_executor_cls = OracleDefaultConnExecutor + dialect_string = "bi_oracle" + data_source_migrator_cls = OracleDataSourceMigrator + + +class OracleTableCoreSourceDefinition(SQLTableCoreSourceDefinitionBase): + source_type = SOURCE_TYPE_ORACLE_TABLE + source_cls = OracleDataSource + + +class OracleSubselectCoreSourceDefinition(SQLSubselectCoreSourceDefinitionBase): + source_type = SOURCE_TYPE_ORACLE_SUBSELECT + source_cls = OracleSubselectDataSource + + +class OracleCoreConnector(CoreConnector): + backend_type = BACKEND_TYPE_ORACLE + connection_definitions = (OracleCoreConnectionDefinition,) + source_definitions = ( + OracleTableCoreSourceDefinition, + OracleSubselectCoreSourceDefinition, + ) + rqe_adapter_classes = frozenset({OracleDefaultAdapter}) + sa_types = SQLALCHEMY_ORACLE_TYPES diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/constants.py b/lib/dl_connector_oracle/dl_connector_oracle/core/constants.py new file mode 100644 index 000000000..becbd9b49 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/constants.py @@ -0,0 +1,22 @@ +from enum import ( + Enum, + unique, +) + +from dl_constants.enums import ( + ConnectionType, + DataSourceType, + SourceBackendType, +) + + +BACKEND_TYPE_ORACLE = SourceBackendType.declare("ORACLE") +CONNECTION_TYPE_ORACLE = ConnectionType.declare("oracle") +SOURCE_TYPE_ORACLE_TABLE = DataSourceType.declare("ORACLE_TABLE") +SOURCE_TYPE_ORACLE_SUBSELECT = DataSourceType.declare("ORACLE_SUBSELECT") + + +@unique +class OracleDbNameType(Enum): + sid = "sid" + service_name = "service_name" diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/data_source.py b/lib/dl_connector_oracle/dl_connector_oracle/core/data_source.py new file mode 100644 index 000000000..31d422620 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/data_source.py @@ -0,0 +1,54 @@ +from __future__ import annotations + +import logging +from typing import ( + Any, + Optional, +) + +from dl_constants.enums import DataSourceType +from dl_core.data_source.sql import ( + BaseSQLDataSource, + StandardSchemaSQLDataSource, + SubselectDataSource, + require_table_name, +) +from dl_core.utils import sa_plain_text + +from dl_connector_oracle.core.constants import ( + CONNECTION_TYPE_ORACLE, + SOURCE_TYPE_ORACLE_SUBSELECT, + SOURCE_TYPE_ORACLE_TABLE, +) +from dl_connector_oracle.core.query_compiler import OracleQueryCompiler + + +LOGGER = logging.getLogger(__name__) + + +class OracleDataSourceMixin(BaseSQLDataSource): + compiler_cls = OracleQueryCompiler + + conn_type = CONNECTION_TYPE_ORACLE + + @classmethod + def is_compatible_with_type(cls, source_type: DataSourceType) -> bool: + return source_type in (SOURCE_TYPE_ORACLE_TABLE, SOURCE_TYPE_ORACLE_SUBSELECT) + + +class OracleDataSource(OracleDataSourceMixin, StandardSchemaSQLDataSource): + """Oracle table""" + + @require_table_name + def get_sql_source(self, alias: Optional[str] = None) -> Any: + q = self.quote + alias_str = "" if alias is None else f" {q(alias)}" + schema_str = "" if self.schema_name is None else f"{q(self.schema_name)}." + return sa_plain_text(f"{schema_str}{q(self.table_name)}{alias_str}") + + +class OracleSubselectDataSource(OracleDataSourceMixin, SubselectDataSource): + """Oracle subselect""" + + # In oracle, `(select …) as source` doesn't work, only `(select …) source`. + _subquery_alias_joiner = " " diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/data_source_migration.py b/lib/dl_connector_oracle/dl_connector_oracle/core/data_source_migration.py new file mode 100644 index 000000000..7e249c710 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/data_source_migration.py @@ -0,0 +1,11 @@ +from dl_core.connectors.sql_base.data_source_migration import DefaultSQLDataSourceMigrator + +from dl_connector_oracle.core.constants import ( + SOURCE_TYPE_ORACLE_SUBSELECT, + SOURCE_TYPE_ORACLE_TABLE, +) + + +class OracleDataSourceMigrator(DefaultSQLDataSourceMigrator): + table_source_type = SOURCE_TYPE_ORACLE_TABLE + subselect_source_type = SOURCE_TYPE_ORACLE_SUBSELECT diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/dto.py b/lib/dl_connector_oracle/dl_connector_oracle/core/dto.py new file mode 100644 index 000000000..ed8640e8e --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/dto.py @@ -0,0 +1,15 @@ +import attr + +from dl_core.connection_models.dto_defs import DefaultSQLDTO + +from dl_connector_oracle.core.constants import ( + CONNECTION_TYPE_ORACLE, + OracleDbNameType, +) + + +@attr.s(frozen=True) +class OracleConnDTO(DefaultSQLDTO): # noqa + conn_type = CONNECTION_TYPE_ORACLE + + db_name_type: OracleDbNameType = attr.ib(kw_only=True) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/query_compiler.py b/lib/dl_connector_oracle/dl_connector_oracle/core/query_compiler.py new file mode 100644 index 000000000..1fe9c689a --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/query_compiler.py @@ -0,0 +1,7 @@ +from __future__ import annotations + +from dl_core.connectors.base.query_compiler import QueryCompiler + + +class OracleQueryCompiler(QueryCompiler): + force_nulls_lower_than_values = True diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/sa_types.py b/lib/dl_connector_oracle/dl_connector_oracle/core/sa_types.py new file mode 100644 index 000000000..6460f09bb --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/sa_types.py @@ -0,0 +1,41 @@ +from __future__ import annotations + +import sqlalchemy as sa +from sqlalchemy.dialects.oracle import base as or_types # not all data types are imported in init in older SA versions + +from dl_core.db.sa_types_base import ( + lengthed_instantiator, + make_native_type, + simple_instantiator, +) + +from dl_connector_oracle.core.constants import CONNECTION_TYPE_ORACLE + + +SQLALCHEMY_ORACLE_BASE_TYPES = ( + or_types.NUMBER, + or_types.BINARY_FLOAT, + or_types.BINARY_DOUBLE, + or_types.DATE, + or_types.TIMESTAMP, +) +SQLALCHEMY_ORACLE_LENGTHED_TYPES = ( + or_types.CHAR, + sa.NCHAR, + or_types.VARCHAR, + or_types.NVARCHAR, + or_types.VARCHAR2, + or_types.NVARCHAR2, +) +SQLALCHEMY_ORACLE_TYPES = { + **{ + make_native_type(CONNECTION_TYPE_ORACLE, typecls): simple_instantiator(typecls) + for typecls in SQLALCHEMY_ORACLE_BASE_TYPES + }, + # A tricky substitution (possibly in the wrong place): + make_native_type(CONNECTION_TYPE_ORACLE, sa.Integer): simple_instantiator(or_types.NUMBER), + **{ + make_native_type(CONNECTION_TYPE_ORACLE, typecls): lengthed_instantiator(typecls) + for typecls in SQLALCHEMY_ORACLE_LENGTHED_TYPES + }, +} diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/storage_schemas/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/core/storage_schemas/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/storage_schemas/connection.py b/lib/dl_connector_oracle/dl_connector_oracle/core/storage_schemas/connection.py new file mode 100644 index 000000000..be5b6d1ab --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/storage_schemas/connection.py @@ -0,0 +1,14 @@ +from marshmallow import fields as ma_fields + +from dl_core.us_manager.storage_schemas.connection import ConnectionSQLDataStorageSchema + +from dl_connector_oracle.core.constants import OracleDbNameType +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle + + +class ConnectionSQLOracleDataStorageSchema( + ConnectionSQLDataStorageSchema[ConnectionSQLOracle.DataModel], +): + TARGET_CLS = ConnectionSQLOracle.DataModel + + db_name_type = ma_fields.Enum(OracleDbNameType, required=True, allow_none=False) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/target_dto.py b/lib/dl_connector_oracle/dl_connector_oracle/core/target_dto.py new file mode 100644 index 000000000..2c461ce53 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/target_dto.py @@ -0,0 +1,37 @@ +from __future__ import annotations + +from typing import ( + TYPE_CHECKING, + Type, + TypeVar, +) + +import attr + +from dl_core.connection_executors.models.connection_target_dto_base import BaseSQLConnTargetDTO + +from dl_connector_oracle.core.constants import OracleDbNameType + + +if TYPE_CHECKING: + from dl_constants.types import TJSONLike + + +_CT_DTO_TV = TypeVar("_CT_DTO_TV", bound="OracleConnTargetDTO") + + +@attr.s(frozen=True) +class OracleConnTargetDTO(BaseSQLConnTargetDTO): + db_name_type: OracleDbNameType = attr.ib() + + def to_jsonable_dict(self) -> dict[str, TJSONLike]: + d = super().to_jsonable_dict() + return { + **d, + "db_name_type": self.db_name_type.name, + } + + @classmethod + def _from_jsonable_dict(cls: Type[_CT_DTO_TV], data: dict) -> _CT_DTO_TV: + prepared_data = {**data, "db_name_type": OracleDbNameType[data["db_name_type"]]} + return cls(**prepared_data) # type: ignore diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/testing/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/core/testing/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/testing/connection.py b/lib/dl_connector_oracle/dl_connector_oracle/core/testing/connection.py new file mode 100644 index 000000000..25de30a15 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/testing/connection.py @@ -0,0 +1,40 @@ +from typing import ( + Any, + Optional, +) +import uuid + +from dl_constants.enums import RawSQLLevel +from dl_core.us_manager.us_manager_sync import SyncUSManager + +from dl_connector_oracle.core.constants import CONNECTION_TYPE_ORACLE +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle + + +def make_oracle_saved_connection( + sync_usm: SyncUSManager, + db_name: Optional[str], + host: str, + port: Optional[int], + username: Optional[str], + password: Optional[str], + raw_sql_level: RawSQLLevel = RawSQLLevel.off, + **kwargs: Any, +) -> ConnectionSQLOracle: + conn_name = "oracle test conn {}".format(uuid.uuid4()) + conn = ConnectionSQLOracle.create_from_dict( + data_dict=ConnectionSQLOracle.DataModel( + db_name=db_name, + host=host, + port=port, + username=username, + password=password, + raw_sql_level=raw_sql_level, + ), + ds_key=conn_name, + type_=CONNECTION_TYPE_ORACLE.name, + us_manager=sync_usm, + **kwargs, + ) + sync_usm.save(conn) + return conn diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/type_transformer.py b/lib/dl_connector_oracle/dl_connector_oracle/core/type_transformer.py new file mode 100644 index 000000000..7bebdf6ab --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/type_transformer.py @@ -0,0 +1,56 @@ +import sqlalchemy as sa +from sqlalchemy.dialects.oracle import base as or_types # not all data types are imported in init in older SA versions + +from dl_constants.enums import UserDataType +from dl_core.db.conversion_base import ( + TypeTransformer, + make_native_type, +) + +from dl_connector_oracle.core.constants import CONNECTION_TYPE_ORACLE + + +class OracleServerTypeTransformer(TypeTransformer): + conn_type = CONNECTION_TYPE_ORACLE + native_to_user_map = { + # No separate type for integer. Number acts as DECIMAL with customizable precision. + **{ + make_native_type(CONNECTION_TYPE_ORACLE, t): UserDataType.integer + for t in (sa.Integer,) # pseudo type used if scale == 0 + }, + **{ + make_native_type(CONNECTION_TYPE_ORACLE, t): UserDataType.float # type: ignore # TODO: fix + for t in (or_types.NUMBER, or_types.BINARY_FLOAT, or_types.BINARY_DOUBLE) + }, + **{ + make_native_type(CONNECTION_TYPE_ORACLE, t): UserDataType.string + for t in ( + or_types.CHAR, + or_types.VARCHAR, + or_types.VARCHAR2, + sa.NCHAR, + or_types.NVARCHAR, + or_types.NVARCHAR2, + ) + }, + **{ + make_native_type(CONNECTION_TYPE_ORACLE, t): UserDataType.genericdatetime # type: ignore # TODO: fix + for t in (or_types.DATE, or_types.TIMESTAMP) + }, + # No separate type for date, it's the same as for datetime + make_native_type(CONNECTION_TYPE_ORACLE, sa.sql.sqltypes.NullType): UserDataType.unsupported, + } + user_to_native_map = { + UserDataType.integer: make_native_type(CONNECTION_TYPE_ORACLE, sa.INTEGER), + UserDataType.float: make_native_type(CONNECTION_TYPE_ORACLE, or_types.BINARY_DOUBLE), + UserDataType.boolean: make_native_type(CONNECTION_TYPE_ORACLE, or_types.NUMBER), + UserDataType.string: make_native_type(CONNECTION_TYPE_ORACLE, or_types.NVARCHAR2), + UserDataType.date: make_native_type(CONNECTION_TYPE_ORACLE, or_types.DATE), + UserDataType.datetime: make_native_type(CONNECTION_TYPE_ORACLE, or_types.DATE), + UserDataType.genericdatetime: make_native_type(CONNECTION_TYPE_ORACLE, or_types.DATE), + UserDataType.geopoint: make_native_type(CONNECTION_TYPE_ORACLE, or_types.NVARCHAR2), + UserDataType.geopolygon: make_native_type(CONNECTION_TYPE_ORACLE, or_types.NVARCHAR2), + UserDataType.uuid: make_native_type(CONNECTION_TYPE_ORACLE, or_types.NVARCHAR2), + UserDataType.markup: make_native_type(CONNECTION_TYPE_ORACLE, or_types.NVARCHAR2), + UserDataType.unsupported: make_native_type(CONNECTION_TYPE_ORACLE, sa.sql.sqltypes.NullType), + } diff --git a/lib/dl_connector_oracle/dl_connector_oracle/core/us_connection.py b/lib/dl_connector_oracle/dl_connector_oracle/core/us_connection.py new file mode 100644 index 000000000..bcf01e118 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/core/us_connection.py @@ -0,0 +1,70 @@ +from __future__ import annotations + +from typing import ( + Callable, + ClassVar, +) + +import attr + +from dl_core.connection_executors.sync_base import SyncConnExecutorBase +from dl_core.us_connection_base import ( + ClassicConnectionSQL, + ConnectionBase, + DataSourceTemplate, +) +from dl_i18n.localizer_base import Localizer + +from dl_connector_oracle.core.constants import ( + SOURCE_TYPE_ORACLE_SUBSELECT, + SOURCE_TYPE_ORACLE_TABLE, + OracleDbNameType, +) +from dl_connector_oracle.core.dto import OracleConnDTO + + +class ConnectionSQLOracle(ClassicConnectionSQL): + has_schema: ClassVar[bool] = True + # Default schema is usually defined on a per-user basis, + # so it's better to omit the schema if it isn't explicitly specified. + default_schema_name = None + source_type = SOURCE_TYPE_ORACLE_TABLE + allowed_source_types = frozenset((SOURCE_TYPE_ORACLE_TABLE, SOURCE_TYPE_ORACLE_SUBSELECT)) + allow_dashsql: ClassVar[bool] = True + allow_cache: ClassVar[bool] = True + is_always_user_source: ClassVar[bool] = True + + @attr.s(kw_only=True) + class DataModel(ClassicConnectionSQL.DataModel): + db_name_type: OracleDbNameType = attr.ib(default=OracleDbNameType.service_name) + + def get_conn_dto(self) -> OracleConnDTO: + return OracleConnDTO( + conn_id=self.uuid, + host=self.data.host, + multihosts=self.parse_multihosts(), # type: ignore # TODO: fix + port=self.data.port, + db_name=self.data.db_name, + db_name_type=self.data.db_name_type, + username=self.data.username, + password=self.password, + ) + + def get_data_source_template_templates(self, localizer: Localizer) -> list[DataSourceTemplate]: + return self._make_subselect_templates(source_type=SOURCE_TYPE_ORACLE_SUBSELECT, localizer=localizer) + + def get_parameter_combinations( + self, conn_executor_factory: Callable[[ConnectionBase], SyncConnExecutorBase] + ) -> list[dict]: + if not self.db_name: + return [] + + assert self.has_schema + return [ + dict(schema_name=tid.schema_name, table_name=tid.table_name) + for tid in self.get_tables(schema_name=None, conn_executor_factory=conn_executor_factory) + ] + + @property + def allow_public_usage(self) -> bool: + return True diff --git a/lib/dl_connector_oracle/dl_connector_oracle/db_testing/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/db_testing/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/db_testing/connector.py b/lib/dl_connector_oracle/dl_connector_oracle/db_testing/connector.py new file mode 100644 index 000000000..8555bc2eb --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/db_testing/connector.py @@ -0,0 +1,7 @@ +from dl_db_testing.connectors.base.connector import DbTestingConnector + +from dl_connector_oracle.db_testing.engine_wrapper import OracleEngineWrapper + + +class OracleDbTestingConnector(DbTestingConnector): + engine_wrapper_classes = (OracleEngineWrapper,) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/db_testing/engine_wrapper.py b/lib/dl_connector_oracle/dl_connector_oracle/db_testing/engine_wrapper.py new file mode 100644 index 000000000..4cce9b09e --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/db_testing/engine_wrapper.py @@ -0,0 +1,55 @@ +import re +from typing import ( + Any, + Optional, + Sequence, +) + +import sqlalchemy as sa + +from dl_db_testing.database.engine_wrapper import EngineWrapperBase + + +class OracleEngineWrapper(EngineWrapperBase): + URL_PREFIX = "oracle" # Not using the bi_* version because we only need basic functionality here + + def execute(self, query: Any, *multiparams: Any, **params: Any): # type: ignore # TODO: fix + # FIXME: Note the following problem: + # https://stackoverflow.com/questions/54709396/incorrect-type-conversion-with-cx-oracle-and-sqlalchemy-queries + + return super().execute(query, *multiparams, **params) + + def count_sql_sessions(self) -> int: + # noinspection SqlDialectInspection + cur = self.execute("SELECT * FROM sys.V_$SESSION") + try: + lines = cur.fetchall() + return len(lines) + finally: + cur.close() + + def get_conn_credentials(self, full: bool = False) -> dict: + cred_prefix, oracle_dsn = str(self.url).split("@") + username, password = cred_prefix.split("//")[1].split(":") + match = re.search( + r"HOST=(?P[^)]+)\).*PORT=(?P\d+)\).*(SERVICE_NAME|SID)=(?P[^)]+)\)", oracle_dsn + ) + return dict( + host=match.group("host"), # type: ignore # TODO: fix + port=int(match.group("port")), # type: ignore # TODO: fix + username=username, + password=password, + db_name=match.group("db_name"), # type: ignore # TODO: fix + ) + + def insert_into_table(self, table: sa.Table, data: Sequence[dict]) -> None: + # Multi-row insert doesn't work correctly + for row in data: + self.execute(table.insert(row)) + + def create_schema(self, schema_name: str) -> None: + self.execute(f"CREATE USER {self.quote(schema_name)} IDENTIFIED BY qwerty") + self.execute(f"GRANT ALL PRIVILEGES TO {self.quote(schema_name)}") + + def get_version(self) -> Optional[str]: + return self.execute("SELECT * FROM V$VERSION").scalar() diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/connector.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/connector.py new file mode 100644 index 000000000..115218ca6 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/connector.py @@ -0,0 +1,27 @@ +from sqlalchemy.dialects.oracle.base import OracleDialect as SAOracleDialect + +from dl_formula.connectors.base.connector import FormulaConnector +from dl_formula.mutation.general import OptimizeUnaryBoolFunctions + +from dl_connector_oracle.formula.constants import OracleDialect as OracleDialectNS +from dl_connector_oracle.formula.context_processor import OracleContextPostprocessor +from dl_connector_oracle.formula.definitions.all import DEFINITIONS +from dl_connector_oracle.formula.literal import OracleLiteralizer +from dl_connector_oracle.formula.type_constructor import OracleTypeConstructor + + +class OracleFormulaConnector(FormulaConnector): + dialect_ns_cls = OracleDialectNS + dialects = OracleDialectNS.ORACLE + default_dialect = OracleDialectNS.ORACLE + op_definitions = DEFINITIONS + literalizer_cls = OracleLiteralizer + context_processor_cls = OracleContextPostprocessor + type_constructor_cls = OracleTypeConstructor + sa_dialect = SAOracleDialect() + + @classmethod + def registration_hook(cls) -> None: + OptimizeUnaryBoolFunctions.register_dialect( + "isnull", cls.dialects, f=lambda x: x is None or (isinstance(x, str) and x == "") + ) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/constants.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/constants.py new file mode 100644 index 000000000..eef590a00 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/constants.py @@ -0,0 +1,13 @@ +from dl_formula.core.dialect import ( + DialectName, + DialectNamespace, + simple_combo, +) + + +DIALECT_NAME_ORACLE = DialectName.declare("ORACLE") + + +class OracleDialect(DialectNamespace): + ORACLE_12_0 = simple_combo(name=DIALECT_NAME_ORACLE, version=(12, 0)) + ORACLE = ORACLE_12_0 diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/context_processor.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/context_processor.py new file mode 100644 index 000000000..bec8a8930 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/context_processor.py @@ -0,0 +1,31 @@ +from typing import cast + +import sqlalchemy as sa +from sqlalchemy.sql.elements import ClauseElement + +from dl_formula.connectors.base.context_processor import BooleanlessContextPostprocessor +from dl_formula.core.datatype import DataType + + +class OracleContextPostprocessor(BooleanlessContextPostprocessor): + def booleanize_expression(self, data_type: DataType, expression: ClauseElement) -> ClauseElement: + if data_type in ( + DataType.CONST_BOOLEAN, + DataType.CONST_INTEGER, + DataType.CONST_FLOAT, + DataType.BOOLEAN, + DataType.INTEGER, + DataType.FLOAT, + ): + expression = cast(ClauseElement, expression != 0) + elif data_type in (DataType.CONST_STRING, DataType.STRING): + expression = expression.isnot(None) + elif data_type in (DataType.CONST_DATE, DataType.CONST_DATETIME, DataType.DATE, DataType.DATETIME): + expression = cast(ClauseElement, sa.literal(1) == 1) + else: + expression = cast(ClauseElement, expression == 1) + + return expression + + def debooleanize_expression(self, data_type: DataType, expression: ClauseElement) -> ClauseElement: + return sa.case(whens=[(expression, sa.literal(1))], else_=sa.literal(0)) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/all.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/all.py new file mode 100644 index 000000000..672d63fb6 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/all.py @@ -0,0 +1,26 @@ +from dl_connector_oracle.formula.definitions.conditional_blocks import DEFINITIONS_COND_BLOCKS +from dl_connector_oracle.formula.definitions.functions_aggregation import DEFINITIONS_AGG +from dl_connector_oracle.formula.definitions.functions_datetime import DEFINITIONS_DATETIME +from dl_connector_oracle.formula.definitions.functions_logical import DEFINITIONS_LOGICAL +from dl_connector_oracle.formula.definitions.functions_markup import DEFINITIONS_MARKUP +from dl_connector_oracle.formula.definitions.functions_math import DEFINITIONS_MATH +from dl_connector_oracle.formula.definitions.functions_string import DEFINITIONS_STRING +from dl_connector_oracle.formula.definitions.functions_type import DEFINITIONS_TYPE +from dl_connector_oracle.formula.definitions.operators_binary import DEFINITIONS_BINARY +from dl_connector_oracle.formula.definitions.operators_ternary import DEFINITIONS_TERNARY +from dl_connector_oracle.formula.definitions.operators_unary import DEFINITIONS_UNARY + + +DEFINITIONS = [ + *DEFINITIONS_COND_BLOCKS, + *DEFINITIONS_AGG, + *DEFINITIONS_DATETIME, + *DEFINITIONS_LOGICAL, + *DEFINITIONS_MARKUP, + *DEFINITIONS_MATH, + *DEFINITIONS_STRING, + *DEFINITIONS_TYPE, + *DEFINITIONS_UNARY, + *DEFINITIONS_BINARY, + *DEFINITIONS_TERNARY, +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/conditional_blocks.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/conditional_blocks.py new file mode 100644 index 000000000..5c3d08b17 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/conditional_blocks.py @@ -0,0 +1,16 @@ +from dl_formula.definitions.base import TranslationVariant +import dl_formula.definitions.conditional_blocks as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_COND_BLOCKS = [ + # _case_block_ + base.CaseBlock.for_dialect(D.ORACLE), + # _if_block_ + base.IfBlock3.for_dialect(D.ORACLE), + base.IfBlockMulti.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_aggregation.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_aggregation.py new file mode 100644 index 000000000..7955f4aa3 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_aggregation.py @@ -0,0 +1,71 @@ +import sqlalchemy as sa + +from dl_formula.definitions.base import TranslationVariant +from dl_formula.definitions.common import ( + quantile_value, + within_group, +) +import dl_formula.definitions.functions_aggregation as base +from dl_formula.definitions.literals import un_literal + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_AGG = [ + # avg + base.AggAvgFromNumber.for_dialect(D.ORACLE), + base.AggAvgFromDate.for_dialect(D.ORACLE), + base.AggAvgFromDatetime.for_dialect(D.ORACLE), + base.AggAvgFromDatetimeTZ.for_dialect(D.ORACLE), + # avg_if + base.AggAvgIf.for_dialect(D.ORACLE), + # count + base.AggCount0.for_dialect(D.ORACLE), + base.AggCount1.for_dialect(D.ORACLE), + # count_if + base.AggCountIf.for_dialect(D.ORACLE), + # countd + base.AggCountd.for_dialect(D.ORACLE), + # countd_approx + base.AggCountdApprox( + variants=[ + V(D.ORACLE_12_0, sa.func.APPROX_COUNT_DISTINCT), + ] + ), + # countd_if + base.AggCountdIf.for_dialect(D.ORACLE), + # max + base.AggMax.for_dialect(D.ORACLE), + # median + base.AggMedian( + variants=[ + V(D.ORACLE, lambda expr: within_group(sa.func.PERCENTILE_DISC(0.5), expr)), + ] + ), + # min + base.AggMin.for_dialect(D.ORACLE), + # quantile + base.AggQuantile( + variants=[ + V( + D.ORACLE, + lambda expr, quant: within_group(sa.func.percentile_disc(quantile_value(un_literal(quant))), expr), + ), + ] + ), + # stdev + base.AggStdev.for_dialect(D.ORACLE), + # stdevp + base.AggStdevp.for_dialect(D.ORACLE), + # sum + base.AggSum.for_dialect(D.ORACLE), + # sum_if + base.AggSumIf.for_dialect(D.ORACLE), + # var + base.AggVar.for_dialect(D.ORACLE), + # varp + base.AggVarp.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_datetime.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_datetime.py new file mode 100644 index 000000000..c317eb6cb --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_datetime.py @@ -0,0 +1,175 @@ +from __future__ import annotations + +import sqlalchemy as sa + +from dl_formula.core.datatype import DataType +from dl_formula.definitions.args import ArgTypeSequence +from dl_formula.definitions.base import TranslationVariant +from dl_formula.definitions.common import raw_sql +from dl_formula.definitions.common_datetime import ( + EPOCH_START_D, + EPOCH_START_DOW, + datetime_interval, +) +import dl_formula.definitions.functions_datetime as base +from dl_formula.definitions.literals import ( + literal, + un_literal, +) + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +class FuncDatetrunc2Oracle(base.FuncDatetrunc2): + _oracle_fmt_map = { + "minute": "MI", + "hour": "HH", + "day": "DDD", + "week": "IW", + "month": "MM", + "quarter": "Q", + "year": "YYYY", + } + + variants = [ + V( + D.ORACLE, + lambda date, unit: ( + sa.func.TRUNC( + date, literal(FuncDatetrunc2Oracle._oracle_fmt_map[base.norm_datetrunc_unit(unit)], d=D.ORACLE) + ) + if base.norm_datetrunc_unit(unit) in FuncDatetrunc2Oracle._oracle_fmt_map + else date + ), + ), + ] + argument_types = [ + ArgTypeSequence( + [ + {DataType.DATE, DataType.DATETIME, DataType.GENERICDATETIME}, # TODO: DataType.DATETIMETZ + DataType.CONST_STRING, + ] + ), + ] + + +DEFINITIONS_DATETIME = [ + # dateadd + base.FuncDateadd1.for_dialect(D.ORACLE), + base.FuncDateadd2Unit.for_dialect(D.ORACLE), + base.FuncDateadd2Number.for_dialect(D.ORACLE), + base.FuncDateadd3Legacy.for_dialect(D.ORACLE), + base.FuncDateadd3DateConstNum( + variants=[ + V( + D.ORACLE, + lambda date, what, num: ( + sa.cast(date + datetime_interval(un_literal(what), un_literal(num), literal_mult=True), sa.Date) + ), + ), + ] + ), + base.FuncDateadd3DatetimeConstNum( + variants=[ + V( + D.ORACLE, + lambda dt, what, num: ( + sa.cast(dt + datetime_interval(what.value, num.value, literal_mult=True), sa.Date) + ), + ), + ] + ), + # datepart + base.FuncDatepart2Legacy.for_dialect(D.ORACLE), + base.FuncDatepart2.for_dialect(D.ORACLE), + base.FuncDatepart3Const.for_dialect(D.ORACLE), + base.FuncDatepart3NonConst.for_dialect(D.ORACLE), + # datetrunc + FuncDatetrunc2Oracle(), + # day + base.FuncDay( + variants=[ + V(D.ORACLE, lambda date: sa.func.extract(raw_sql("DAY"), date)), + ] + ), + # dayofweek + base.FuncDayofweek1.for_dialect(D.ORACLE), + base.FuncDayofweek2( + variants=[ + V( + D.ORACLE, + lambda date, firstday: sa.cast( + date - EPOCH_START_D - (EPOCH_START_DOW + base.norm_fd(firstday) - 1), sa.Integer + ) + % 7 + + 1, + ), + ] + ), + # genericnow + base.FuncGenericNow( + variants=[ + V(D.ORACLE, lambda: sa.type_coerce(sa.cast(raw_sql("SYSTIMESTAMP"), sa.Date), sa.DateTime)), + ] + ), + # hour + base.FuncHourDate.for_dialect(D.ORACLE), + base.FuncHourDatetime( + variants=[ + V(D.ORACLE, lambda date: sa.func.extract(raw_sql("HOUR"), sa.cast(date, sa.TIMESTAMP))), + ] + ), + # minute + base.FuncMinuteDate.for_dialect(D.ORACLE), + base.FuncMinuteDatetime( + variants=[ + V(D.ORACLE, lambda date: sa.func.extract(raw_sql("MINUTE"), sa.cast(date, sa.TIMESTAMP))), + ] + ), + # month + base.FuncMonth( + variants=[ + V(D.ORACLE, lambda date: sa.func.extract(raw_sql("MONTH"), date)), + ] + ), + # now + base.FuncNow( + variants=[ + V(D.ORACLE, lambda: sa.type_coerce(sa.cast(raw_sql("SYSTIMESTAMP"), sa.Date), sa.DateTime)), + ] + ), + # quarter + base.FuncQuarter( + variants=[ + V(D.ORACLE, lambda date: sa.func.TRUNC((sa.func.extract(raw_sql("MONTH"), date) + 2) / 3)), + ] + ), + # second + base.FuncSecondDate.for_dialect(D.ORACLE), + base.FuncSecondDatetime( + variants=[ + V(D.ORACLE, lambda date: sa.func.extract(raw_sql("SECOND"), sa.cast(date, sa.TIMESTAMP))), + ] + ), + # today + base.FuncToday( + variants=[ + V(D.ORACLE, lambda: sa.type_coerce(sa.func.TRUNC(sa.cast(raw_sql("SYSTIMESTAMP"), sa.Date)), sa.Date)), + ] + ), + # week + base.FuncWeek( + variants=[ + V(D.ORACLE, lambda date: sa.cast(sa.func.TO_CHAR(date, "IW"), sa.Integer)), + ] + ), + # year + base.FuncYear( + variants=[ + V(D.ORACLE, lambda date: sa.func.extract(raw_sql("YEAR"), date)), + ] + ), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_logical.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_logical.py new file mode 100644 index 000000000..ab560672c --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_logical.py @@ -0,0 +1,33 @@ +import sqlalchemy as sa + +from dl_formula.definitions.base import TranslationVariant +import dl_formula.definitions.functions_logical as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_LOGICAL = [ + # case + base.FuncCase.for_dialect(D.ORACLE), + # if + base.FuncIf.for_dialect(D.ORACLE), + # ifnull + base.FuncIfnull( + variants=[ + V(D.ORACLE, sa.func.NVL), + ] + ), + # iif + base.FuncIif3Legacy.for_dialect(D.ORACLE), + # isnull + base.FuncIsnull.for_dialect(D.ORACLE), + # zn + base.FuncZn( + variants=[ + V(D.ORACLE, lambda x: sa.func.NVL(x, 0)), + ] + ), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_markup.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_markup.py new file mode 100644 index 000000000..104560486 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_markup.py @@ -0,0 +1,20 @@ +import dl_formula.definitions.functions_markup as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +DEFINITIONS_MARKUP = [ + # + + base.BinaryPlusMarkup.for_dialect(D.ORACLE), + # __str + base.FuncInternalStrConst.for_dialect(D.ORACLE), + base.FuncInternalStr.for_dialect(D.ORACLE), + # bold + base.FuncBold.for_dialect(D.ORACLE), + # italic + base.FuncItalics.for_dialect(D.ORACLE), + # markup + base.ConcatMultiMarkup.for_dialect(D.ORACLE), + # url + base.FuncUrl.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_math.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_math.py new file mode 100644 index 000000000..8f33db508 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_math.py @@ -0,0 +1,114 @@ +import math + +import sqlalchemy as sa + +from dl_formula.definitions.base import TranslationVariant +import dl_formula.definitions.functions_math as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_MATH = [ + # abs + base.FuncAbs.for_dialect(D.ORACLE), + # acos + base.FuncAcos.for_dialect(D.ORACLE), + # asin + base.FuncAsin.for_dialect(D.ORACLE), + # atan + base.FuncAtan.for_dialect(D.ORACLE), + # atan2 + base.FuncAtan2.for_dialect(D.ORACLE), + # ceiling + base.FuncCeiling.for_dialect(D.ORACLE), + # cos + base.FuncCos.for_dialect(D.ORACLE), + # cot + base.FuncCot( + variants=[ + V(D.ORACLE, lambda x: sa.func.COS(x) / sa.func.SIN(x)), + ] + ), + # degrees + base.FuncDegrees( + variants=[ + V(D.ORACLE, lambda x: x / math.pi * 180), + ] + ), + # div + base.FuncDivBasic( + variants=[ + V(D.ORACLE, lambda x, y: sa.func.TRUNC(x / y)), + ] + ), + # div_safe + base.FuncDivSafe2( + variants=[ + V(D.ORACLE, lambda x, y: sa.func.TRUNC(x / sa.func.nullif(y, 0))), + ] + ), + base.FuncDivSafe3( + variants=[ + V(D.ORACLE, lambda x, y, default: sa.func.coalesce(sa.func.TRUNC(x / sa.func.nullif(y, 0)), default)), + ] + ), + # exp + base.FuncExp.for_dialect(D.ORACLE), + # fdiv_safe + base.FuncFDivSafe2.for_dialect(D.ORACLE), + base.FuncFDivSafe3.for_dialect(D.ORACLE), + # floor + base.FuncFloor.for_dialect(D.ORACLE), + # greatest + base.FuncGreatest1.for_dialect(D.ORACLE), + base.FuncGreatestMain.for_dialect(D.ORACLE), + base.GreatestMulti.for_dialect(D.ORACLE), + # least + base.FuncLeast1.for_dialect(D.ORACLE), + base.FuncLeastMain.for_dialect(D.ORACLE), + base.LeastMulti.for_dialect(D.ORACLE), + # ln + base.FuncLn.for_dialect(D.ORACLE), + # log + base.FuncLog.for_dialect(D.ORACLE), + # log10 + base.FuncLog10( + variants=[ + V(D.ORACLE, lambda x: sa.func.LOG(10, x)), + ] + ), + # pi + base.FuncPi( + variants=[ + V(D.ORACLE, lambda: sa.literal(math.pi)), + ] + ), + # power + base.FuncPower.for_dialect(D.ORACLE), + # radians + base.FuncRadians( + variants=[ + V(D.ORACLE, lambda x: x * math.pi / 180), + ] + ), + # round + base.FuncRound1.for_dialect(D.ORACLE), + base.FuncRound2.for_dialect(D.ORACLE), + # sign + base.FuncSign.for_dialect(D.ORACLE), + # sin + base.FuncSin.for_dialect(D.ORACLE), + # sqrt + base.FuncSqrt.for_dialect(D.ORACLE), + # square + base.FuncSquare( + variants=[ + V(D.ORACLE, lambda x: sa.func.POWER(x, 2)), + ] + ), + # tan + base.FuncTan.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_string.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_string.py new file mode 100644 index 000000000..cae1ad501 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_string.py @@ -0,0 +1,164 @@ +import sqlalchemy as sa + +from dl_formula.definitions.base import TranslationVariant +from dl_formula.definitions.common import make_binary_chain +import dl_formula.definitions.functions_string as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_STRING = [ + # ascii + base.FuncAscii.for_dialect(D.ORACLE), + # char + base.FuncChar( + variants=[ + V(D.ORACLE, sa.func.CHR), + ] + ), + # concat + base.Concat1.for_dialect((D.ORACLE)), + base.ConcatMultiStrConst.for_dialect(D.ORACLE), + base.ConcatMultiStr( + variants=[ + V( + D.ORACLE, + lambda *args: make_binary_chain( + # chain of 2-arg CONCAT function calls + (lambda x, y: sa.func.CONCAT(x, y)), + *args, + wrap_as_nodes=False, + ), + ), + ] + ), + base.ConcatMultiAny.for_dialect(D.ORACLE), + # contains + base.FuncContainsConst.for_dialect(D.ORACLE), + base.FuncContainsNonConst( + variants=[ + V(D.ORACLE, lambda x, y: sa.func.INSTR(x, y) > 0), + ] + ), + base.FuncContainsNonString.for_dialect(D.ORACLE), + # notcontains + base.FuncNotContainsConst.for_dialect(D.ORACLE), + base.FuncNotContainsNonConst.for_dialect(D.ORACLE), + base.FuncNotContainsNonString.for_dialect(D.ORACLE), + # endswith + base.FuncEndswithConst.for_dialect(D.ORACLE), + base.FuncEndswithNonConst( + variants=[ + V( + D.ORACLE, + lambda x, y: (sa.func.SUBSTR(x, sa.func.LENGTH(x) - sa.func.LENGTH(y) + 1, sa.func.LENGTH(y)) == y), + ), + ] + ), + base.FuncEndswithNonString.for_dialect(D.ORACLE), + # find + base.FuncFind2( + variants=[ + V(D.ORACLE, sa.func.INSTR), + ] + ), + base.FuncFind3( + variants=[ + V(D.ORACLE, sa.func.INSTR), + ] + ), + # icontains + base.FuncIContainsNonConst.for_dialect(D.ORACLE), + base.FuncIContainsNonString.for_dialect(D.ORACLE), + # iendswith + base.FuncIEndswithNonConst.for_dialect(D.ORACLE), + base.FuncIEndswithNonString.for_dialect(D.ORACLE), + # istartswith + base.FuncIStartswithNonConst.for_dialect(D.ORACLE), + base.FuncIStartswithNonString.for_dialect(D.ORACLE), + # left + base.FuncLeft( + variants=[ + V(D.ORACLE, lambda x, y: sa.func.SUBSTR(x, 1, y)), + ] + ), + # len + base.FuncLenString( + variants=[ + V(D.ORACLE, sa.func.LENGTH), + ] + ), + # lower + base.FuncLowerConst.for_dialect(D.ORACLE), + base.FuncLowerNonConst.for_dialect(D.ORACLE), + # ltrim + base.FuncLtrim.for_dialect(D.ORACLE), + # regexp_extract + base.FuncRegexpExtract( + variants=[ + V(D.ORACLE, sa.func.REGEXP_SUBSTR), + ] + ), + # regexp_extract_nth + base.FuncRegexpExtractNth( + variants=[ + V(D.ORACLE, lambda text, pattern, ind: sa.func.REGEXP_SUBSTR(text, pattern, 1, ind)), + ] + ), + # regexp_match + base.FuncRegexpMatch( + variants=[ + V(D.ORACLE, lambda text, pattern: sa.func.REGEXP_SUBSTR(text, pattern).isnot(None)), + ] + ), + # regexp_replace + base.FuncRegexpReplace( + variants=[ + V(D.ORACLE, sa.func.REGEXP_REPLACE), + ] + ), + # replace + base.FuncReplace.for_dialect(D.ORACLE), + # right + base.FuncRight( + variants=[ + V(D.ORACLE, lambda x, y: sa.func.SUBSTR(x, sa.func.LENGTH(x) - y + 1, y)), + ] + ), + # rtrim + base.FuncRtrim.for_dialect(D.ORACLE), + # space + base.FuncSpaceConst.for_dialect(D.ORACLE), + base.FuncSpaceNonConst( + variants=[ + V(D.ORACLE, lambda size: sa.func.SUBSTR(sa.func.LPAD(".", size + 1, " "), 1, size)), + ] + ), + # startswith + base.FuncStartswithConst.for_dialect(D.ORACLE), + base.FuncStartswithNonConst( + variants=[ + V(D.ORACLE, lambda x, y: (sa.func.SUBSTR(x, 1, sa.func.LENGTH(y)) == y)), + ] + ), + base.FuncStartswithNonString.for_dialect(D.ORACLE), + # substr + base.FuncSubstr2( + variants=[ + V(D.ORACLE, lambda text, start: sa.func.SUBSTR(text, start, sa.func.LENGTH(text) - start + 1)), + ] + ), + base.FuncSubstr3( + variants=[ + V(D.ORACLE, sa.func.SUBSTR), + ] + ), + # trim + base.FuncTrim.for_dialect(D.ORACLE), + # upper + base.FuncUpperConst.for_dialect(D.ORACLE), + base.FuncUpperNonConst.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_type.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_type.py new file mode 100644 index 000000000..1559c438a --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/functions_type.py @@ -0,0 +1,206 @@ +import sqlalchemy as sa +import sqlalchemy.dialects.oracle.base as sa_oracle + +from dl_formula.definitions.base import TranslationVariant +from dl_formula.definitions.common_datetime import ( + DAY_SEC, + EPOCH_START_D, +) +import dl_formula.definitions.functions_type as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_TYPE = [ + # bool + base.FuncBoolFromNull( + variants=[ + V(D.ORACLE, lambda _: sa.cast(sa.null(), sa_oracle.INTEGER())), + ] + ), + base.FuncBoolFromNumber.for_dialect(D.ORACLE), + base.FuncBoolFromBool.for_dialect(D.ORACLE), + base.FuncBoolFromStrGeo( + variants=[ + V(D.ORACLE, lambda value: value.isnot(None)), + ] + ), + base.FuncBoolFromDateDatetime.for_dialect(D.ORACLE), + # date + base.FuncDate1FromNull.for_dialect(D.ORACLE), + base.FuncDate1FromDatetime( + variants=[ + V(D.ORACLE, lambda expr: expr), + ] + ), + base.FuncDate1FromString( + variants=[ + V(D.ORACLE, lambda expr: sa.func.TO_DATE(expr, "YYYY-MM-DD")), + ] + ), + base.FuncDate1FromNumber( + variants=[ + V(D.ORACLE, lambda expr: EPOCH_START_D + expr / DAY_SEC), + ] + ), + # datetime + base.FuncDatetime1FromNull( + variants=[ + V(D.ORACLE, lambda _: sa.cast(sa.null(), sa.Date())), + ] + ), + base.FuncDatetime1FromDatetime.for_dialect(D.ORACLE), + base.FuncDatetime1FromDate( + variants=[ + V(D.ORACLE, lambda expr: expr), + ] + ), + base.FuncDatetime1FromNumber( + variants=[ + V(D.ORACLE, lambda expr: EPOCH_START_D + expr / DAY_SEC), + ] + ), + base.FuncDatetime1FromString( + variants=[ + V(D.ORACLE, lambda expr: sa.func.TO_DATE(expr, "YYYY-MM-DD HH:MI:SS")), + ] + ), + # datetimetz + base.FuncDatetimeTZConst.for_dialect(D.ORACLE), + # float + base.FuncFloatNumber( + variants=[ + V(D.ORACLE, lambda value: sa.cast(value, sa_oracle.BINARY_DOUBLE)), + ] + ), + base.FuncFloatString( + variants=[ + V(D.ORACLE, lambda value: sa.cast(value, sa_oracle.BINARY_DOUBLE)), + ] + ), + base.FuncFloatFromBool( + variants=[ + V(D.ORACLE, lambda value: sa.cast(value, sa_oracle.BINARY_DOUBLE)), + ] + ), + base.FuncFloatFromDate( + variants=[ + V(D.ORACLE, lambda value: sa.type_coerce((value - EPOCH_START_D) * DAY_SEC, sa_oracle.BINARY_DOUBLE)), + ] + ), + base.FuncFloatFromDatetime( + variants=[ + V(D.ORACLE, lambda value: sa.type_coerce((value - EPOCH_START_D) * DAY_SEC, sa_oracle.BINARY_DOUBLE)), + ] + ), + base.FuncFloatFromGenericDatetime( + variants=[ + V(D.ORACLE, lambda value: sa.type_coerce((value - EPOCH_START_D) * DAY_SEC, sa_oracle.BINARY_DOUBLE)), + ] + ), + # genericdatetime + base.FuncGenericDatetime1FromNull( + variants=[ + V(D.ORACLE, lambda _: sa.cast(sa.null(), sa.Date())), + ] + ), + base.FuncGenericDatetime1FromDatetime.for_dialect(D.ORACLE), + base.FuncGenericDatetime1FromDate( + variants=[ + V(D.ORACLE, lambda expr: expr), + ] + ), + base.FuncGenericDatetime1FromNumber( + variants=[ + V(D.ORACLE, lambda expr: EPOCH_START_D + expr / DAY_SEC), + ] + ), + base.FuncGenericDatetime1FromString( + variants=[ + V(D.ORACLE, lambda expr: sa.func.TO_DATE(expr, "YYYY-MM-DD HH:MI:SS")), + ] + ), + # geopoint + base.FuncGeopointFromStr.for_dialect(D.ORACLE), + base.FuncGeopointFromCoords.for_dialect(D.ORACLE), + # geopolygon + base.FuncGeopolygon.for_dialect(D.ORACLE), + # int + base.FuncIntFromNull( + variants=[ + V(D.ORACLE, lambda _: sa.cast(sa.null(), sa_oracle.INTEGER())), + ] + ), + base.FuncIntFromInt.for_dialect(D.ORACLE), + base.FuncIntFromFloat( + variants=[ + V(D.ORACLE, lambda value: sa.cast(sa.func.FLOOR(value), sa.Integer)), + ] + ), + base.FuncIntFromBool( + variants=[ + V(D.ORACLE, lambda value: value), + ] + ), + base.FuncIntFromStr( + variants=[ + V(D.ORACLE, lambda value: sa.cast(value, sa.Integer)), + ] + ), + base.FuncIntFromDate( + variants=[ + V(D.ORACLE, lambda value: sa.cast((value - EPOCH_START_D) * DAY_SEC, sa.Integer)), + ] + ), + base.FuncIntFromDatetime( + variants=[ + V(D.ORACLE, lambda value: sa.cast((value - EPOCH_START_D) * DAY_SEC, sa.Integer)), + ] + ), + base.FuncIntFromGenericDatetime( + variants=[ + V(D.ORACLE, lambda value: sa.cast((value - EPOCH_START_D) * DAY_SEC, sa.Integer)), + ] + ), + # str + base.FuncStrFromUnsupported( + variants=[ + V(D.ORACLE, sa.func.TO_CHAR), + ] + ), + base.FuncStrFromInteger( + variants=[ + V(D.ORACLE, sa.func.TO_CHAR), + ] + ), + base.FuncStrFromFloat( + variants=[ + V(D.ORACLE, sa.func.TO_CHAR), + ] + ), + base.FuncStrFromBool( + variants=[ + V( + D.ORACLE, + lambda value: sa.case( + whens=[(value.is_(None), sa.null()), (value != sa.literal(0), "True")], else_="False" + ), + ), + ] + ), + base.FuncStrFromStrGeo.for_dialect(D.ORACLE), + base.FuncStrFromDate( + variants=[ + V(D.ORACLE, lambda value: sa.func.TO_CHAR(value, "YYYY-MM-DD")), + ] + ), + base.FuncStrFromDatetime( + variants=[ + V(D.ORACLE, lambda value: sa.func.TO_CHAR(value, "YYYY-MM-DD HH:MI:SS")), + ] + ), + base.FuncStrFromString.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_binary.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_binary.py new file mode 100644 index 000000000..3346a7b61 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_binary.py @@ -0,0 +1,120 @@ +import sqlalchemy as sa + +from dl_formula.definitions.base import TranslationVariant +import dl_formula.definitions.operators_binary as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_BINARY = [ + # != + base.BinaryNotEqual.for_dialect(D.ORACLE), + # % + base.BinaryModInteger.for_dialect(D.ORACLE), + base.BinaryModFloat.for_dialect(D.ORACLE), + # * + base.BinaryMultNumbers.for_dialect(D.ORACLE), + base.BinaryMultStringConst.for_dialect(D.ORACLE), + base.BinaryMultStringNonConst( + variants=[ + V( + D.ORACLE, + lambda text, size: sa.func.SUBSTR( + sa.func.LPAD(".", size * sa.func.LENGTH(text) + 1, text), 1, size * sa.func.LENGTH(text) + ), + ), + ] + ), + # + + base.BinaryPlusNumbers.for_dialect(D.ORACLE), + base.BinaryPlusStrings.for_dialect(D.ORACLE), + base.BinaryPlusDateInt( + variants=[ + V(D.ORACLE, lambda date, days: date + days), + ] + ), + base.BinaryPlusDateFloat( + variants=[ + V(D.ORACLE, lambda date, days: date + days), + ] + ), + base.BinaryPlusDatetimeNumber( + variants=[ + V(D.ORACLE, lambda dt, days: dt + days), + ] + ), + base.BinaryPlusGenericDatetimeNumber( + variants=[ + V(D.ORACLE, lambda dt, days: dt + days), + ] + ), + # - + base.BinaryMinusNumbers.for_dialect(D.ORACLE), + base.BinaryMinusDateInt( + variants=[ + V(D.ORACLE, lambda date, days: date - days), + ] + ), + base.BinaryMinusDateFloat( + variants=[ + V(D.ORACLE, lambda date, days: date - days), + ] + ), + base.BinaryMinusDatetimeNumber( + variants=[ + V(D.ORACLE, lambda dt, days: dt - days), + ] + ), + base.BinaryMinusGenericDatetimeNumber( + variants=[ + V(D.ORACLE, lambda dt, days: dt - days), + ] + ), + base.BinaryMinusDates.for_dialect(D.ORACLE), + base.BinaryMinusDatetimes( + variants=[ + V(D.ORACLE, lambda left, right: left - right), + ] + ), + base.BinaryMinusGenericDatetimes( + variants=[ + V(D.ORACLE, lambda left, right: left - right), + ] + ), + # / + base.BinaryDivInt.for_dialect(D.ORACLE), + base.BinaryDivFloat.for_dialect(D.ORACLE), + # < + base.BinaryLessThan.for_dialect(D.ORACLE), + # <= + base.BinaryLessThanOrEqual.for_dialect(D.ORACLE), + # == + base.BinaryEqual.for_dialect(D.ORACLE), + # > + base.BinaryGreaterThan.for_dialect(D.ORACLE), + # >= + base.BinaryGreaterThanOrEqual.for_dialect(D.ORACLE), + # ^ + base.BinaryPower.for_dialect(D.ORACLE), + # _!= + base.BinaryNotEqualInternal.for_dialect(D.ORACLE), + # _== + base.BinaryEqualInternal.for_dialect(D.ORACLE), + # _dneq + base.BinaryEqualDenullified.for_dialect(D.ORACLE), + # and + base.BinaryAnd.for_dialect(D.ORACLE), + # in + base.BinaryIn.for_dialect(D.ORACLE), + # like + base.BinaryLike.for_dialect(D.ORACLE), + # notin + base.BinaryNotIn.for_dialect(D.ORACLE), + # notlike + base.BinaryNotLike.for_dialect(D.ORACLE), + # or + base.BinaryOr.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_ternary.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_ternary.py new file mode 100644 index 000000000..ebc99e481 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_ternary.py @@ -0,0 +1,11 @@ +import dl_formula.definitions.operators_ternary as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +DEFINITIONS_TERNARY = [ + # between + base.TernaryBetween.for_dialect(D.ORACLE), + # notbetween + base.TernaryNotBetween.for_dialect(D.ORACLE), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_unary.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_unary.py new file mode 100644 index 000000000..e473070f7 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/definitions/operators_unary.py @@ -0,0 +1,62 @@ +import sqlalchemy as sa + +from dl_formula.definitions.base import TranslationVariant +import dl_formula.definitions.operators_unary as base + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +V = TranslationVariant.make + + +DEFINITIONS_UNARY = [ + # isfalse + base.UnaryIsFalseStringGeo( + variants=[ + V(D.ORACLE, lambda x: x.is_(None)), + ] + ), + base.UnaryIsFalseNumbers.for_dialect(D.ORACLE), + base.UnaryIsFalseDateTime( + variants=[ + V(D.ORACLE, lambda x: sa.literal(0)), + ] + ), + base.UnaryIsFalseBoolean( + variants=[ + V(D.ORACLE, lambda x: x == 0), + ] + ), + # istrue + base.UnaryIsTrueStringGeo( + variants=[ + V(D.ORACLE, lambda x: x.isnot(None)), + ] + ), + base.UnaryIsTrueNumbers.for_dialect(D.ORACLE), + base.UnaryIsTrueDateTime( + variants=[ + V(D.ORACLE, lambda x: sa.literal(1)), + ] + ), + base.UnaryIsTrueBoolean( + variants=[ + V(D.ORACLE, lambda x: x != 0), + ] + ), + # neg + base.UnaryNegate.for_dialect(D.ORACLE), + # not + base.UnaryNotBool.for_dialect(D.ORACLE), + base.UnaryNotNumbers.for_dialect(D.ORACLE), + base.UnaryNotStringGeo( + variants=[ + V(D.ORACLE, lambda x: x.is_(None)), + ] + ), + base.UnaryNotDateDatetime( + variants=[ + V(D.ORACLE, lambda x: sa.literal(0)), + ] + ), +] diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/literal.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/literal.py new file mode 100644 index 000000000..7ebe21b63 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/literal.py @@ -0,0 +1,25 @@ +import sqlalchemy as sa + +from dl_formula.connectors.base.literal import ( + Literal, + Literalizer, +) +from dl_formula.core.dialect import DialectCombo + + +class OracleLiteralizer(Literalizer): + __slots__ = () + + def literal_bool(self, value: bool, dialect: DialectCombo) -> Literal: + return sa.literal(int(value)) + + def literal_str(self, value: str, dialect: DialectCombo) -> Literal: + # SA uses NVARCHAR2 for string literals by default, + # but CHAR has better compatibility with some of the functions, so try to use it + # with a fallback to NCHAR for non-ASCII strings + try: + value.encode("ascii") + type_ = sa.CHAR(len(value)) + except UnicodeEncodeError: + type_ = sa.NCHAR(len(value)) + return sa.literal(value, type_) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula/type_constructor.py b/lib/dl_connector_oracle/dl_connector_oracle/formula/type_constructor.py new file mode 100644 index 000000000..4d6f5034e --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula/type_constructor.py @@ -0,0 +1,19 @@ +import sqlalchemy as sa +import sqlalchemy.dialects.oracle.base as sa_oracle +from sqlalchemy.types import TypeEngine + +from dl_formula.connectors.base.type_constructor import DefaultSATypeConstructor +from dl_formula.core.datatype import DataType + + +class OracleTypeConstructor(DefaultSATypeConstructor): + def get_sa_type(self, data_type: DataType) -> TypeEngine: + type_map: dict[DataType, TypeEngine] = { + DataType.BOOLEAN: sa_oracle.NUMBER(1, 0), + DataType.DATETIME: sa.Date(), + DataType.GENERICDATETIME: sa.Date(), + } + if (type_eng := type_map.get(data_type)) is not None: + return type_eng + else: + return super().get_sa_type(data_type) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/human_dialects.py b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/human_dialects.py new file mode 100644 index 000000000..f7f33e338 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/human_dialects.py @@ -0,0 +1,13 @@ +from dl_formula_ref.texts import StyledDialect + +from dl_connector_oracle.formula.constants import OracleDialect +from dl_connector_oracle.formula_ref.i18n import Translatable + + +HUMAN_DIALECTS = { + OracleDialect.ORACLE_12_0: StyledDialect( + "`Oracle Database 12c (12.1)`", + "`Oracle`
`Database 12c`
`(12.1)`", + Translatable("`Oracle Database` version `12c (12.1)`"), + ), +} diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/i18n.py b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/i18n.py new file mode 100644 index 000000000..d6d112eec --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/i18n.py @@ -0,0 +1,23 @@ +import os + +import attr + +from dl_i18n.localizer_base import Translatable as BaseTranslatable +from dl_i18n.localizer_base import TranslationConfig + +import dl_connector_oracle as package + + +DOMAIN = f"dl_formula_ref_{package.__name__}" + +_LOCALE_DIR = os.path.join(os.path.dirname(__file__), "..", "locales") + +CONFIGS = [ + TranslationConfig(path=_LOCALE_DIR, domain=DOMAIN, locale="en"), + TranslationConfig(path=_LOCALE_DIR, domain=DOMAIN, locale="ru"), +] + + +@attr.s +class Translatable(BaseTranslatable): + domain: str = attr.ib(default=DOMAIN) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/plugin.py b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/plugin.py new file mode 100644 index 000000000..ea7bf37b7 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/formula_ref/plugin.py @@ -0,0 +1,12 @@ +from dl_formula_ref.plugins.base.plugin import FormulaRefPlugin + +from dl_connector_oracle.formula.constants import OracleDialect +from dl_connector_oracle.formula_ref.human_dialects import HUMAN_DIALECTS +from dl_connector_oracle.formula_ref.i18n import CONFIGS + + +class OracleFormulaRefPlugin(FormulaRefPlugin): + any_dialects = frozenset((*OracleDialect.ORACLE.to_list(),)) + compeng_support_dialects = frozenset((*OracleDialect.ORACLE.to_list(),)) + human_dialects = HUMAN_DIALECTS + translation_configs = frozenset(CONFIGS) diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_connector_oracle.mo b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_connector_oracle.mo new file mode 100644 index 000000000..b3a479353 Binary files /dev/null and b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_connector_oracle.mo differ diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_connector_oracle.po b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_connector_oracle.po new file mode 100644 index 000000000..4112a48a9 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_connector_oracle.po @@ -0,0 +1,19 @@ +# Copyright (c) 2023 YANDEX LLC +# This file is distributed under the same license as the DataLens package. +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: datalens-opensource@yandex-team.ru\n" +"POT-Creation-Date: 2023-09-22 08:10+0000\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" + +msgid "label_connector-oracle" +msgstr "Oracle Database" + +msgid "value_db-connect-method-service-name" +msgstr "Service name" + +msgid "value_db-connect-method-sid" +msgstr "SID" diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.mo b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.mo new file mode 100644 index 000000000..00261bf60 Binary files /dev/null and b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.mo differ diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.po b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.po new file mode 100644 index 000000000..4bff4811d --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/locales/en/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.po @@ -0,0 +1,13 @@ +# Copyright (c) 2023 YANDEX LLC +# This file is distributed under the same license as the DataLens package. +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: datalens-opensource@yandex-team.ru\n" +"POT-Creation-Date: 2023-09-22 08:11+0000\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" + +msgid "`Oracle Database` version `12c (12.1)`" +msgstr "" diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_connector_oracle.mo b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_connector_oracle.mo new file mode 100644 index 000000000..4bd0d9abd Binary files /dev/null and b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_connector_oracle.mo differ diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_connector_oracle.po b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_connector_oracle.po new file mode 100644 index 000000000..91afadc66 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_connector_oracle.po @@ -0,0 +1,19 @@ +# Copyright (c) 2023 YANDEX LLC +# This file is distributed under the same license as the DataLens package. +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: datalens-opensource@yandex-team.ru\n" +"POT-Creation-Date: 2023-09-22 08:10+0000\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" + +msgid "label_connector-oracle" +msgstr "Oracle Database" + +msgid "value_db-connect-method-service-name" +msgstr "Имя сервиса" + +msgid "value_db-connect-method-sid" +msgstr "SID" diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.mo b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.mo new file mode 100644 index 000000000..c21e919e5 Binary files /dev/null and b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.mo differ diff --git a/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.po b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.po new file mode 100644 index 000000000..8ce63e442 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle/locales/ru/LC_MESSAGES/dl_formula_ref_dl_connector_oracle.po @@ -0,0 +1,13 @@ +# Copyright (c) 2023 YANDEX LLC +# This file is distributed under the same license as the DataLens package. +msgid "" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"Report-Msgid-Bugs-To: datalens-opensource@yandex-team.ru\n" +"POT-Creation-Date: 2023-09-22 08:11+0000\n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" + +msgid "`Oracle Database` version `12c (12.1)`" +msgstr "`Oracle Database` версии `12c (12.1)`" diff --git a/lib/dl_connector_oracle/dl_connector_oracle/py.typed b/lib/dl_connector_oracle/dl_connector_oracle/py.typed new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/base.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/base.py new file mode 100644 index 000000000..debfe442e --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/base.py @@ -0,0 +1,60 @@ +import pytest + +from dl_api_lib_testing.configuration import ApiTestEnvironmentConfiguration +from dl_api_lib_testing.connection_base import ConnectionTestBase +from dl_api_lib_testing.data_api_base import StandardizedDataApiTestBase +from dl_api_lib_testing.dataset_base import DatasetTestBase +from dl_constants.enums import RawSQLLevel + +from dl_connector_oracle.core.constants import ( + CONNECTION_TYPE_ORACLE, + SOURCE_TYPE_ORACLE_TABLE, + OracleDbNameType, +) +from dl_connector_oracle_tests.db.config import ( + API_TEST_CONFIG, + CoreConnectionSettings, +) +from dl_connector_oracle_tests.db.core.base import BaseOracleTestClass + + +class OracleConnectionTestBase(BaseOracleTestClass, ConnectionTestBase): + conn_type = CONNECTION_TYPE_ORACLE + bi_compeng_pg_on = False + + @pytest.fixture(scope="class") + def bi_test_config(self) -> ApiTestEnvironmentConfiguration: + return API_TEST_CONFIG + + @pytest.fixture(scope="class") + def connection_params(self) -> dict: + return dict( + db_connect_method=OracleDbNameType.service_name.name, + db_name=CoreConnectionSettings.DB_NAME, + host=CoreConnectionSettings.HOST, + port=CoreConnectionSettings.PORT, + username=CoreConnectionSettings.USERNAME, + password=CoreConnectionSettings.PASSWORD, + **(dict(raw_sql_level=self.raw_sql_level.value) if self.raw_sql_level is not None else {}), + ) + + +class OracleDashSQLConnectionTest(OracleConnectionTestBase): + raw_sql_level = RawSQLLevel.dashsql + + +class OracleDatasetTestBase(OracleConnectionTestBase, DatasetTestBase): + @pytest.fixture(scope="class") + def dataset_params(self, sample_table) -> dict: + return dict( + source_type=SOURCE_TYPE_ORACLE_TABLE.name, + parameters=dict( + db_name=sample_table.db.name, + schema_name=sample_table.schema, + table_name=sample_table.name, + ), + ) + + +class OracleDataApiTestBase(OracleDatasetTestBase, StandardizedDataApiTestBase): + mutation_caches_on = False diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_complex_queries.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_complex_queries.py new file mode 100644 index 000000000..e075123c7 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_complex_queries.py @@ -0,0 +1,7 @@ +from dl_api_lib_testing.connector.complex_queries import DefaultBasicComplexQueryTestSuite + +from dl_connector_oracle_tests.db.api.base import OracleDataApiTestBase + + +class TestOracleBasicComplexQueries(OracleDataApiTestBase, DefaultBasicComplexQueryTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_connection.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_connection.py new file mode 100644 index 000000000..10f10c240 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_connection.py @@ -0,0 +1,7 @@ +from dl_api_lib_testing.connector.connection_suite import DefaultConnectorConnectionTestSuite + +from dl_connector_oracle_tests.db.api.base import OracleConnectionTestBase + + +class TestOracleConnection(OracleConnectionTestBase, DefaultConnectorConnectionTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_dashsql.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_dashsql.py new file mode 100644 index 000000000..a1e8ee08c --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_dashsql.py @@ -0,0 +1,88 @@ +from aiohttp.test_utils import TestClient +import pytest + +from dl_api_lib_testing.connector.dashsql_suite import DefaultDashSQLTestSuite + +from dl_connector_oracle_tests.db.api.base import OracleDashSQLConnectionTest +from dl_connector_oracle_tests.db.config import SUBSELECT_QUERY_FULL + + +class TestOracleDashSQL(OracleDashSQLConnectionTest, DefaultDashSQLTestSuite): + @pytest.fixture(scope="class") + def dashsql_basic_query(self) -> str: + return "select 1, 2, 3 from dual" + + @pytest.mark.asyncio + async def test_result(self, data_api_lowlevel_aiohttp_client: TestClient, saved_connection_id: str): + resp = await self.get_dashsql_response( + data_api_aio=data_api_lowlevel_aiohttp_client, + conn_id=saved_connection_id, + query=SUBSELECT_QUERY_FULL, + ) + + resp_data = await resp.json() + assert resp_data[0]["event"] == "metadata", resp_data + assert resp_data[0]["data"]["names"] == [ + "NUM", + "NUM_STR", + "NUM_INTEGER", + "NUM_NUMBER", + "NUM_BINARY_FLOAT", + "NUM_BINARY_DOUBLE", + "NUM_CHAR", + "NUM_VARCHAR", + "NUM_VARCHAR2", + "NUM_NCHAR", + "NUM_NVARCHAR2", + "NUM_DATE", + "NUM_TIMESTAMP", + "NUM_TIMESTAMP_TZ", + ] + assert resp_data[0]["data"]["driver_types"] == [ + "db_type_number", + "db_type_varchar", + "db_type_number", + "db_type_number", + "db_type_binary_float", + "db_type_binary_double", + "db_type_char", + "db_type_varchar", + "db_type_varchar", + "db_type_nchar", + "db_type_nvarchar", + "db_type_date", + "db_type_timestamp", + "db_type_timestamp_tz", + ] + assert resp_data[0]["data"]["db_types"] == [ + "integer", + "varchar", + "integer", + "integer", + "binary_float", + "binary_double", + "char", + "varchar", + "varchar", + "nchar", + "nvarchar", + "date", + "timestamp", + "timestamp", + ] + assert resp_data[0]["data"]["bi_types"] == [ + "integer", + "string", + "integer", + "integer", + "float", + "float", + "string", + "string", + "string", + "string", + "string", + "genericdatetime", + "genericdatetime", + "genericdatetime", + ] diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_data.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_data.py new file mode 100644 index 000000000..bcf0c6d1b --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_data.py @@ -0,0 +1,39 @@ +from dl_api_lib_testing.connector.data_api_suites import ( + DefaultConnectorDataDistinctTestSuite, + DefaultConnectorDataGroupByFormulaTestSuite, + DefaultConnectorDataPreviewTestSuite, + DefaultConnectorDataRangeTestSuite, + DefaultConnectorDataResultTestSuite, +) +from dl_testing.regulated_test import RegulatedTestParams + +from dl_connector_oracle_tests.db.api.base import OracleDataApiTestBase + + +class TestOracleDataResult(OracleDataApiTestBase, DefaultConnectorDataResultTestSuite): + test_params = RegulatedTestParams( + mark_features_skipped={ + DefaultConnectorDataResultTestSuite.array_support: "Oracle doesn't support arrays", + }, + mark_tests_failed={ + DefaultConnectorDataResultTestSuite.test_get_result_with_string_filter_operations_for_numbers: ( + "BI-4978: Need to ignore the exponent" + ) + }, + ) + + +class TestOracleDataGroupBy(OracleDataApiTestBase, DefaultConnectorDataGroupByFormulaTestSuite): + pass + + +class TestOracleDataRange(OracleDataApiTestBase, DefaultConnectorDataRangeTestSuite): + pass + + +class TestOracleDataDistinct(OracleDataApiTestBase, DefaultConnectorDataDistinctTestSuite): + pass + + +class TestOracleDataPreview(OracleDataApiTestBase, DefaultConnectorDataPreviewTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_dataset.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_dataset.py new file mode 100644 index 000000000..f3352b892 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/api/test_dataset.py @@ -0,0 +1,7 @@ +from dl_api_lib_testing.connector.dataset_suite import DefaultConnectorDatasetTestSuite + +from dl_connector_oracle_tests.db.api.base import OracleDatasetTestBase + + +class TestOracleDataset(OracleDatasetTestBase, DefaultConnectorDatasetTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/config.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/config.py new file mode 100644 index 000000000..1b44c0f95 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/config.py @@ -0,0 +1,79 @@ +import os +from typing import ClassVar + +from dl_api_lib_testing.configuration import ApiTestEnvironmentConfiguration +from dl_core_testing.configuration import DefaultCoreTestConfiguration +from dl_testing.containers import get_test_container_hostport + +from dl_connector_oracle.formula.constants import OracleDialect as D + + +# Infra settings +CORE_TEST_CONFIG = DefaultCoreTestConfiguration( + host_us_http=get_test_container_hostport("us", fallback_port=51811).host, + port_us_http=get_test_container_hostport("us", fallback_port=51811).port, + host_us_pg=get_test_container_hostport("pg-us", fallback_port=51810).host, + port_us_pg_5432=get_test_container_hostport("pg-us", fallback_port=51810).port, + us_master_token="AC1ofiek8coB", +) + +COMPOSE_PROJECT_NAME = os.environ.get("COMPOSE_PROJECT_NAME", "dl_connector_oracle") +ORACLE_CONTAINER_LABEL = "db-oracle" + + +class CoreConnectionSettings: + DB_NAME: ClassVar[str] = "XEPDB1" + HOST: ClassVar[str] = get_test_container_hostport("db-oracle", fallback_port=51800).host + PORT: ClassVar[int] = get_test_container_hostport("db-oracle", fallback_port=51800).port + USERNAME: ClassVar[str] = "datalens" + PASSWORD: ClassVar[str] = "qwerty" + + +DEFAULT_ORACLE_SCHEMA_NAME = "DATALENS" + + +SUBSELECT_QUERY_FULL = r""" +select + num, + 'test' || num as num_str, + cast(num as integer) as num_integer, + cast(num as number) as num_number, + -- cast(num as number(9,9)) as num_number_9_9, + cast(num as binary_float) as num_binary_float, + cast(num as binary_double) as num_binary_double, + cast(num as char) as num_char, + cast(num as varchar(3)) as num_varchar, + cast(num as varchar2(4)) as num_varchar2, + cast(num as nchar) as num_nchar, + -- cast(num as nvarchar(5)) as num_nvarchar, + cast(num as nvarchar2(5)) as num_nvarchar2, + DATE '2020-01-01' + num as num_date, + TIMESTAMP '1999-12-31 23:59:59.10' + numToDSInterval(num, 'second') as num_timestamp, + TIMESTAMP '1999-12-31 23:59:59.10-07:00' + numToDSInterval(num, 'second') as num_timestamp_tz +from ( + select 0 as num from dual + union all + select 1 as num from dual + union all + select 6 as num from dual +) sq +""" + +_DB_URL = ( + f'oracle://datalens:qwerty@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={get_test_container_hostport("db-oracle", fallback_port=51800).host})' + f'(PORT={get_test_container_hostport("db-oracle", fallback_port=51800).port}))(CONNECT_DATA=(SERVICE_NAME={CoreConnectionSettings.DB_NAME})))' +) +SYSDBA_URL = ( + f'oracle://sys:qwerty@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST={get_test_container_hostport("db-oracle", fallback_port=51800).host})' + f'(PORT={get_test_container_hostport("db-oracle", fallback_port=51800).port}))(CONNECT_DATA=(SERVICE_NAME={CoreConnectionSettings.DB_NAME})))?mode=sysdba' +) +DB_CORE_URL = _DB_URL +DB_URLS = { + D.ORACLE_12_0: _DB_URL, +} + +API_TEST_CONFIG = ApiTestEnvironmentConfiguration( + api_connector_ep_names=["oracle"], + core_test_config=CORE_TEST_CONFIG, + ext_query_executer_secret_key="_some_test_secret_key_", +) diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/conftest.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/conftest.py new file mode 100644 index 000000000..6c483d225 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/conftest.py @@ -0,0 +1,35 @@ +from dl_api_lib_testing.initialization import initialize_api_lib_test +from dl_core_testing.database import ( + CoreDbConfig, + CoreDbDispenser, +) +from dl_db_testing.database.engine_wrapper import DbEngineConfig +from dl_formula_testing.forced_literal import forced_literal_use + +from dl_connector_oracle.core.constants import CONNECTION_TYPE_ORACLE +from dl_connector_oracle_tests.db.config import ( + API_TEST_CONFIG, + SYSDBA_URL, +) + + +pytest_plugins = ("aiohttp.pytest_plugin",) # and it, in turn, includes 'pytest_asyncio.plugin' + + +def pytest_configure(config): # noqa + initialize_api_lib_test(pytest_config=config, api_test_config=API_TEST_CONFIG) + initialize_db() + + +def initialize_db(): + # Granting of priveleges con only be done when connecting as sydba + db_dispenser = CoreDbDispenser() + sysdba_db_config = CoreDbConfig(engine_config=DbEngineConfig(url=SYSDBA_URL), conn_type=CONNECTION_TYPE_ORACLE) + sysdba_db = db_dispenser.get_database(db_config=sysdba_db_config) + sysdba_db.execute("GRANT SELECT ON sys.V_$SESSION TO datalens") + + +__all__ = ( + # auto-use fixtures: + "forced_literal_use", +) diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/base.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/base.py new file mode 100644 index 000000000..0263b368e --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/base.py @@ -0,0 +1,59 @@ +import asyncio +from typing import Generator +import uuid + +import pytest + +from dl_core.us_manager.us_manager_sync import SyncUSManager +from dl_core_testing.database import Db +from dl_core_testing.testcases.connection import BaseConnectionTestClass + +from dl_connector_oracle.core.constants import CONNECTION_TYPE_ORACLE +from dl_connector_oracle.core.testing.connection import make_oracle_saved_connection +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle +import dl_connector_oracle_tests.db.config as test_config + + +class BaseOracleTestClass(BaseConnectionTestClass[ConnectionSQLOracle]): + conn_type = CONNECTION_TYPE_ORACLE + core_test_config = test_config.CORE_TEST_CONFIG + + @pytest.fixture(autouse=True) + # FIXME: This fixture is a temporary solution for failing core tests when they are run together with api tests + def loop(self, event_loop: asyncio.AbstractEventLoop) -> Generator[asyncio.AbstractEventLoop, None, None]: + asyncio.set_event_loop(event_loop) + yield event_loop + # Attempt to cover an old version of pytest-asyncio: + # https://github.com/pytest-dev/pytest-asyncio/commit/51d986cec83fdbc14fa08015424c79397afc7ad9 + asyncio.set_event_loop_policy(None) + + @pytest.fixture(scope="class") + def db_url(self) -> str: + return test_config.DB_CORE_URL + + @pytest.fixture(scope="function") + def connection_creation_params(self) -> dict: + return dict( + db_name=test_config.CoreConnectionSettings.DB_NAME, + host=test_config.CoreConnectionSettings.HOST, + port=test_config.CoreConnectionSettings.PORT, + username=test_config.CoreConnectionSettings.USERNAME, + password=test_config.CoreConnectionSettings.PASSWORD, + **(dict(raw_sql_level=self.raw_sql_level) if self.raw_sql_level is not None else {}), + ) + + @pytest.fixture(scope="class") + def _empty_table(self, db: Db) -> None: + # So that template listings are not empty + table_name = f"t_{uuid.uuid4().hex[:6]}" + db.execute(f'CREATE TABLE datalens.{table_name} ("value" VARCHAR2(255))') + + @pytest.fixture(scope="function") + def saved_connection( + self, + sync_us_manager: SyncUSManager, + connection_creation_params: dict, + _empty_table, + ) -> ConnectionSQLOracle: + conn = make_oracle_saved_connection(sync_usm=sync_us_manager, **connection_creation_params) + return conn diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_connection.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_connection.py new file mode 100644 index 000000000..fd8ce4e54 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_connection.py @@ -0,0 +1,35 @@ +from dl_core.us_connection_base import DataSourceTemplate +from dl_core_testing.testcases.connection import DefaultConnectionTestClass + +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle +from dl_connector_oracle_tests.db.config import DEFAULT_ORACLE_SCHEMA_NAME +from dl_connector_oracle_tests.db.core.base import BaseOracleTestClass + + +class TestOracleConnection( + BaseOracleTestClass, + DefaultConnectionTestClass[ConnectionSQLOracle], +): + do_check_data_export_flag = True + + def check_saved_connection(self, conn: ConnectionSQLOracle, params: dict) -> None: + assert conn.uuid is not None + assert conn.data.db_name == params["db_name"] + assert conn.data.host == params["host"] + assert conn.data.port == params["port"] + assert conn.data.username == params["username"] + assert conn.data.password == params["password"] + + def check_data_source_templates( + self, + conn: ConnectionSQLOracle, + dsrc_templates: list[DataSourceTemplate], + ) -> None: + assert dsrc_templates + for dsrc_tmpl in dsrc_templates: + assert dsrc_tmpl.title + if dsrc_tmpl.parameters.get("schema_name") is not None: + assert dsrc_tmpl.group == [dsrc_tmpl.parameters["schema_name"]] + + schema_names = {tmpl.parameters.get("schema_name") for tmpl in dsrc_templates} + assert DEFAULT_ORACLE_SCHEMA_NAME in schema_names diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_connection_executor.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_connection_executor.py new file mode 100644 index 000000000..bd8454bc0 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_connection_executor.py @@ -0,0 +1,76 @@ +from typing import ( + Optional, + Sequence, +) + +import pytest +import sqlalchemy as sa +from sqlalchemy.types import TypeEngine + +from dl_constants.enums import UserDataType +from dl_core.connection_models.common_models import DBIdent +from dl_core_testing.testcases.connection_executor import ( + DefaultAsyncConnectionExecutorTestSuite, + DefaultSyncAsyncConnectionExecutorCheckBase, + DefaultSyncConnectionExecutorTestSuite, +) +from dl_testing.regulated_test import RegulatedTestParams + +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle +from dl_connector_oracle_tests.db.config import CoreConnectionSettings +from dl_connector_oracle_tests.db.core.base import BaseOracleTestClass + + +class OracleSyncAsyncConnectionExecutorCheckBase( + BaseOracleTestClass, + DefaultSyncAsyncConnectionExecutorCheckBase[ConnectionSQLOracle], +): + test_params = RegulatedTestParams( + mark_tests_failed={ + DefaultAsyncConnectionExecutorTestSuite.test_error_on_select_from_nonexistent_source: "", # TODO: FIXME + DefaultAsyncConnectionExecutorTestSuite.test_get_table_schema_info_for_nonexistent_table: ( + "Empty schema is returned instead of an error" + ), # FIXME + }, + ) + + @pytest.fixture(scope="function") + def db_ident(self) -> DBIdent: + return DBIdent(db_name=CoreConnectionSettings.DB_NAME) + + def check_db_version(self, db_version: Optional[str]) -> None: + assert db_version is not None + assert "." in db_version + + def get_schemas_for_type_recognition(self) -> dict[str, Sequence[tuple[TypeEngine, UserDataType]]]: + return { + "standard_types_the_oracle_way": [ + (sa.Numeric(16, 0), UserDataType.integer), + (sa.Numeric(16, 8), UserDataType.float), + (sa.String(length=256), UserDataType.string), + (sa.Date(), UserDataType.genericdatetime), + (sa.DateTime(), UserDataType.genericdatetime), + ], + } + + @pytest.fixture(scope="class") + def query_for_session_check(self) -> str: + return "SELECT 567 FROM DUAL" + + +class TestOracleSyncConnectionExecutor( + OracleSyncAsyncConnectionExecutorCheckBase, + DefaultSyncConnectionExecutorTestSuite[ConnectionSQLOracle], +): + pass + + +class TestOracleAsyncConnectionExecutor( + OracleSyncAsyncConnectionExecutorCheckBase, + DefaultAsyncConnectionExecutorTestSuite[ConnectionSQLOracle], +): + test_params = RegulatedTestParams( + mark_tests_failed={ + DefaultAsyncConnectionExecutorTestSuite.test_closing_sql_sessions: "Sessions not closed", # TODO: FIXME + }, + ) diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_data_source.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_data_source.py new file mode 100644 index 000000000..fe39a3575 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_data_source.py @@ -0,0 +1,126 @@ +import pytest + +from dl_constants.enums import ( + RawSQLLevel, + UserDataType, +) +from dl_core.data_source_spec.sql import ( + StandardSchemaSQLDataSourceSpec, + SubselectDataSourceSpec, +) +from dl_core.db import SchemaColumn +from dl_core_testing.fixtures.sample_tables import TABLE_SPEC_SAMPLE_SUPERSTORE +from dl_core_testing.testcases.data_source import ( + DataSourceTestByViewClass, + DefaultDataSourceTestClass, +) + +from dl_connector_oracle.core.constants import ( + SOURCE_TYPE_ORACLE_SUBSELECT, + SOURCE_TYPE_ORACLE_TABLE, +) +from dl_connector_oracle.core.data_source import ( + OracleDataSource, + OracleSubselectDataSource, +) +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle +from dl_connector_oracle_tests.db.config import SUBSELECT_QUERY_FULL +from dl_connector_oracle_tests.db.core.base import BaseOracleTestClass + + +def _update_utype_for_oracle(user_type: UserDataType) -> UserDataType: + if user_type == UserDataType.date: + return UserDataType.genericdatetime + return user_type + + +SAMPLE_TABLE_SCHEMA_SUPERSTORE_ORACLIZED = [ + (name, _update_utype_for_oracle(user_type)) for name, user_type in TABLE_SPEC_SAMPLE_SUPERSTORE.table_schema +] + + +class TestOracleTableDataSource( + BaseOracleTestClass, + DefaultDataSourceTestClass[ + ConnectionSQLOracle, + StandardSchemaSQLDataSourceSpec, + OracleDataSource, + ], +): + DSRC_CLS = OracleDataSource + + @pytest.fixture(scope="class") + def initial_data_source_spec(self, sample_table) -> StandardSchemaSQLDataSourceSpec: + dsrc_spec = StandardSchemaSQLDataSourceSpec( + source_type=SOURCE_TYPE_ORACLE_TABLE, + db_name=sample_table.db.name, + schema_name=sample_table.schema, + table_name=sample_table.name, + ) + return dsrc_spec + + def get_expected_simplified_schema(self) -> list[tuple[str, UserDataType]]: + return list(SAMPLE_TABLE_SCHEMA_SUPERSTORE_ORACLIZED) + + +class TestOracleSubselectDataSource( + BaseOracleTestClass, + DefaultDataSourceTestClass[ + ConnectionSQLOracle, + SubselectDataSourceSpec, + OracleSubselectDataSource, + ], +): + DSRC_CLS = OracleSubselectDataSource + + raw_sql_level = RawSQLLevel.subselect + + @pytest.fixture(scope="class") + def initial_data_source_spec(self, sample_table) -> SubselectDataSourceSpec: + dsrc_spec = SubselectDataSourceSpec( + source_type=SOURCE_TYPE_ORACLE_SUBSELECT, + subsql=f'SELECT * FROM "{sample_table.name}"', + ) + return dsrc_spec + + def get_expected_simplified_schema(self) -> list[tuple[str, UserDataType]]: + return list(SAMPLE_TABLE_SCHEMA_SUPERSTORE_ORACLIZED) + + +class TestOracleSubselectByView( + BaseOracleTestClass, + DataSourceTestByViewClass[ + ConnectionSQLOracle, + SubselectDataSourceSpec, + OracleSubselectDataSource, + ], +): + DSRC_CLS = OracleSubselectDataSource + + raw_sql_level = RawSQLLevel.subselect + + @pytest.fixture(scope="session") + def initial_data_source_spec(self) -> SubselectDataSourceSpec: + dsrc_spec = SubselectDataSourceSpec( + source_type=SOURCE_TYPE_ORACLE_SUBSELECT, + subsql=SUBSELECT_QUERY_FULL, + ) + return dsrc_spec + + def postprocess_view_schema( + self, view_schema: list[SchemaColumn], cursor_schema: list[SchemaColumn] + ) -> list[SchemaColumn]: + result = super().postprocess_view_schema(view_schema, cursor_schema=cursor_schema) + if len(view_schema) != len(cursor_schema): + return result + + # Actual result seems to depend on the cx_Oracle version too much, + # and DL uses these equivalently anyway. + tnames = ("binary_double", "binary_float") + for idx, schema_col in enumerate(result): + if schema_col.native_type.name in tnames: + cs_name = cursor_schema[idx].native_type.name + if cs_name in tnames: + schema_col = schema_col.clone(native_type=schema_col.native_type.clone(name=cs_name)) + result[idx] = schema_col + return result diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_dataset.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_dataset.py new file mode 100644 index 000000000..3578d6221 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/core/test_dataset.py @@ -0,0 +1,16 @@ +from dl_core_testing.testcases.dataset import DefaultDatasetTestSuite +from dl_testing.regulated_test import RegulatedTestParams + +from dl_connector_oracle.core.constants import SOURCE_TYPE_ORACLE_TABLE +from dl_connector_oracle.core.us_connection import ConnectionSQLOracle +from dl_connector_oracle_tests.db.core.base import BaseOracleTestClass + + +class TestOracleDataset(BaseOracleTestClass, DefaultDatasetTestSuite[ConnectionSQLOracle]): + source_type = SOURCE_TYPE_ORACLE_TABLE + + test_params = RegulatedTestParams( + mark_tests_failed={ + DefaultDatasetTestSuite.test_get_param_hash: "db_name in dsrc", # TODO: FIXME + }, + ) diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/base.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/base.py new file mode 100644 index 000000000..060bf71e0 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/base.py @@ -0,0 +1,19 @@ +import pytest + +from dl_formula_testing.testcases.base import FormulaConnectorTestBase + +from dl_connector_oracle.formula.constants import OracleDialect as D +from dl_connector_oracle_tests.db.config import DB_URLS + + +class OracleTestBase(FormulaConnectorTestBase): + dialect = D.ORACLE_12_0 + supports_arrays = False + supports_uuid = False + bool_is_expression = True + empty_str_is_null = True # '' and NULL are the same thing + null_casts_to_number = True + + @pytest.fixture(scope="class") + def db_url(self) -> str: + return DB_URLS[self.dialect] diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_conditional_blocks.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_conditional_blocks.py new file mode 100644 index 000000000..235d79333 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_conditional_blocks.py @@ -0,0 +1,7 @@ +from dl_formula_testing.testcases.conditional_blocks import DefaultConditionalBlockFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestConditionalBlockOracle(OracleTestBase, DefaultConditionalBlockFormulaConnectorTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_aggregation.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_aggregation.py new file mode 100644 index 000000000..379ea266c --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_aggregation.py @@ -0,0 +1,9 @@ +from dl_formula_testing.testcases.functions_aggregation import DefaultMainAggFunctionFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestMainAggFunctionOracle(OracleTestBase, DefaultMainAggFunctionFormulaConnectorTestSuite): + supports_countd_approx = True + supports_quantile = True + supports_median = True diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_datetime.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_datetime.py new file mode 100644 index 000000000..8bd76c126 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_datetime.py @@ -0,0 +1,8 @@ +from dl_formula_testing.testcases.functions_datetime import DefaultDateTimeFunctionFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestDateTimeFunctionOracle(OracleTestBase, DefaultDateTimeFunctionFormulaConnectorTestSuite): + supports_addition_to_feb_29 = False + supports_deprecated_dateadd = True diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_logical.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_logical.py new file mode 100644 index 000000000..7875d4ac6 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_logical.py @@ -0,0 +1,7 @@ +from dl_formula_testing.testcases.functions_logical import DefaultLogicalFunctionFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestLogicalFunctionOracle(OracleTestBase, DefaultLogicalFunctionFormulaConnectorTestSuite): + supports_iif = True diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_markup.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_markup.py new file mode 100644 index 000000000..50aeb0f29 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_markup.py @@ -0,0 +1,7 @@ +from dl_formula_testing.testcases.functions_markup import DefaultMarkupFunctionFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestMarkupFunctionOracle(OracleTestBase, DefaultMarkupFunctionFormulaConnectorTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_math.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_math.py new file mode 100644 index 000000000..0f09e69a7 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_math.py @@ -0,0 +1,7 @@ +from dl_formula_testing.testcases.functions_math import DefaultMathFunctionFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestMathFunctionOracle(OracleTestBase, DefaultMathFunctionFormulaConnectorTestSuite): + supports_atan_2_in_origin = False diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_string.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_string.py new file mode 100644 index 000000000..3c5ef541f --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_string.py @@ -0,0 +1,17 @@ +import os + +import sqlalchemy as sa + +from dl_formula_testing.evaluator import DbEvaluator +from dl_formula_testing.testcases.functions_string import DefaultStringFunctionFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestStringFunctionOracle(OracleTestBase, DefaultStringFunctionFormulaConnectorTestSuite): + supports_split_3 = False + + def test_oracle_unicode(self, dbe: DbEvaluator) -> None: + assert os.environ.get("NLS_LANG") == ".AL32UTF8" + # assert dbe.db.scalar(sa.select([sa.literal('карл')])) == 'карл' # not working. + assert dbe.db.scalar(sa.literal("карл", type_=sa.sql.sqltypes.NCHAR())) == "карл" diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_type_conversion.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_type_conversion.py new file mode 100644 index 000000000..fb4ab4633 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_functions_type_conversion.py @@ -0,0 +1,78 @@ +from dl_formula_testing.testcases.functions_type_conversion import ( + DefaultBoolTypeFunctionFormulaConnectorTestSuite, + DefaultDateTypeFunctionFormulaConnectorTestSuite, + DefaultFloatTypeFunctionFormulaConnectorTestSuite, + DefaultGenericDatetimeTypeFunctionFormulaConnectorTestSuite, + DefaultGeopointTypeFunctionFormulaConnectorTestSuite, + DefaultGeopolygonTypeFunctionFormulaConnectorTestSuite, + DefaultIntTypeFunctionFormulaConnectorTestSuite, + DefaultStrTypeFunctionFormulaConnectorTestSuite, +) + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +# STR + + +class TestStrTypeFunctionOracle(OracleTestBase, DefaultStrTypeFunctionFormulaConnectorTestSuite): + zero_float_to_str_value = "0" + skip_custom_tz = True + + +# FLOAT + + +class TestFloatTypeFunctionOracle(OracleTestBase, DefaultFloatTypeFunctionFormulaConnectorTestSuite): + pass + + +# BOOL + + +class TestBoolTypeFunctionOracle(OracleTestBase, DefaultBoolTypeFunctionFormulaConnectorTestSuite): + pass + + +# INT + + +class TestIntTypeFunctionOracle(OracleTestBase, DefaultIntTypeFunctionFormulaConnectorTestSuite): + pass + + +# DATE + + +class TestDateTypeFunctionOracle(OracleTestBase, DefaultDateTypeFunctionFormulaConnectorTestSuite): + pass + + +# GENERICDATETIME (& DATETIME) + + +class TestGenericDatetimeTypeFunctionOracle( + OracleTestBase, + DefaultGenericDatetimeTypeFunctionFormulaConnectorTestSuite, +): + pass + + +# GEOPOINT + + +class TestGeopointTypeFunctionOracle( + OracleTestBase, + DefaultGeopointTypeFunctionFormulaConnectorTestSuite, +): + pass + + +# GEOPOLYGON + + +class TestGeopolygonTypeFunctionOracle( + OracleTestBase, + DefaultGeopolygonTypeFunctionFormulaConnectorTestSuite, +): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_literals.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_literals.py new file mode 100644 index 000000000..71214d566 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_literals.py @@ -0,0 +1,9 @@ +from dl_formula_testing.testcases.literals import DefaultLiteralFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestConditionalBlockOracle(OracleTestBase, DefaultLiteralFormulaConnectorTestSuite): + supports_microseconds = False + supports_utc = False + supports_custom_tz = False diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_misc_funcs.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_misc_funcs.py new file mode 100644 index 000000000..4c4002b33 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_misc_funcs.py @@ -0,0 +1,7 @@ +from dl_formula_testing.testcases.misc_funcs import DefaultMiscFunctionalityConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestMiscFunctionalityOracle(OracleTestBase, DefaultMiscFunctionalityConnectorTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_operators.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_operators.py new file mode 100644 index 000000000..59dfcc78b --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/db/formula/test_operators.py @@ -0,0 +1,7 @@ +from dl_formula_testing.testcases.operators import DefaultOperatorFormulaConnectorTestSuite + +from dl_connector_oracle_tests.db.formula.base import OracleTestBase + + +class TestOperatorOracle(OracleTestBase, DefaultOperatorFormulaConnectorTestSuite): + pass diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/__init__.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/conftest.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/conftest.py new file mode 100644 index 000000000..e69de29bb diff --git a/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/test_connection_form.py b/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/test_connection_form.py new file mode 100644 index 000000000..bdd6a4208 --- /dev/null +++ b/lib/dl_connector_oracle/dl_connector_oracle_tests/unit/test_connection_form.py @@ -0,0 +1,10 @@ +from dl_api_connector.i18n.localizer import CONFIGS as BI_API_CONNECTOR_CONFIGS +from dl_api_lib_testing.connection_form_base import ConnectionFormTestBase + +from dl_connector_oracle.api.connection_form.form_config import OracleConnectionFormFactory +from dl_connector_oracle.api.i18n.localizer import CONFIGS as BI_CONNECTOR_ORACLE_CONFIGS + + +class TestOracleConnectionForm(ConnectionFormTestBase): + CONN_FORM_FACTORY_CLS = OracleConnectionFormFactory + TRANSLATION_CONFIGS = BI_CONNECTOR_ORACLE_CONFIGS + BI_API_CONNECTOR_CONFIGS diff --git a/lib/dl_connector_oracle/docker-compose.yml b/lib/dl_connector_oracle/docker-compose.yml new file mode 100644 index 000000000..a6e4cab23 --- /dev/null +++ b/lib/dl_connector_oracle/docker-compose.yml @@ -0,0 +1,42 @@ +version: '3.7' + +x-constants: + US_MASTER_TOKEN: &c-us-master-token "AC1ofiek8coB" + +services: + db-oracle: + labels: + datalens.ci.service: db-oracle + ports: + - "51800:1521" + # image: "ghcr.io/gvenzl/oracle-xe:21.3.0-slim-faststart" + image: "ghcr.io/gvenzl/oracle-xe:18-slim-faststart" + environment: + ORACLE_PASSWORD: "qwerty" + APP_USER: "datalens" + APP_USER_PASSWORD: "qwerty" + + # INFRA + pg-us: + build: + context: ../../mainrepo/lib/testenv-common/images + dockerfile: Dockerfile.pg-us + environment: + POSTGRES_DB: us-db-ci_purgeable + POSTGRES_USER: us + POSTGRES_PASSWORD: us + ports: + - "51810:5432" + + us: + build: + context: ../../mainrepo/lib/testenv-common/images + dockerfile: Dockerfile.us + depends_on: + - pg-us + environment: + POSTGRES_DSN_LIST: "postgres://us:us@pg-us:5432/us-db-ci_purgeable" + AUTH_POLICY: "required" + MASTER_TOKEN: *c-us-master-token + ports: + - "51811:80" diff --git a/lib/dl_connector_oracle/pyproject.toml b/lib/dl_connector_oracle/pyproject.toml new file mode 100644 index 000000000..ec5ebd450 --- /dev/null +++ b/lib/dl_connector_oracle/pyproject.toml @@ -0,0 +1,91 @@ + +[tool.poetry] +name = "datalens-connector-oracle" +version = "0.0.1" +description = "" +authors = ["Grigory Statsenko "] +packages = [{include = "dl_connector_oracle"}] +license = "MIT" +readme = "README.md" + + +[tool.poetry.dependencies] +attrs = ">=22.2.0" +marshmallow = ">=3.19.0" +oracledb = ">=1.3.1" +python = ">=3.10, <3.12" +sqlalchemy = ">=1.4.46, <2.0" +datalens-api-commons = {path = "../dl_api_commons"} +datalens-constants = {path = "../dl_constants"} +datalens-formula-ref = {path = "../dl_formula_ref"} +datalens-i18n = {path = "../dl_i18n"} +datalens-formula = {path = "../dl_formula"} +datalens-configs = {path = "../dl_configs"} +datalens-api-connector = {path = "../dl_api_connector"} +datalens-core = {path = "../dl_core"} +datalens-sqlalchemy-oracle = {path = "../dl_sqlalchemy_oracle"} +datalens-db-testing = {path = "../dl_db_testing"} + +[tool.poetry.plugins] +[tool.poetry.plugins."dl_api_lib.connectors"] +oracle = "dl_connector_oracle.api.connector:OracleApiConnector" + +[tool.poetry.plugins."dl_core.connectors"] +oracle = "dl_connector_oracle.core.connector:OracleCoreConnector" + +[tool.poetry.plugins."dl_db_testing.connectors"] +oracle = "dl_connector_oracle.db_testing.connector:OracleDbTestingConnector" + +[tool.poetry.plugins."dl_formula.connectors"] +oracle = "dl_connector_oracle.formula.connector:OracleFormulaConnector" + +[tool.poetry.plugins."dl_formula_ref.plugins"] +oracle = "dl_connector_oracle.formula_ref.plugin:OracleFormulaRefPlugin" + +[tool.poetry.group] +[tool.poetry.group.tests.dependencies] +oracledb = ">=1.3.1" +pytest = ">=7.2.2" +pytest-asyncio = ">=0.20.3" +datalens-api-lib-testing = {path = "../dl_api_lib_testing"} +datalens-formula-testing = {path = "../dl_formula_testing"} +datalens-testing = {path = "../dl_testing"} +datalens-core-testing = {path = "../dl_core_testing"} + +[build-system] +build-backend = "poetry.core.masonry.api" +requires = [ + "poetry-core", +] + +[tool.pytest.ini_options] +minversion = "6.0" +addopts = "-ra" +testpaths = ["dl_connector_oracle_tests/db", "dl_connector_oracle_tests/unit"] + + + +[datalens.pytest.db] +root_dir = "dl_connector_oracle_tests/" +target_path = "db unit" +labels = ["fat"] + +[datalens.pytest.unit] +root_dir = "dl_connector_oracle_tests/" +target_path = "unit" +skip_compose = "true" + +[tool.mypy] +warn_unused_configs = true +disallow_untyped_defs = true +check_untyped_defs = true +strict_optional = true + +[datalens.i18n.domains] +dl_connector_oracle = [ + {path = "dl_connector_oracle/api"}, + {path = "dl_connector_oracle/core"}, +] +dl_formula_ref_dl_connector_oracle = [ + {path = "dl_connector_oracle/formula_ref"}, +] diff --git a/metapkg/poetry.lock b/metapkg/poetry.lock index 6497ef78f..a14546db6 100644 --- a/metapkg/poetry.lock +++ b/metapkg/poetry.lock @@ -1331,6 +1331,35 @@ datalens-i18n = {path = "../dl_i18n"} type = "directory" url = "../lib/dl_connector_greenplum" +[[package]] +name = "datalens-connector-oracle" +version = "0.0.1" +description = "" +optional = false +python-versions = ">=3.10, <3.12" +files = [] +develop = false + +[package.dependencies] +attrs = ">=22.2.0" +datalens-api-commons = {path = "../dl_api_commons"} +datalens-api-connector = {path = "../dl_api_connector"} +datalens-configs = {path = "../dl_configs"} +datalens-constants = {path = "../dl_constants"} +datalens-core = {path = "../dl_core"} +datalens-db-testing = {path = "../dl_db_testing"} +datalens-formula = {path = "../dl_formula"} +datalens-formula-ref = {path = "../dl_formula_ref"} +datalens-i18n = {path = "../dl_i18n"} +datalens-sqlalchemy-oracle = {path = "../dl_sqlalchemy_oracle"} +marshmallow = ">=3.19.0" +oracledb = ">=1.3.1" +sqlalchemy = ">=1.4.46, <2.0" + +[package.source] +type = "directory" +url = "../lib/dl_connector_oracle" + [[package]] name = "datalens-connector-postgresql" version = "0.0.1" @@ -2588,6 +2617,7 @@ files = [ {file = "greenlet-2.0.2-cp27-cp27m-win32.whl", hash = "sha256:6c3acb79b0bfd4fe733dff8bc62695283b57949ebcca05ae5c129eb606ff2d74"}, {file = "greenlet-2.0.2-cp27-cp27m-win_amd64.whl", hash = "sha256:283737e0da3f08bd637b5ad058507e578dd462db259f7f6e4c5c365ba4ee9343"}, {file = "greenlet-2.0.2-cp27-cp27mu-manylinux2010_x86_64.whl", hash = "sha256:d27ec7509b9c18b6d73f2f5ede2622441de812e7b1a80bbd446cb0633bd3d5ae"}, + {file = "greenlet-2.0.2-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:d967650d3f56af314b72df7089d96cda1083a7fc2da05b375d2bc48c82ab3f3c"}, {file = "greenlet-2.0.2-cp310-cp310-macosx_11_0_x86_64.whl", hash = "sha256:30bcf80dda7f15ac77ba5af2b961bdd9dbc77fd4ac6105cee85b0d0a5fcf74df"}, {file = "greenlet-2.0.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:26fbfce90728d82bc9e6c38ea4d038cba20b7faf8a0ca53a9c07b67318d46088"}, {file = "greenlet-2.0.2-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9190f09060ea4debddd24665d6804b995a9c122ef5917ab26e1566dcc712ceeb"}, @@ -2596,6 +2626,7 @@ files = [ {file = "greenlet-2.0.2-cp310-cp310-musllinux_1_1_x86_64.whl", hash = "sha256:76ae285c8104046b3a7f06b42f29c7b73f77683df18c49ab5af7983994c2dd91"}, {file = "greenlet-2.0.2-cp310-cp310-win_amd64.whl", hash = "sha256:2d4686f195e32d36b4d7cf2d166857dbd0ee9f3d20ae349b6bf8afc8485b3645"}, {file = "greenlet-2.0.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:c4302695ad8027363e96311df24ee28978162cdcdd2006476c43970b384a244c"}, + {file = "greenlet-2.0.2-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:d4606a527e30548153be1a9f155f4e283d109ffba663a15856089fb55f933e47"}, {file = "greenlet-2.0.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c48f54ef8e05f04d6eff74b8233f6063cb1ed960243eacc474ee73a2ea8573ca"}, {file = "greenlet-2.0.2-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a1846f1b999e78e13837c93c778dcfc3365902cfb8d1bdb7dd73ead37059f0d0"}, {file = "greenlet-2.0.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3a06ad5312349fec0ab944664b01d26f8d1f05009566339ac6f63f56589bc1a2"}, @@ -2625,6 +2656,7 @@ files = [ {file = "greenlet-2.0.2-cp37-cp37m-win32.whl", hash = "sha256:3f6ea9bd35eb450837a3d80e77b517ea5bc56b4647f5502cd28de13675ee12f7"}, {file = "greenlet-2.0.2-cp37-cp37m-win_amd64.whl", hash = "sha256:7492e2b7bd7c9b9916388d9df23fa49d9b88ac0640db0a5b4ecc2b653bf451e3"}, {file = "greenlet-2.0.2-cp38-cp38-macosx_10_15_x86_64.whl", hash = "sha256:b864ba53912b6c3ab6bcb2beb19f19edd01a6bfcbdfe1f37ddd1778abfe75a30"}, + {file = "greenlet-2.0.2-cp38-cp38-macosx_11_0_arm64.whl", hash = "sha256:1087300cf9700bbf455b1b97e24db18f2f77b55302a68272c56209d5587c12d1"}, {file = "greenlet-2.0.2-cp38-cp38-manylinux2010_x86_64.whl", hash = "sha256:ba2956617f1c42598a308a84c6cf021a90ff3862eddafd20c3333d50f0edb45b"}, {file = "greenlet-2.0.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:fc3a569657468b6f3fb60587e48356fe512c1754ca05a564f11366ac9e306526"}, {file = "greenlet-2.0.2-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8eab883b3b2a38cc1e050819ef06a7e6344d4a990d24d45bc6f2cf959045a45b"}, @@ -2633,6 +2665,7 @@ files = [ {file = "greenlet-2.0.2-cp38-cp38-musllinux_1_1_x86_64.whl", hash = "sha256:b0ef99cdbe2b682b9ccbb964743a6aca37905fda5e0452e5ee239b1654d37f2a"}, {file = "greenlet-2.0.2-cp38-cp38-win32.whl", hash = "sha256:b80f600eddddce72320dbbc8e3784d16bd3fb7b517e82476d8da921f27d4b249"}, {file = "greenlet-2.0.2-cp38-cp38-win_amd64.whl", hash = "sha256:4d2e11331fc0c02b6e84b0d28ece3a36e0548ee1a1ce9ddde03752d9b79bba40"}, + {file = "greenlet-2.0.2-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:8512a0c38cfd4e66a858ddd1b17705587900dd760c6003998e9472b77b56d417"}, {file = "greenlet-2.0.2-cp39-cp39-macosx_11_0_x86_64.whl", hash = "sha256:88d9ab96491d38a5ab7c56dd7a3cc37d83336ecc564e4e8816dbed12e5aaefc8"}, {file = "greenlet-2.0.2-cp39-cp39-manylinux2010_x86_64.whl", hash = "sha256:561091a7be172ab497a3527602d467e2b3fbe75f9e783d8b8ce403fa414f71a6"}, {file = "greenlet-2.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:971ce5e14dc5e73715755d0ca2975ac88cfdaefcaab078a284fea6cfabf866df"}, @@ -3202,6 +3235,16 @@ files = [ {file = "MarkupSafe-2.1.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:5bbe06f8eeafd38e5d0a4894ffec89378b6c6a625ff57e3028921f8ff59318ac"}, {file = "MarkupSafe-2.1.3-cp311-cp311-win32.whl", hash = "sha256:dd15ff04ffd7e05ffcb7fe79f1b98041b8ea30ae9234aed2a9168b5797c3effb"}, {file = "MarkupSafe-2.1.3-cp311-cp311-win_amd64.whl", hash = "sha256:134da1eca9ec0ae528110ccc9e48041e0828d79f24121a1a146161103c76e686"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:f698de3fd0c4e6972b92290a45bd9b1536bffe8c6759c62471efaa8acb4c37bc"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:aa57bd9cf8ae831a362185ee444e15a93ecb2e344c8e52e4d721ea3ab6ef1823"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ffcc3f7c66b5f5b7931a5aa68fc9cecc51e685ef90282f4a82f0f5e9b704ad11"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47d4f1c5f80fc62fdd7777d0d40a2e9dda0a05883ab11374334f6c4de38adffd"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:1f67c7038d560d92149c060157d623c542173016c4babc0c1913cca0564b9939"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash = "sha256:9aad3c1755095ce347e26488214ef77e0485a3c34a50c5a5e2471dff60b9dd9c"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_i686.whl", hash = "sha256:14ff806850827afd6b07a5f32bd917fb7f45b046ba40c57abdb636674a8b559c"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash = "sha256:8f9293864fe09b8149f0cc42ce56e3f0e54de883a9de90cd427f191c346eb2e1"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-win32.whl", hash = "sha256:715d3562f79d540f251b99ebd6d8baa547118974341db04f5ad06d5ea3eb8007"}, + {file = "MarkupSafe-2.1.3-cp312-cp312-win_amd64.whl", hash = "sha256:1b8dd8c3fd14349433c79fa8abeb573a55fc0fdd769133baac1f5e07abf54aeb"}, {file = "MarkupSafe-2.1.3-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:8e254ae696c88d98da6555f5ace2279cf7cd5b3f52be2b5cf97feafe883b58d2"}, {file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cb0932dc158471523c9637e807d9bfb93e06a95cbf010f1a38b98623b929ef2b"}, {file = "MarkupSafe-2.1.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9402b03f1a1b4dc4c19845e5c749e3ab82d5078d16a2a4c2cd2df62d57bb0707"}, @@ -5662,4 +5705,4 @@ testing = ["big-O", "flake8 (<5)", "jaraco.functools", "jaraco.itertools", "more [metadata] lock-version = "2.0" python-versions = ">=3.10, <3.12" -content-hash = "9fd76caa30603ae3e611eafefc4ad420784c44cc3c95966ff5c1b1fb1225673e" +content-hash = "6d76ec4a72df400e56946137de5856df445ed9b8203510d58f3b83e0b7871ec7" diff --git a/metapkg/pyproject.toml b/metapkg/pyproject.toml index a8b16a182..bc9b2157b 100644 --- a/metapkg/pyproject.toml +++ b/metapkg/pyproject.toml @@ -130,6 +130,7 @@ datalens-db-testing = {path = "../lib/dl_db_testing"} datalens-file-uploader-worker-lib = {path = "../lib/dl_file_uploader_worker_lib"} datalens-connector-bigquery = {path = "../lib/dl_connector_bigquery"} datalens-task-processor = {path = "../lib/dl_task_processor"} +datalens-connector-oracle = {path = "../lib/dl_connector_oracle"} # # This section defines dependencies groups per app.