Merge branch 'main' into sr-factory-in-tests

.gitignore

@@ -15,3 +15,4 @@ ci_artifacts
 .DS_Store
 artifacts
 Taskfile.yml
+.obsidian

README.md

@@ -1,40 +1,15 @@
-## Management tasks
+# datalens-backend
 
-Running tasks:
-```
-task <task_name>
-```
+## About
 
+This is the repository for the back-end implementation of DataLens
 
-### Environment (`env:`)
+Head over to the [Knowledge Base](kb/index.md) for documentation on this repo.
 
-- `task env:devenv`:
-  Create development/testing environment (run it from a package dir)
-- `task env:devenv-d`:
-  Create development/testing environment in detached mode (run it from a package dir)
-- `task env:ensure_venv`: Command to create virtual env for the mainrepo tools. 
-  It requires presence of .env in the mainrepo/tools.
+[Code of conduct](CODE_OF_CONDUCT.md)
 
+[Contributing](CONTRIBUTING.md)
 
-### Generation (`gen:`)
+## License
 
-- `task gen:antlr`:
-  (Re-)generate ANTLR code files for formula
-- `task gen:i18n-po`:
-  Sync/generate `.po` files for package (run it from a package dir)
-- `task gen:i18n-binaries`:
-  Generate binary `.mo` files from `.po` files for package (run it from a package dir)
-
-
-### Code quality (`cq:`)
-
-Experimental tasks to check and fix source files. 
-
-- `task cq:fix_changed`:
-  Apply all auto-fixes
-- `task cq:check_changed`:
-  Check for any non-conformity in code style/format/lint
-- `task cq:fix_dir -- {single dir}`:
-  Apply all auto-fixes to the given dir absolute path
-- `task cq:check_dir -- {single dir}`:
-  Check for any non-conformity in code style/format/lint in the given dir abs path
+`datalens-backend` is available under the Apache 2.0 license.

kb/index.md

@@ -0,0 +1,7 @@
+# Datalens Backend Knowledge Base
+
+Welcome to the KB!
+
+Topics:
+- [Working with this KB](using_kb.md) (editor configuration)
+- [Repository tooling](tooling/index.md)

kb/test_embeds.md

@@ -0,0 +1,15 @@
+This is a file to test embedded diagrams. 
+
+
+``` plantuml
+@startuml
+digraph foo {
+  node [style=rounded]
+  node1 [shape=box]
+  node2 [fillcolor=yellow, style="rounded,filled", shape=diamond]
+  node3 [shape=record, label="{ a | b | c }"]
+
+  node1 -> node2 -> node3
+};
+@enduml
+```

kb/tooling/index.md

@@ -0,0 +1,8 @@
+# Repository Tooling
+
+This section is about the custom tooling available in this repo.
+
+- [task commands](task_commands.md) - a set of (`make`-like) shortcuts for various commands and scripts for repository management, development, testing, etc.
+- tools from `terrarium` ([README](../../terrarium/README.md)):
+  - [dl-git](../../terrarium/dl_gitmanager/README.md) - a wrapper for advanced git commands, for usage mainly in the CI workflow
+  - [dl-repo / dl-package](../../terrarium/dl_repmanager/README.md) - tools for managing and inspecting packages, their dependencies, meta-packages, etc.

kb/tooling/task_commands.md

@@ -0,0 +1,47 @@
+## Management tasks
+
+These commands require the ``taskfile`` tool. See [this page](https://taskfile.dev/installation/)
+for installation options.
+
+Running tasks:
+```
+task <task_name>
+```
+
+
+### Environment (`env:`)
+
+Working with the testing/development environment.
+
+- `task env:devenv`:
+  Create development/testing environment (run it from a package dir)
+- `task env:devenv-d`:
+  Create development/testing environment in detached mode (run it from a package dir)
+- `task env:ensure_venv`: Command to create virtual env for the mainrepo tools. 
+  It requires presence of .env in the mainrepo/tools.
+
+
+### Generation (`gen:`)
+
+Generating files to be used from the code.
+
+- `task gen:antlr`:
+  (Re-)generate ANTLR code files for formula
+- `task gen:i18n-po`:
+  Sync/generate `.po` files for package (run it from a package dir)
+- `task gen:i18n-binaries`:
+  Generate binary `.mo` files from `.po` files for package (run it from a package dir)
+
+
+### Code quality (`cq:`)
+
+Checking and fixing source files. 
+
+- `task cq:fix_changed`:
+  Apply all auto-fixes
+- `task cq:check_changed`:
+  Check for any non-conformity in code style/format/lint
+- `task cq:fix_dir -- {single dir}`:
+  Apply all auto-fixes to the given dir absolute path
+- `task cq:check_dir -- {single dir}`:
+  Check for any non-conformity in code style/format/lint in the given dir abs path

kb/using_kb.md

@@ -0,0 +1,27 @@
+# Working with the KB
+
+This KB is mostly `Markdown`, but it has embedded PlantUML charts.
+These usually don't work out-of-the-box and might require a little additional configuration.
+
+To check the correect rendering of embedded charts open [this file](test_embeds.md).
+If your editor supports this, and everything is configured correctly,
+you should see a rendered charts.
+
+Here are two options you can use to work with this KB
+
+## PyCharm
+
+- Install `graphviz`.
+- Install and enable the Markdown plugin in PyCharm.
+
+In theory this should be enough, but you may find that charts give you a rendering error
+about not finding "dot".
+In this case find the `dot` executable in your system and copy it to `/opt/local/bin/dot`.
+It should work now.
+
+## Obsidian
+
+- Install the `Obsidian` app
+- Install and enable the PlantUML plugin in Obsidian.
+  You might need to configure the path to the dot executable.
+- Open the kb folder as a vault

lib/dl_api_lib/dl_api_lib_tests/db/data_api/result/test_parameters.py

@@ -0,0 +1,100 @@
+from __future__ import annotations
+
+from http import HTTPStatus
+
+import pytest
+
+from dl_api_client.dsmaker.primitives import (
+    IntegerParameterValue,
+    RangeParameterValueConstraint,
+)
+from dl_api_client.dsmaker.shortcuts.dataset import (
+    add_formulas_to_dataset,
+    add_parameters_to_dataset,
+)
+from dl_api_client.dsmaker.shortcuts.result_data import get_data_rows
+from dl_api_lib_tests.db.base import DefaultApiTestBase
+from dl_constants.enums import UserDataType
+
+
+class TestParameters(DefaultApiTestBase):
+    @pytest.mark.parametrize(
+        ("multiplier", "expected_status_code"),
+        (
+            (None, HTTPStatus.OK),
+            (2, HTTPStatus.OK),
+            (5, HTTPStatus.OK),
+            (-1, HTTPStatus.BAD_REQUEST),
+        ),
+    )
+    def test_parameter_in_formula(self, control_api, data_api, saved_dataset, multiplier, expected_status_code):
+        default_multiplier = 1
+        ds = add_parameters_to_dataset(
+            api_v1=control_api,
+            dataset_id=saved_dataset.id,
+            parameters={
+                "Multiplier": (
+                    IntegerParameterValue(default_multiplier),
+                    RangeParameterValueConstraint(min=IntegerParameterValue(default_multiplier)),
+                ),
+            },
+        )
+
+        integer_field = next(field for field in saved_dataset.result_schema if field.data_type == UserDataType.integer)
+        ds = add_formulas_to_dataset(
+            api_v1=control_api,
+            dataset=ds,
+            formulas={
+                "Multiplied Field": f"[{integer_field.title}] * [Multiplier]",
+            },
+        )
+
+        result_resp = data_api.get_result(
+            dataset=ds,
+            fields=[
+                integer_field,
+                ds.find_field(title="Multiplier"),
+                ds.find_field(title="Multiplied Field"),
+            ],
+            parameters=[
+                ds.find_field(title="Multiplier").parameter_value(multiplier),
+            ],
+            fail_ok=True,
+        )
+        assert result_resp.status_code == expected_status_code, result_resp.json
+
+        if expected_status_code == HTTPStatus.OK:
+            data_rows = get_data_rows(result_resp)
+            assert data_rows
+            for row in data_rows:
+                assert int(row[1]) == (multiplier or default_multiplier)
+                assert int(row[0]) * int(row[1]) == int(row[2])
+
+    def test_parameter_no_constraint(self, control_api, data_api, dataset_id):
+        ds = add_parameters_to_dataset(
+            api_v1=control_api,
+            dataset_id=dataset_id,
+            parameters={
+                "Param": (IntegerParameterValue(0), None),
+            },
+        )
+        ds = add_formulas_to_dataset(
+            api_v1=control_api,
+            dataset=ds,
+            formulas={
+                "Value": "[Param]",
+            },
+        )
+
+        result_resp = data_api.get_result(
+            dataset=ds,
+            fields=[
+                ds.find_field(title="Value"),
+            ],
+            parameters=[
+                ds.find_field(title="Param").parameter_value(1),
+            ],
+            limit=1,
+        )
+        assert result_resp.status_code == HTTPStatus.OK, result_resp.json
+        assert int(get_data_rows(result_resp)[0][0]) == 1

lib/dl_connector_mysql/dl_connector_mysql/core/async_adapters_mysql.py

@@ -146,11 +146,17 @@ async def _execute_by_steps(self, db_adapter_query: DBAdapterQuery) -> AsyncIter
 
         chunk_size = db_adapter_query.get_effective_chunk_size(self._default_chunk_size)
         query = db_adapter_query.query
+        debug_compiled_query = db_adapter_query.debug_compiled_query
         escape_percent = not db_adapter_query.is_dashsql_query  # DON'T escape only for dashsql
         compiled_query, compiled_query_parameters = compile_mysql_query(
             query, dialect=self._dialect, escape_percent=escape_percent
         )
-        debug_query = query if isinstance(query, str) else compile_query_for_debug(query, self._dialect)
+        debug_query = None
+        if self._target_dto.pass_db_query_to_user:
+            if debug_compiled_query is not None:
+                debug_query = debug_compiled_query
+            else:
+                debug_query = query if isinstance(query, str) else compile_query_for_debug(query, self._dialect)
 
         with self.handle_execution_error(debug_query):
             async with self._get_connection(db_adapter_query.db_name) as conn:

lib/dl_connector_mysql/dl_connector_mysql/core/error_transformer.py

@@ -36,6 +36,16 @@ def _(exc: Exception) -> bool:
     return _
 
 
+def is_source_connect_async_error() -> ExcMatchCondition:
+    def _(exc: Exception) -> bool:
+        if isinstance(exc, pymysql.OperationalError):
+            if len(exc.args) >= 2 and exc.args[0] == 2003:
+                return True
+            return False
+
+    return _
+
+
 class AsyncMysqlChainedDbErrorTransformer(error_transformer.ChainedDbErrorTransformer):
     @staticmethod
     def _get_error_kw(
@@ -53,6 +63,10 @@ def _get_error_kw(
 
 async_mysql_db_error_transformer: DbErrorTransformer = AsyncMysqlChainedDbErrorTransformer(
     (
+        Rule(
+            when=is_source_connect_async_error(),
+            then_raise=exc.SourceConnectError,
+        ),
         Rule(
             when=is_table_does_not_exist_async_error(),
             then_raise=MysqlSourceDoesNotExistError,

lib/dl_connector_mysql/dl_connector_mysql_tests/db/core/test_adapter.py

@@ -0,0 +1,19 @@
+from dl_core_testing.testcases.adapter import BaseAsyncAdapterTestClass
+from dl_testing.regulated_test import RegulatedTestParams
+
+from dl_connector_mysql.core.async_adapters_mysql import AsyncMySQLAdapter
+from dl_connector_mysql.core.target_dto import MySQLConnTargetDTO
+from dl_connector_mysql_tests.db.core.base import BaseMySQLTestClass
+
+
+class TestAsyncMySQLAdapter(
+    BaseMySQLTestClass,
+    BaseAsyncAdapterTestClass[MySQLConnTargetDTO],
+):
+    test_params = RegulatedTestParams(
+        mark_tests_skipped={
+            BaseAsyncAdapterTestClass.test_default_pass_db_query_to_user: "Not relevant",
+        },
+    )
+
+    ASYNC_ADAPTER_CLS = AsyncMySQLAdapter

terrarium/dl_gitmanager/README.md

@@ -14,13 +14,13 @@ pip install -Ue <path-to-package>
 
 Show the main help message
 
-```
+```bash
 dl-git --help
 dl-git --h
 ```
 
 The `--help` (`-h`) option can also be used for any command:
-```
+```bash
 dl-git <command> --help
 ```
 
@@ -43,7 +43,7 @@ If the path is inside a submodule, then the submodule is considered to be the ro
 
 List files that have changed between two given revisions including the changes in all submodules
 
-```
+```bash
 dl-git range-diff-paths --base <base> --head <head>
 dl-git range-diff-paths --base <base> --head <head> --absolute
 dl-git range-diff-paths --base <base> --head <head> --only-added-commits
@@ -53,7 +53,7 @@ Here `base` and `head` can be a commit ID, branch name, `HEAD~3` or any similar
 that is usually accepted by git.
 These arguments are optional. By default `head` is `HEAD` and `base` is `HEAD~1`.
 Thgis means you can use the following command to see the diff of the last commit in the current branch:
-```
+```bash
 dl-git range-diff-paths
 ```