Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

External Data Sources: RPC implementation for the describe methods + ydb scheme describe support #14509

Merged
merged 4 commits into from
Feb 13, 2025

Conversation

jepett0
Copy link
Collaborator

@jepett0 jepett0 commented Feb 12, 2025

Implement describe external data sources / external tables RPCs. The implementations of the RPCs are almost identical and require deduplication (probably, via a common base class for both RPC implementation classes). However, we are in a hurry and this task:

will be done separately.

ydb scheme describe is also half-baked (only the --format proto-json-base64 works as intended). Pretty format will be done in the next PRs in the series.

Note: YDB CLI is released separately and more often than YDB server versions. Users might end up with a newer CLI that supports enhanced external data source and table descriptions, but the server does not implement the RPCs yet. This users would get an error when they attempt to call the NYdb::NTable::TSession::DescribeExternalDataSource or DescribeExternalTable methods. This is a minor inconvenience, but not a blocker in my opinion. Only the targeted calls like ydb scheme describe my_external_data_source will fail (side note: ydb scheme ls -l -R does not use the new RPCs), so the existing user scripts should be just fine.

Changelog entry

Changelog category

  • Not for changelog (changelog entry is not required)

Additional information

This comment was marked as outdated.

This comment was marked as outdated.

@jepett0 jepett0 force-pushed the ExternalDataSource.describe_rpc.1 branch from 611d523 to 9807afd Compare February 13, 2025 02:26
@jepett0 jepett0 linked an issue Feb 13, 2025 that may be closed by this pull request

This comment was marked as outdated.

This comment was marked as outdated.

@jepett0 jepett0 requested a review from dorooleg February 13, 2025 07:35
pnv1
pnv1 previously approved these changes Feb 13, 2025
Copy link
Collaborator

@pnv1 pnv1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLI part LGTM

Gazizonoki
Gazizonoki previously approved these changes Feb 13, 2025
Copy link
Collaborator

@Gazizonoki Gazizonoki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SDK LGTM

@jepett0 jepett0 dismissed stale reviews from Gazizonoki and pnv1 via b872b0c February 13, 2025 15:20
@jepett0 jepett0 requested review from CyberROFL and pnv1 February 13, 2025 15:21
@jepett0 jepett0 requested a review from Gazizonoki February 13, 2025 15:21
Copy link

github-actions bot commented Feb 13, 2025

2025-02-13 15:23:11 UTC Pre-commit check linux-x86_64-release-asan for a0aa27b has started.
2025-02-13 15:23:45 UTC Artifacts will be uploaded here
2025-02-13 15:27:19 UTC ya make is running...
🟡 2025-02-13 16:54:25 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13932 13862 0 24 9 37

2025-02-13 16:56:25 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-13 17:09:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
113 (only retried tests) 74 0 4 3 32

2025-02-13 17:09:17 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-02-13 17:21:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
71 (only retried tests) 34 0 4 3 30

🟢 2025-02-13 17:21:44 UTC Build successful.
🟡 2025-02-13 17:22:16 UTC ydbd size 3.6 GiB changed* by +1.9 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 54f9570 merge: a0aa27b diff diff %
ydbd size 3 879 350 288 Bytes 3 881 366 560 Bytes +1.9 MiB +0.052%
ydbd stripped size 1 358 503 552 Bytes 1 359 208 928 Bytes +688.8 KiB +0.052%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

Copy link

github-actions bot commented Feb 13, 2025

2025-02-13 15:23:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for a0aa27b has started.
2025-02-13 15:23:55 UTC Artifacts will be uploaded here
2025-02-13 15:27:35 UTC ya make is running...
🟡 2025-02-13 16:41:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
28102 25499 0 3 2464 136

2025-02-13 16:44:37 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-13 17:02:38 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
216 (only retried tests) 88 0 0 1 127

🟢 2025-02-13 17:02:45 UTC Build successful.
🟡 2025-02-13 17:03:43 UTC ydbd size 2.1 GiB changed* by +1.1 MiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 54f9570 merge: a0aa27b diff diff %
ydbd size 2 230 621 472 Bytes 2 231 806 376 Bytes +1.1 MiB +0.053%
ydbd stripped size 473 503 224 Bytes 473 719 096 Bytes +210.8 KiB +0.046%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@jepett0 jepett0 enabled auto-merge (squash) February 13, 2025 17:05
@jepett0 jepett0 merged commit 68fb393 into ydb-platform:main Feb 13, 2025
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

external data sources: describe
4 participants