Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(robot-server): Use a non-nullable indexed enum for protocol_kind #15822

Merged
merged 8 commits into from
Jul 29, 2024

Conversation

SyntaxColoring
Copy link
Contributor

@SyntaxColoring SyntaxColoring commented Jul 29, 2024

Overview

Some tweaks to the protocol.protocol_kind SQL column, to throw in with database changes that we're doing anyway.

Changelog

  • Use an enum (both in the SQL sense and in the Python sense), to stop us from storing bad values, and to make the whole Python side a bit more type-safe. Follows the pattern set in refactor(robot-server): move run time params to their own DB table #15650. Before, we were just using a plain str.
  • Do not allow NULL values. NULL was semantically identical to ProtocolKind.STANDARD and was only stored by the 4->5 migration. Our older migration system could only add nullable=True columns, but that's no longer the case.
  • Add an index so queries like "get all the quick-transfer protocols" or "count how many standard protocols there are" don't have to linearly scan the whole table. The higher-level code doesn't take advantage of this yet because it currently does its filtering in Python, but we should aim to move this kind of thing into SQL.

Database compatibility and merge sequencing

This intends to follow the plan started in #15650, which I understand as this:

The current public release is on schema 4. edge is on schema 5. #15650 abandons schema 5 in favor of 5.1. After #15650 merges, this PR, and a few others from @sanni-t, will merge in quick succession to modify 5.1 in-place. Then, a final PR will turn 5.1 into 6 and create a 5.0->6 migration. The aim of all of this is to make a few improvements on the database from what we have in 5, without introducing a migration for each of those improvements, and without tossing away the schema-5.0 data that's accumulated on all our internal robots.

Test Plan and Hands on Testing

  • Run EXPLAIN QUERY PLAN in the sqlite3 CLI and make sure the index is working properly.
  • Manually set up a dev server with an old database and inspect the SQL to make sure the migration happens properly.

Otherwise, I'm leaning on automated tests.

Review requests

  • Am I understanding the merge sequencing plan correctly?
  • Is the merge sequence stack already too precarious for us to put this on top of it, too?

Risk assessment

This will make the final PR in the plan, which finalizes the 5->6 split, slightly more complicated. It's one more change that we have to remember to handle correctly.

@SyntaxColoring SyntaxColoring requested a review from a team as a code owner July 29, 2024 17:17
@SyntaxColoring SyntaxColoring changed the title refactor(robot-server): Tighten up protocol_kind SQL column refactor(robot-server): Use a non-nullable indexed enum for protocol_kind Jul 29, 2024
parsed_rtp = json.loads(run_time_parameter_values)
else:
parsed_rtp = {}
if not isinstance(protocol_kind, ProtocolKind):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that protocol_kind: Form default is ProtocolKind.STANDARD, do we need to set it here explicitly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's weird. We should not, but apparently we do. See the TODO comments above.

Copy link
Contributor

@vegano1 vegano1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cleans things up, and sets up the DB for faster access.
Thank you for putting this up!

@SyntaxColoring
Copy link
Contributor Author

Discussed with @sanni-t: Will rebase and reimplement atop #15818, instead of merging into edge following #15650.

@SyntaxColoring SyntaxColoring marked this pull request as draft July 29, 2024 18:26
@SyntaxColoring SyntaxColoring force-pushed the protocols_table_fixups branch from 46f8a52 to f4e4207 Compare July 29, 2024 21:32
@SyntaxColoring SyntaxColoring changed the base branch from AUTH-527-move-rtps-to-their-own-table to split-database-into-v5-and-v6 July 29, 2024 21:32
@SyntaxColoring SyntaxColoring marked this pull request as ready for review July 29, 2024 21:32
Copy link
Member

@sanni-t sanni-t left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

else {}
)
if not isinstance(protocol_kind, ProtocolKind):
protocol_kind = ProtocolKind.STANDARD
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think previously if the protocol kind specified in the request was not one of the defined kinds then it would raise an error. Are we intentionally ignoring it now?

Copy link
Contributor Author

@SyntaxColoring SyntaxColoring Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the actual running server, FastAPI will still reject invalid protocolKinds.

        {
            "id": "InvalidRequest",
            "title": "Invalid Request",
            "detail": "value is not a valid enumeration member; permitted: 'standard', 'quick-transfer'",
            "source": {
                "pointer": "/protocolKind"
            },
            "errorCode": "4000"
        }

This is a weird unit-test-only conditional.

I think this can be fixed by adopting FastAPI's newer Annotated[...] syntax instead of the current = Depends(...) syntax, which I'm working on on the side.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right right!

Ya, switching to Annotated would be nice.

@SyntaxColoring SyntaxColoring merged commit 1ab48ab into split-database-into-v5-and-v6 Jul 29, 2024
7 checks passed
@SyntaxColoring SyntaxColoring deleted the protocols_table_fixups branch July 29, 2024 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants