chore: Migrate Python projects from Pydantic v1 to v2 #14871

ahiuchingau · 2024-04-11T16:57:30Z

Overview

Let's try this again

Closes PLAT-326.

Test Plan

Changelog

Update our robot software (and everything that depends on it) from Pydantic v1 to Pydantic v2:

api
robot-server
shared-data
g-code-testing
hardware
server-utils
system-server
hardware-testing (re-lock only, depends on api)
abr-testing (re-lock only, depends on api)
performance-metrics (re-lock only, depends on shared-data)

buildroot changes in [todo]

Review requests

Risk assessment

SyntaxColoring

I'm about 60% through the changes so far. Mostly looks great, only a few comments and questions:

SyntaxColoring · 2024-04-11T17:06:50Z

api/src/opentrons/calibration_storage/ot2/models/v1.py

+DatetimeType = typing.Annotated[
+    datetime,
+    PlainSerializer(lambda x: x.isoformat(), when_used="json"),
+]
+


I'm inferring that this is replacing the old json_encoders config. Does something need to replace the old json_decoders config, to match?

SyntaxColoring · 2024-04-11T17:09:12Z

api/src/opentrons/calibration_storage/ot3/models/v1.py

+DatetimeType = typing.Annotated[
+    datetime,
+    PlainSerializer(lambda x: x.isoformat(), when_used="json"),
+]


api/src/opentrons/protocol_engine/clients/sync_client.py

SyntaxColoring · 2024-04-11T17:20:06Z

api/src/opentrons/protocol_engine/commands/command_unions.py

+# Each time a TypeAdapter is instantiated, it will construct a new validator and
+# serializer. To improve performance, TypeAdapters are instantiated once.
+# See https://docs.pydantic.dev/latest/concepts/performance/#typeadapter-instantiated-once
+CommandCreateAdatper: TypeAdapter[CommandCreate] = TypeAdapter(CommandCreate)  # type: ignore[arg-type]


Typo: Adatper instead of Adapter

What is this TypeAdapter doing?

SyntaxColoring · 2024-04-11T17:31:16Z

api/src/opentrons/protocol_engine/state/commands.py

@@ -230,7 +230,7 @@ def handle_action(self, action: Action) -> None:  # noqa: C901
            # request > command mapping, figure out how to type precisely
            # (or wait for a future mypy version that can figure it out).
            # For now, unit tests cover mapping every request type
-            queued_command = action.request._CommandCls.construct(


Can this be .model_construct, for performance reasons?

SyntaxColoring · 2024-04-11T17:38:44Z

shared-data/python/opentrons_shared_data/types.py

+class Vec3f(BaseModel, Generic[NumberType]):
+    """A 3D Vector."""
+
+    x: NumberType
+    y: NumberType
+    z: NumberType


The name Vec3f comes from "vector of 3 floats." If this is generic across other kinds of NumberType, let's rename it to just Vec3.

Since we're really pushing this deep down into common shared code, I'm worried that someone will come along later and add an = 0 default for something without realizing that it will affect a million things. Let's spell out that this is specifically for vectors where all elements are required, to hopefully prompt people to use a different type if that's not the behavior that they want.

Suggested change

class Vec3f(BaseModel, Generic[NumberType]):

"""A 3D Vector."""

x: NumberType

y: NumberType

z: NumberType

class Vec3f(BaseModel, Generic[NumberType]):

"""A 3D vector where all elements are required."""

x: NumberType

y: NumberType

z: NumberType

SyntaxColoring · 2024-04-11T17:41:18Z

shared-data/python/tests/pipette/test_validate_schema.py

+    assert loaded_model.channels == types.PipetteChannelType.NINETY_SIX_CHANNEL
+
+    model_dict = loaded_model.model_dump()
+    # each field should be the value of the enum class


SyntaxColoring

I'm caught up!

SyntaxColoring · 2024-04-12T15:19:28Z

api/src/opentrons/protocol_runner/legacy_command_mapper.py

Same as the other comment—for performance reasons, can we change the .construct()s to .model_construct()s instead of changing them to fully-validating __init__() calls?

robot-server/robot_server/log.txt

SyntaxColoring · 2024-04-12T15:50:12Z

robot-server/setup.py

-    "fastapi==0.99.1",
+    "fastapi>=0.100.0",
    "python-dotenv==1.0.1",
    "python-multipart==0.0.6",
-    "pydantic==1.10.12",
+    "pydantic>=2.0.0,<3",


Do these need to be pinned to a specific version like they were before?

Please don't pin to exact versions unless strictly necessary. See e.g. #11905

This is robot-server, which is not published on PyPI, so it should not affect what gets pulled in when you do pip install opentrons.

I'm a bit shaky on our packaging mechanisms, but I think this specific setup.py may not even affect what we install on robots. I'm sure we can make this less confusing somehow. But for the sake of not changing too many things in this PR, I think we'll keep the ==.

Conflicts: * api/Pipfile.lock (took edge, will need to re-lock) * api/src/opentrons/calibration_storage/deck_configuration.py * api/src/opentrons/cli/analyze.py * api/src/opentrons/protocol_api/core/engine/protocol.py * api/src/opentrons/protocol_engine/commands/calibration/calibrate_gripper.py * api/src/opentrons/protocol_engine/commands/calibration/calibrate_pipette.py * api/src/opentrons/protocol_engine/commands/command.py * api/src/opentrons/protocol_engine/commands/custom.py * api/src/opentrons/protocol_engine/types.py * api/src/opentrons/protocol_runner/legacy_command_mapper.py * api/tests/opentrons/cli/test_cli.py * api/tests/opentrons/protocol_engine/state/command_fixtures.py * api/tests/opentrons/protocol_engine/state/test_addressable_area_store.py * api/tests/opentrons/protocol_engine/state/test_geometry_view.py * api/tests/opentrons/protocol_runner/test_protocol_runner.py * robot-server/Pipfile * robot-server/Pipfile.lock (took edge, will need to re-lock) * robot-server/robot_server/deck_configuration/defaults.py * robot-server/robot_server/persistence/pydantic.py * shared-data/command/schemas/8.json (took edge, will need to regenerate) * shared-data/python/tests/deck/test_typechecks.py

Merging for updates to robot-server tests.

SyntaxColoring · 2024-08-16T20:49:33Z

An update on where this stands at the time of writing:

The big problem to solve is that there's a pretty severe hit to robot-server startup time. On Flex, it goes from ~46s to ~1m54s, and it's probably worse on OT-2.
- The Pydantic people are aware of the problem, but it doesn't look like there will be an imminent fix.
- defer_build might help, but some quick experiments didn't show it making a huge difference. I wouldn't expect it to shave more than 20 seconds, and that's being optimistic.
- There is a large FastAPI component to the problem, having to do with the way it nests routers within each other. We might be able to work around it by flattening our routers. I don't know exactly what that would look like yet. A proof of concept would be a good next step.
There are still various tests that are failing for one reason or another. So far, I haven't seen anything that fundamentally concerns me. Just a lot of little things to work through.
edge is a moving target and there are always going to be more merge conflicts. One thing that would help is ignoring the trivial rename DeprecationWarnings for now—maybe even remove the fixes that we have so far from this PR—and fix them in incremental follow-ups.

sfoster1 · 2024-09-03T18:11:30Z

Per pydantic/pydantic#6748 (comment) now may be the time to evaluate the latest pydantic2 release

sadly this includes getting rid of some nice typing that won't work until we have future peps

This carries a lot of speedups relative to 2.6; maybe it's even usable now.

Probably needed for pydantic updates

codecov · 2024-09-06T14:45:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 60.23%. Comparing base (094be6e) to head (cdbf235).
Report is 19 commits behind head on edge.

Additional details and impacted files

@@             Coverage Diff             @@
##             edge   #14871       +/-   ##
===========================================
- Coverage   92.43%   60.23%   -32.21%     
===========================================
  Files          77      191      +114     
  Lines        1283    10667     +9384     
===========================================
+ Hits         1186     6425     +5239     
- Misses         97     4242     +4145

Flag	Coverage Δ
hardware	`55.82% <100.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...are/opentrons_hardware/drivers/can_bus/settings.py	`95.77% <100.00%> (ø)`

... and 113 files with indirect coverage changes

strangemonad · 2024-10-24T15:38:42Z

please please please prioritize this. Pydantic should have fixed many import perf issues. It's a major pain that we can't easily interoperate between core models in parts of our codebase.

ahiuchingau added 14 commits April 11, 2024 13:01

update pydantic & fix all mypy errors

c2d17fb

hardware: update pydantic & fix all mypy errors

cae6028

hardware: fix Pipfile

8779c87

api: update pydantic & fix mypy errors

19f7110

shared-data: add Vec3f

b5c8eab

shared-data: rename labware_definition to models

ac77352

fix api mypy test errors

ef889d4

update api

b3ea918

fix protocol_engine tests

ea9e17e

fix(api): tests

90b7fab

update command schema 8.json

16b69eb

shared-data: deprecate json_encoder and use v2 serializer

528a825

fix api tests

ff83a57

api(src): update deprecated code

80a7000

ahiuchingau force-pushed the chore_update-pydantic-v2 branch from e83d1b5 to 80a7000 Compare April 11, 2024 17:03

ahiuchingau added 2 commits April 11, 2024 13:12

api: linter

1291aa1

shared-data: remove deprecated code

27bbbec

SyntaxColoring reviewed Apr 11, 2024

View reviewed changes

ahiuchingau added 6 commits April 11, 2024 13:51

shared-data: mypy.ini removed commented out override

b646339

robot-server & server-utils: update pipenv

7588a3f

robot-server: add pydantic_settings

0b391da

robot-server: bump-pydantic to v2

8114936

robot-server: fix lint errors

52b5899

robot-server: fix one test

9b02831

SyntaxColoring reviewed Apr 12, 2024

View reviewed changes

make lint

2728cca

SyntaxColoring mentioned this pull request Apr 15, 2024

feat: Upgrade pydantic to v2 #13983

Open

ahiuchingau force-pushed the chore_update-pydantic-v2 branch from 949e25a to 2728cca Compare April 17, 2024 18:50

ahiuchingau and others added 2 commits April 19, 2024 13:09

add model serializer

c47cb10

SyntaxColoring mentioned this pull request May 28, 2024

chore(pydantic): V1->V2 migration #14805

Closed

Merge remote-tracking branch 'origin/edge' into chore_update-pydantic-v2

ff3db4e

Merging for updates to robot-server tests.

SyntaxColoring closed this May 28, 2024

SyntaxColoring deleted the chore_update-pydantic-v2 branch May 28, 2024 17:14

SyntaxColoring reopened this May 28, 2024

SyntaxColoring added 4 commits June 7, 2024 10:46

g-code-testing: Run bump-pydantic [WIP]

14f5c10

WIP: Fix some slow stuff with TypeAdapter?

01f0ed9

robot-server: Update error messages in tests.

de2b400

Revert BaseModel -> BaseResponseBody changes.

d674c66

sfoster1 added 8 commits September 3, 2024 17:33

Merge branch 'edge' into chore_update-pydantic-v2

1d95e40

chore: format and fix lints

18a3b22

sadly this includes getting rid of some nice typing that won't work until we have future peps

s-d: fix tests

9417cd0

api: fix testst

315a370

schema format

fea1fad

make this installable per pypa/setuptools#4483

7fecd09

Merge branch 'edge' into chore_update-pydantic-v2

b0c849f

chore: pydantic 2.9.0

8082363

This carries a lot of speedups relative to 2.6; maybe it's even usable now.

sfoster1 force-pushed the chore_update-pydantic-v2 branch from 57305be to 8082363 Compare September 5, 2024 18:39

sfoster1 added 2 commits September 5, 2024 14:41

chore: mypy 1.11.0

b3f7e04

Probably needed for pydantic updates

chore: py lint fixes

5a781f1

sfoster1 force-pushed the chore_update-pydantic-v2 branch from 03a63b4 to 5a781f1 Compare September 5, 2024 20:32

sfoster1 added 2 commits September 5, 2024 17:36

chore: fix hardware-testing

bab482b

chore: fix api runtime type dependencies

aaf8f58

sfoster1 added 2 commits September 6, 2024 10:49

fixup command schema

9159f4a

fixup some server tests

666b3ff

Merge branch 'edge' into chore_update-pydantic-v2

cdbf235

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: Migrate Python projects from Pydantic v1 to v2 #14871

chore: Migrate Python projects from Pydantic v1 to v2 #14871

ahiuchingau commented Apr 11, 2024 •

edited by SyntaxColoring

Loading

SyntaxColoring left a comment

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring Apr 11, 2024

SyntaxColoring left a comment

SyntaxColoring Apr 12, 2024

SyntaxColoring Apr 12, 2024

ecederstrand May 24, 2024 •

edited

Loading

SyntaxColoring May 24, 2024 •

edited

Loading

SyntaxColoring commented Aug 16, 2024

sfoster1 commented Sep 3, 2024

codecov bot commented Sep 6, 2024 •

edited

Loading

strangemonad commented Oct 24, 2024

chore: Migrate Python projects from Pydantic v1 to v2 #14871

Are you sure you want to change the base?

chore: Migrate Python projects from Pydantic v1 to v2 #14871

Conversation

ahiuchingau commented Apr 11, 2024 • edited by SyntaxColoring Loading

Overview

Test Plan

Changelog

Review requests

Risk assessment

SyntaxColoring left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SyntaxColoring left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ecederstrand May 24, 2024 • edited Loading

Choose a reason for hiding this comment

SyntaxColoring May 24, 2024 • edited Loading

Choose a reason for hiding this comment

SyntaxColoring commented Aug 16, 2024

sfoster1 commented Sep 3, 2024

codecov bot commented Sep 6, 2024 • edited Loading

Codecov Report

strangemonad commented Oct 24, 2024

ahiuchingau commented Apr 11, 2024 •

edited by SyntaxColoring

Loading

ecederstrand May 24, 2024 •

edited

Loading

SyntaxColoring May 24, 2024 •

edited

Loading

codecov bot commented Sep 6, 2024 •

edited

Loading