feature: [VRD-894] New classes for PipelineStep, PipelineGraph, and RegisteredPipeline #4020

ewagner-verta · 2023-08-17T23:05:06Z

Impact and Context

All new functionality. This PR provides the base set of classes needed for working with Pipelines.

Public Docs Rendered

PipelineStep

PipelineGraph

RegisteredPipeline

Risks and Area of Effect

Low risk, as this does not change any existing functionality and does not yet add the Client class tools that will use these classes.

Testing

Unit test
Deployed to dev env
Other (explain)

Reverting

Contains Migration - Do Not Revert

github-actions · 2023-08-17T23:12:29Z

Common Code Coverage

Overall Project	15.14%	🍏

There is no coverage information present for the Files changed

github-actions · 2023-08-17T23:12:31Z

Backend Code Coverage

Overall Project	61.32%	🍏

There is no coverage information present for the Files changed

liuverta

Will check on tests later today

client/verta/tests/conftest.py

client/verta/tests/unit_tests/conftest.py

client/verta/verta/pipeline/_registered_pipeline.py

client/verta/tests/unit_tests/conftest.py

client/verta/tests/unit_tests/pipeline/test_pipeline_graph.py

client/verta/tests/unit_tests/pipeline/test_registered_pipeline.py

…t is actually a factory fixture

Co-authored-by: Liu <[email protected]>

…e.py Co-authored-by: Liu <[email protected]>

…namically in case of changes to fixture

…e.py Co-authored-by: Liu <[email protected]>

… to avoid confusion with pytest fixtures

Co-authored-by: Liu <[email protected]>

…ng steps of a graph

client/verta/verta/pipeline/_pipeline_graph.py

…MV name via UI

Co-authored-by: Liu <[email protected]>

…bles

client/verta/tests/unit_tests/pipeline/test_pipeline_graph.py

client/verta/verta/pipeline/_pipeline_graph.py

client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

client/verta/verta/pipeline/_pipeline_graph.py

client/verta/verta/pipeline/_pipeline_step.py

client/verta/verta/pipeline/_registered_pipeline.py

client/verta/verta/pipeline/_pipeline_graph.py

liuverta · 2023-08-29T17:52:05Z

client/verta/verta/pipeline/_pipeline_step.py

+            registered_model_version
+        )
+        self._predecessors = self.set_predecessors(predecessors)
+        self._registered_model: RegisteredModel = self._get_registered_model()


self._registered_model_version = self.set_registered_model_version( registered_model_version )

already sets self._registered_model. So we're fetching the RM twice during __init__() (which also results in two got existing RegisteredModel print statements) which may be a little janky. But we can address this later.

Fixing this won't be a huge deal, but may be best left to another ticket

Co-authored-by: Liu <[email protected]>

…e.py Co-authored-by: Liu <[email protected]>

Co-authored-by: Liu <[email protected]>

…ull scheme and socket dynamically

client/verta/verta/pipeline/_pipeline_graph.py

Co-authored-by: Liu <[email protected]>

feature: New classes for handling pipelines

0f4dbc9

liuverta suggested changes Aug 19, 2023

View reviewed changes

liuverta suggested changes Aug 20, 2023

View reviewed changes

ewagner-verta and others added 25 commits August 22, 2023 10:43

Merge remote-tracking branch 'origin/main' into VRD-894_pipelines_CRUD

87ab35c

fix: drop change to pytest fixture

ce4e3f3

test: fix up unit tests for changes to mock rmv fixture

b1bd32d

test: update simple pipeline fixture and doc string to reflect that i…

d54fc0a

…t is actually a factory fixture

test: guarantee uniqueness in mock rmv fixture ids

2d01a70

Update client/verta/tests/unit_tests/conftest.py

4a237cc

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/conftest.py

b7a07c6

Co-authored-by: Liu <[email protected]>

test: use static string for name in mock pipe step

2e24387

Update client/verta/tests/unit_tests/conftest.py

b8e70ff

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_graph.py

c6bb7e1

Co-authored-by: Liu <[email protected]>

test: change naming of function scoped variable

ed63443

Update client/verta/tests/unit_tests/pipeline/test_registered_pipelin…

6b6443f

…e.py Co-authored-by: Liu <[email protected]>

test: update mocked responses to pull connection scheme and socket dy…

4828aab

…namically in case of changes to fixture

test: fix incorrect comment line

b16ec5f

Update client/verta/tests/unit_tests/pipeline/test_registered_pipelin…

1a1f826

…e.py Co-authored-by: Liu <[email protected]>

test: drop unused strategies and remove 'mock' from naming convention…

692c369

… to avoid confusion with pytest fixtures

test: refactor pipeline definition strategy to be linear pipeline

63d0f6c

Update client/verta/verta/pipeline/_pipeline_graph.py

362c6b0

Co-authored-by: Liu <[email protected]>

fix: make pipeline_step set_steps function return the steps

a4389de

fix: update repr formatting for PipelineGraph

37f9bbc

Update client/verta/verta/pipeline/_pipeline_graph.py

1c18e23

Co-authored-by: Liu <[email protected]>

Update client/verta/verta/pipeline/_pipeline_step.py

947b994

Co-authored-by: Liu <[email protected]>

fix: remove 'optional' tag in doc string for required variable

62ea6bb

fix: use more sepcific error messaging for type violations when setti…

97e940e

…ng steps of a graph

docs: update doc string with missing params

eb44a21

liuverta suggested changes Aug 29, 2023

View reviewed changes

client/verta/verta/pipeline/_pipeline_graph.py Outdated Show resolved Hide resolved

client/verta/verta/pipeline/_pipeline_graph.py Outdated Show resolved Hide resolved

ewagner-verta and others added 5 commits August 29, 2023 18:03

refactor: make RegisteredPipeline properly responsive to changes in R…

d9d98f9

…MV name via UI

Merge remote-tracking branch 'origin/main' into VRD-894_pipelines_CRUD

79576ff

Update client/verta/verta/pipeline/_pipeline_graph.py

86b1032

Co-authored-by: Liu <[email protected]>

refactor: move type casting from validation function to setter function

93df142

fix: type annotation for PipelineStep init to accept a union of itera…

0c04859

…bles

liuverta suggested changes Aug 29, 2023

View reviewed changes

ewagner-verta and others added 15 commits August 29, 2023 20:44

Update client/verta/tests/unit_tests/pipeline/test_pipeline_graph.py

1fb8e1e

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

e58cb00

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

33e5511

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

304bef1

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

3db95af

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

073a058

Co-authored-by: Liu <[email protected]>

Update client/verta/tests/unit_tests/pipeline/test_pipeline_step.py

4eade59

Co-authored-by: Liu <[email protected]>

docs: simplify doc strings

0085a8d

Update client/verta/tests/unit_tests/pipeline/test_registered_pipelin…

38b1add

…e.py Co-authored-by: Liu <[email protected]>

Update client/verta/verta/pipeline/_pipeline_step.py

695ff3a

Co-authored-by: Liu <[email protected]>

refactor: drop newline from repr

530fc31

refactor: move validate predecessors call to validate steps function

3b1807f

feature: check for uniqueness of step names

c2a367e

refactor: black formatting

0712c0f

fix: use len comparison to avoid empty list != empty set issue, and p…

0266f9f

…ull scheme and socket dynamically

ewagner-verta requested a review from liuverta August 29, 2023 21:41

ewagner-verta added 2 commits August 30, 2023 18:47

Merge remote-tracking branch 'origin/main' into VRD-894_pipelines_CRUD

95b1b6a

fix: circular import problem

95bbca9

liuverta suggested changes Aug 30, 2023

View reviewed changes

client/verta/verta/pipeline/_pipeline_graph.py Outdated Show resolved Hide resolved

Update client/verta/verta/pipeline/_pipeline_graph.py

fe3c11d

Co-authored-by: Liu <[email protected]>

liuverta approved these changes Aug 30, 2023

View reviewed changes

ewagner-verta merged commit ec9ac77 into main Aug 30, 2023
1 check passed

ewagner-verta deleted the VRD-894_pipelines_CRUD branch August 30, 2023 18:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: [VRD-894] New classes for PipelineStep, PipelineGraph, and RegisteredPipeline #4020

feature: [VRD-894] New classes for PipelineStep, PipelineGraph, and RegisteredPipeline #4020

ewagner-verta commented Aug 17, 2023 •

edited

Loading

github-actions bot commented Aug 17, 2023

github-actions bot commented Aug 17, 2023 •

edited

Loading

liuverta left a comment

liuverta Aug 29, 2023

ewagner-verta Aug 29, 2023

feature: [VRD-894] New classes for PipelineStep, PipelineGraph, and RegisteredPipeline #4020

feature: [VRD-894] New classes for PipelineStep, PipelineGraph, and RegisteredPipeline #4020

Conversation

ewagner-verta commented Aug 17, 2023 • edited Loading

Impact and Context

Public Docs Rendered

PipelineStep

PipelineGraph

RegisteredPipeline

Risks and Area of Effect

Testing

Reverting

github-actions bot commented Aug 17, 2023

Common Code Coverage

github-actions bot commented Aug 17, 2023 • edited Loading

Backend Code Coverage

liuverta left a comment

Choose a reason for hiding this comment

liuverta Aug 29, 2023

Choose a reason for hiding this comment

ewagner-verta Aug 29, 2023

Choose a reason for hiding this comment

ewagner-verta commented Aug 17, 2023 •

edited

Loading

github-actions bot commented Aug 17, 2023 •

edited

Loading