-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add support for pipeline deserialization callbacks #7518
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
topic:tests
topic:core
type:documentation
Improvements on the docs
labels
Apr 9, 2024
…epset-ai/haystack into feature/pipeline-deserialization-hook
shadeMe
requested review from
dfokina and
vblagoje
and removed request for
a team and
vblagoje
April 9, 2024 16:15
Pull Request Test Coverage Report for Build 8633397767Warning: This coverage report may be inaccurate.This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.
Details
💛 - Coveralls |
Co-authored-by: Silvano Cerza <[email protected]>
silvanocerza
approved these changes
Apr 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
2.x
Related to Haystack v2.0
topic:core
topic:pipeline
topic:tests
type:documentation
Improvements on the docs
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation:
The motivation behind this change is to allow for external code to introspective modification of a pipeline's underlying components without having to worry about the serialization details. A concrete usecase enabled by this feature is a way to override the init parameters of individual components at deserialization time, creating different variations of the same pipeline (to be leveraged by the upcoming evaluation harness scaffolding for pipelines).
Changes:
This PR modifies the following methods in the
Pipeline
class in a non-breaking manner:load
loads
from_dict
All of the above methods now have an additional, optional parameter that accepts a
DeserializationCallbacks
object, which contains callbacks that get invoked during the various stages of the deserialization process. We use a wrapper/container here for forward-compatibility (should we choose to add more callbacks in the future).Currently, we support just one callback, namely the one the is invoked during the component pre-init stage (before the component's
__init__
method/constructor is called). The callback is allowed to inspect the init parameters and modify them inplace. The modified parameters are then passed to the constructor after the callback.Since each component can have its own custom serialization logic, generic approaches to handle this at the pipeline-level will not suffice - Any modifications to the init parameters will require the overridden values to be in the same serialized format expected by the component's serialization logic. Since we do not have access to this logic, we need to place the hook after the init parameters have been deserialized, i.e., just before the invocation of the component's constructor. To that end, we implement an internal context manager in the
ComponentMeta
metaclass that tests for the presence of such a hook/callback and detours the constructor.How did you test it?
Unit tests.
Checklist
fix:
,feat:
,build:
,chore:
,ci:
,docs:
,style:
,refactor:
,perf:
,test:
.