Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix CI due to lazy-loading #1481

Merged
merged 10 commits into from
Aug 14, 2023
Merged

Fix CI due to lazy-loading #1481

merged 10 commits into from
Aug 14, 2023

Conversation

noklam
Copy link
Contributor

@noklam noklam commented Aug 11, 2023

Description

try:
from kedro_datasets import ( # isort:skip
json,
matplotlib,
plotly,
tracking,
)
except ImportError:
from kedro.extras.datasets import ( # Safe since ImportErrors are suppressed within kedro.
json,
matplotlib,
plotly,
tracking,
)
from kedro.io import DataCatalog
from kedro.io.core import get_filepath_str
from kedro.pipeline import Pipeline
from semver import VersionInfo

This is no longer valid because kedro-datasets is lazily load and ImportError is not produce immediately.

The original code is slightly confusing, because ImportError can happens in 2 situations

  • When a specific dependency is missing
  • When kedro-datasets is not installed (This is what the original code intend to catch and have nothing to do with whether dependencies (i.e. pandas, matplotlib) is installed or not.

This is the comments in the code base

# The dataset type is available as an attribute if and only if the import from kedro
# did not suppress an ImportError. i.e. hasattr(matplotlib, "MatplotlibWriter") is True
# when matplotlib dependencies are installed.

This is no longer True. Previously, if the dependencies is not installed, MatplotlibWriter won't be available due to the suppression of ImportError.

Development notes

  • Catch ImportError for lazy-evaluation (kedro_datasets), catch AttributeError for eager evaluation (kedro.extras)

QA notes

Checklist

  • Read the contributing guidelines
  • Opened this PR as a 'Draft Pull Request' if it is work-in-progress
  • Updated the documentation to reflect the code changes
  • Added new entries to the RELEASE.md file
  • Added tests to cover my changes

@noklam
Copy link
Contributor Author

noklam commented Aug 14, 2023

Attempt to fix it in kedro-org/kedro-plugins#298

@noklam noklam marked this pull request as ready for review August 14, 2023 16:09
@tynandebold tynandebold requested review from deepyaman and ravi-kumar-pilla and removed request for yetudada August 14, 2023 16:12
Copy link
Member

@tynandebold tynandebold left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. All checks are passing. Thanks so much.

RELEASE.md Outdated Show resolved Hide resolved
@noklam
Copy link
Contributor Author

noklam commented Aug 14, 2023

@tynandebold @ravi-kumar-pilla The test start failing after I merge from main, would you have a look at this? Feel free to merge it whenever it passed.

@ravi-kumar-pilla ravi-kumar-pilla merged commit cbdfbc9 into main Aug 14, 2023
1 check passed
@ravi-kumar-pilla ravi-kumar-pilla deleted the fix/lazy-loading branch August 14, 2023 23:51
@tynandebold tynandebold mentioned this pull request Aug 17, 2023
5 tasks
@noklam noklam changed the title Attempt to fix CI due to lazy-loading Fix CI due to lazy-loading Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lazy loading datasets causes Kedro-Viz builds to fail
3 participants