Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Weaviate Resource #87

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

shimon-cherrypick
Copy link

@shimon-cherrypick shimon-cherrypick commented Jan 5, 2025

Summary & Motivation

  • An integration of dagster with Weaviate - supports both locally-hosted and cloud Weaviate databases.

How I Tested These Changes

  • Added unit-tests that create a local weaviate instance, and use the WeaviateLocalResource to query it.
  • Added mock-tests for the cloud version. The mock-tests validate that the connection parameters (auth, heades, cluster_url) are correctly passed to the Weaviate library.
  • Tested connection to weaviate-cloud locally, using a temporary Weaviate project. includes testing authentication with an API_KEY, and adding additional HTTP headers (e.g. API_KEY for Cohere)

Changelog

Ensure that an entry has been created in CHANGELOG.md outlining additions, deletions, and/or modifications.

See: keepachangelog.com

@shimon-cherrypick shimon-cherrypick changed the title Initial Weaviate resource Add Weaviate Resource Jan 5, 2025
@yuhan yuhan requested review from yuhan and dpeng817 January 6, 2025 05:16
Copy link

@dpeng817 dpeng817 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approach for the overall resource pattern looks good. Some nits, requests for additional tests, and comments around usage patterns.

@shimon-cherrypick shimon-cherrypick force-pushed the main branch 3 times, most recently from e9e1e58 to ff9dfc8 Compare January 7, 2025 21:21
## Summary & Motivation

- An integration of dagster with [Weaviate](https://weaviate.io/) - supports both locally-hosted and cloud Weaviate databases.
- Includes WeaviateCloudResource, and WeaviateLocalResource.

## How I Tested These Changes

- Added unit-tests that create a local weaviate instance, and use the WeaviateLocalResource to query it.
- Tested connection to weaviate-cloud locally, using a temporary Weaviate project. includes testing authentication with an API_KEY, and adding additional HTTP headers (e.g. API_KEY for Cohere)

## Changelog

Ensure that an entry has been created in `CHANGELOG.md` outlining additions, deletions, and/or modifications.

See: [keepachangelog.com](https://keepachangelog.com/en/1.0.0/)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants