Skip to content

Commit

Permalink
Collections v3 client (#102)
Browse files Browse the repository at this point in the history
* 3.0.0-beta.7

* Allow hnsw/flat & pq/bq combos, improve coverage, fix named vectors properties parsing

* Add support for proxying using fetch API and grpc.js options

* Support forward proxies for http and tunnel proxies for grpc. Add deadline for grpc client

* Remove debug logging from proxies

* Skip flaky proxy test in CI

* Fix strong typing of generics-based collection creation

* 3.0.0-beta.8

* Add all missing vectorizers, revert back to overloads of query/groupby types, make generate required param of generate queries

* Replace WeaviateNextClient with WeaviateClient

* 3.0.0-beta.9

* Fix parsing of vectorizePropertyName for named vectors, fix strange typing behaviour with named vectors

* 3.0.0-beta.10

* Add new modules, add unit tests for vectorizer, only have namedVectorizer factory

* 3.0.0-beta.11

* Export submodules from collections/index

* 3.0.0-beta.12

* Add default generics to Collection type

* 3.0.0-beta.13

* Minor fixes to typing names

* 3.0.0-beta.14

* Add missing offset arg to all queries

* 3.0.0-beta.15

* Add missing misc getters to client namespace

* 3.0.0-beta.16

* Add tokenization option to weaviate.configure

* Handle non-generic collections more comprehensively using undefined type

* Make package releasable to a wider number of environments

- Do away with importing from weaviate-client/node, didn't work with type hinting .js code
- Now all imports are at the root of weaviate-client
- Make the package a true ESM module with module: NodeNext
- Requires all imports change to .js and /index.js
- Require esModuleInterop to serve cjs and esm due to dependency on protobufjs
- Add beginnings of typedoc build to make HTML docs from docstrings

* Fix tests with new packaging format

* Fix odd TS false positive in VS code

* Fix missed imports from previous commit

* Fix missed imports from previous commit

* Fix .data inputs for non-generic collections

* Add missing rerank functionalty to all queries

* Add vectorDistances object to configure

* Skip rerank tests if no OpenAI API key present

* Remove cp src command from build

* 3.0.0-beta.17

* Refactor directory structure and add docs

- Change modules that had everything in index.ts to only export user facing objects
- Add docstrings to all public methods and namespaces
- Make number of small bugfixes and API improvements ready for wider release

* 3.0.0-beta.18

* Add missing `config.update` method and `weaviate.reconfigure` object factory

* 3.0.0-beta.19

* Hotfix bm25 and hybrid search param parsing

* Fix search property names in unit tests

* 3.0.0-beta.20

* Add missing uuid types to dataType in config

* Add explicit graphql dep due to peer dep in graphql-request

* 3.0.0-beta.21

* Fix typing for non-generic group by queries

* 3.0.0-beta.22

* Automatically capitlize user-supplied collections names

* Add support for specifying BackupConfig when creating and restoring backups

* Fix backup tests colliding with data in other tests

* Add support for groupby bm25 and hybrid queries

* Add DbVersionSupport checks to collections API

* Respond to user feedback

- Add collection.exists method as shortcut for collections.exists
- Provide client.close() method to allow cleaning of resource
- Fix aggregate typing bug

* Update CI to test Node 18 and Node 20

* 3.0.0-beta.23

* Introduce `.use()` instead of `.get()` and deprecate `.get()` with a warning

* Allow returnMetadata: 'all' to get all metadata at once

* Split legacy and named vectorizers, deprecate legacy

* Fix updating vector index config of legacy vector collection

* Make single named vectorizer name generic rather than literal `'default'`

* Refactoring of named vectors workflow

- Replace configure.namedVectorizer() with configure.namedVectorizer methods for each vectorizer type
- Improve typing system around named vectors both internally and externally

* Rename namedVectorizers to vectorizers everywhere

* Fix errors in integration test

* Refactor to use new ListValue proto message with 1.25

* Improve typings for docs generation

* Unignore docs dir to be used on GH pages

* Update docs generation to include favicon

* Update CI image

* Update CI image

* Update CI to latest main image

* Update CI image to latest main

* Update CI image to latest main

* Update CI to test all supported Weaviate and Node vers

* Add Node 22, rearrange to list through Weaviate vers

* Remove tests for 1.23.x since only named vectors supported in TSv3

* Fix typing of output in cluster.nodes()

* Update to TS v5

* Wait for reranker stack

* Update CI to latest image

* Use latest image with restore fixes, refresh schema with `main` not `master`

* Revert CI to latest stable

* Replace vectorizeClassName with vectorizeCollectionName everywhere

* Fix erroneous str replacements

* Update to latest image

* Update cluster stack to use RAFT env vars

* Update CI to latest main image

* Handle differing API surfaces between versions

* Fix errors in integration test

* Revert changes to `.get()` and `.use()`

* Refactor usage of `.use()` to `.get()` in integration tests

* Add reranker-voyageai to configure

* Implement specific errors for certain failure modes

* Remove `skipVectorization` from `PropertyConfigCreate`

* Update publish step to tag as next

* 3.0.0-rc.0

* Align connection helper methods with Python versions

* Undo release to rc

* Run `npm audit fix`

* Update `README.md` to point to GH pages typedocs

* Export helper type options

* Fix port number in integration test

* Test Node22 support in CI

* 3.0.0-rc.0

* Reinstate `skipVectorization` field

* 3.0.0-rc.1

* Undo rc.1 release

* 3.0.0-rc.1

* Update CI images to latest

* Add `tenants.getByName` using new grpc method

* Update query integration tests to use correct vector property name

* Update CI image and version checks

* Fix regex matching for prerelease versions

* Fix port number for Weaviate connection in backup integration test

* Fix parsing of listValue for 1.24 and 1.25 rc

* Update CI to latest `stable/v1.25` image

* Fix conflicting test prots

* Fix vector in test

* Fix strangely new TS errors

* Fix backup logic when loop waiting for finish

* Add remaining methods and tests, improve naming conventions

* Add checks for DB version support of new TenantsGet grpc method

* Remove redundant version support check in bm25/hybrid

* Add BC support for hybrid near subsearches

* Fix missing fields in Multi2VecPalm

* Fix inputs to generative methods and add unit tests for them

* Make projectId in Text2VecPalm optional

* Add missing generative modules and unit tests

* Make a number of fixesand improvements to the connection logic
- Allow string api keys
- Fix unhandled internal promise
- Add error for WeaviateStartUp
- Refactor connection journery tests to use v3 client

* Clean host params of scheme prefixes and emit warnings when detected

* Align `ClientParams` with Python client

* Add support for dynamic index and remove defaults from configure methods

* Handle dynamic index with BC gracefully in client

* Permit supplying undefined vectorizer to use server default

* 3.0.0-rc.2

* Fix multi2vec types and configure options

* Bump CI version to latest

* Fix missing `IsNull` case in `Serialize.operator`

* Fix errors introduced by misaligned merge from main

* Fix tests with old weaviate and incompatible features

* 3.0.0-rc.3

* Add concrete types for GenerativeObject conditional

* Fix parsing of reranker config in collections.create, improve config returns

* Fix test of generative config return

* Add test for filtering on null date

* Make a number of improvements and fixes:
- Fix bug with near object search
- Allow making making named vector without its name, thereby becoming default
- Invert conditional types to help with strict: false users
- Rename WCS to WCD

* Export the v2 client from index.ts to help ease migration

* Rename <>ObjectType to <>GenericObject

* Remove commented out code

* Remove commented out code

* Add docstrings for types

* Undo change to conditional type avoiding circ dep

* Skip tests of broken server behaviour to make release

* Flip conditional type for PropertiesMetrics due to strict: false users

* Flip conditional type for strict: false users

* Rename `vectorizer` -> `vectorizers` in `CollectionConfig`

* Fix broken commit

* Rename vectorizer -> vectorizers in tests

* Fix bad CTRL+F replace

* Add missing change from commit

* Add test for making a non-generic collection with no properties

* Flip conditional type for strict: false users

* Make more improvements:
- Remove legacy vectorisation process
- Refactor config.update to use new vectorization logic
- Add timeouts to requests

* Fix passing timeout to grpc and handling clear

* Add skipInitChecks and handle close channel on init error

* 3.0.0-rc.4

* Reintroduce BC/FC for 1.23.7 upwards, add back deprecated connectToWCS, add weaviate.configGuards

* Fix parsing of vectors in insertMany, add more types to typedoc, fix accidental autoschema in tests

* 3.0.0-rc.5

* Add missing test of 1.23 to CI

* Undo adding 1.23 tests, too many breaks with v2 tests

* Export important sub namespaces of configure

* Update docs

* Rename to `connectToWeaviateCloud` add back removed but deprecated methods

* Add deprecation warnings to connect methods

* Update docs

* Update 1.24 CI image

* 3.0.0-rc.6

* Update CI image for 1.25

* Remove constraint that properties must be present if generic is provided/inferred

* Add missing text2VecOllama staticmethod to configure

* Update schema to 1.25.2 refactoring tests to handle new field

* Return promise from beforeAll to make it block

* Make `collectionName` unique in WCS journey to avoid flake collisions

* Fix assertion

* Export all objects of configure as part of module

* Change CI to publish GA tag on next release push

* Add missing exported types to docs, format code with prettier organised imports

* Fix bad import

* 3.0.0

* Add missing autoTenantActivation to multiTenancyConfig

* 3.0.1

* Fix type of `HybridNearTextSubSearch` to be same as python client

* 3.0.2

* Rename `vectorName` to `name` in `VectorConfigUpdate`

* 3.0.3

* Test adding step to build docs in CI

* Add missing vers to uses:

* Call as `docker compose` rather than `docker-compose` in CI

* Rearrange order to fail fast while debugging

* Add missing dep install step to docs build

* Remove npm run format from docs script, use relative path

* Move docs deployment to publishing job

* Delete static /docs from the src code

* Add `docs/` to `.gitignore` file

* 3.0.4

* Add missing permissions to publish step for GH pages

* 3.0.4

* Fix order of build steps in publish job

* 3.0.4

* Add permission to upload release to GH

* 3.0.4

* 3.0.5

* Fix CUD ops inserting vectors without names in >=1.24

* 3.0.6

* Fix bug when aggregating with multi-tenancy

* 3.0.7

* Fix parsing of files and buffers in nearMedia methods

* 3.0.8

* Ensure message is intialised in the base WeaviateError class

* Update ver in CI

* Remove commented code, simplify expects
  • Loading branch information
tsmith023 authored Jun 27, 2024
1 parent 89b0a36 commit 565fb7f
Show file tree
Hide file tree
Showing 216 changed files with 33,648 additions and 6,495 deletions.
1 change: 1 addition & 0 deletions .eslintignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
node_modules
dist
types
docs
2 changes: 1 addition & 1 deletion .eslintrc.js → .eslintrc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ module.exports = {
'no-native-reassign': 'error',
'no-negated-condition': 'off',
'no-negated-in-lhs': 'error',
'no-nested-ternary': 'error',
'no-nested-ternary': 'off',
'no-new': 'error',
'no-new-func': 'error',
'no-new-object': 'error',
Expand Down
44 changes: 34 additions & 10 deletions .github/workflows/main.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,23 +6,32 @@ on:
- '**'
pull_request:


env:
WEAVIATE_VERSION: 1.25.1
WEAVIATE_124: 1.24.19
WEAVIATE_125: 1.25.5

jobs:
checks:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
node: [
"18.x",
"20.x",
"22.x"
]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
node-version: '18.x'
node-version: ${{ matrix.node }}
- name: "Run checks"
run: |
npm install
npm ci
npm run lint
npm run format:check
npm run docs
tests:
needs: checks
Expand All @@ -31,9 +40,12 @@ jobs:
fail-fast: false
matrix:
versions: [
{ node: "18.x", weaviate: $WEAVIATE_VERSION},
{ node: "20.x", weaviate: $WEAVIATE_VERSION},
{ node: "22.x", weaviate: $WEAVIATE_VERSION}
{ node: "18.x", weaviate: $WEAVIATE_124},
{ node: "20.x", weaviate: $WEAVIATE_124},
{ node: "22.x", weaviate: $WEAVIATE_124},
{ node: "18.x", weaviate: $WEAVIATE_125},
{ node: "20.x", weaviate: $WEAVIATE_125},
{ node: "22.x", weaviate: $WEAVIATE_125}
]
steps:
- uses: actions/checkout@v3
Expand All @@ -42,7 +54,7 @@ jobs:
node-version: ${{ matrix.versions.node }}
- name: "Install dependencies"
run: |
npm install
npm ci
ci/run_dependencies.sh ${{ matrix.versions.weaviate }}
- name: "Run tests with authentication tests"
if: ${{ !github.event.pull_request.head.repo.fork }}
Expand All @@ -66,19 +78,31 @@ jobs:
needs: tests
if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags')
runs-on: ubuntu-latest
permissions:
contents: write # to upload the release
pages: write # to deploy to Pages
id-token: write # to verify the deployment originates from an appropriate source
steps:
- uses: actions/checkout@v3
# Setup .npmrc file to publish to npm
- uses: actions/setup-node@v3
with:
node-version: '18.x'
registry-url: 'https://registry.npmjs.org'
- run: npm ci && npm run build
- run: npm ci
- run: npm run build
- run: npm publish
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_AUTOMATION_TOKEN }}
- run: npm run docs
- name: "Upload docs as pages artifact"
uses: actions/upload-pages-artifact@v3
with:
path: ./docs
- name: "Deploy the uploaded pages artifact"
uses: actions/deploy-pages@v4
- name: "Create a GitHub release"
uses: softprops/action-gh-release@v1
with:
generate_release_notes: true
draft: true
draft: true
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,6 @@ weaviate-data/
*.tgz
.npmrc
.eslintcache
test.sh
test.sh
scratch/
docs/
3 changes: 3 additions & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
src/proto/**/*.ts
src/dist/**/*
src/docs/**/*.js
3 changes: 2 additions & 1 deletion .prettierrc
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,6 @@
"trailingComma": "es5",
"bracketSpacing": true,
"singleQuote": true,
"printWidth": 110
"printWidth": 110,
"tabWidth": 2
}
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
# Weaviate TypeScript client <img alt='Weaviate logo' src='https://weaviate.io/img/site/weaviate-logo-light.png' width='148' align='right' />
# Weaviate JS/TS client <img alt='Weaviate logo' src='https://weaviate.io/img/site/weaviate-logo-light.png' width='148' align='right' />

Official TypeScript client for easy interaction with a Weaviate instance.
Official JS/TS client for easy interaction with a Weaviate instance.

## Documentation

- [Documentation](https://weaviate.io/developers/weaviate/client-libraries/typescript).
- [General Documentation](https://weaviate.io/developers/weaviate/client-libraries/typescript).
- [Client-specific Documentation](https://weaviate.github.io/typescript-client/)

## Support

Expand Down
4 changes: 2 additions & 2 deletions ci/compose.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ function ls_compose {

function exec_all {
for file in $(ls_compose); do
docker-compose -f $(echo "ci/${file} ${1}")
docker compose -f $(echo "ci/${file} ${1}")
done
}

Expand All @@ -21,5 +21,5 @@ function compose_down_all {
}

function all_weaviate_ports {
echo "8080 8081 8082 8083 8085 8086 8087 8088"
echo "8078 8079 8080 8081 8082 8083 8085 8086 8087 8088 8089 8090"
}
23 changes: 23 additions & 0 deletions ci/docker-compose-async.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
version: '3.4'
services:
weaviate_async:
command:
- --host
- 0.0.0.0
- --port
- '8090'
- --scheme
- http
image: semitechnologies/weaviate:${WEAVIATE_VERSION}
ports:
- "8078:8090"
- "50049:50051"
restart: on-failure:0
environment:
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
CLUSTER_HOSTNAME: 'node1'
ASYNC_INDEXING: 'true'
DISABLE_TELEMETRY: 'true'
19 changes: 19 additions & 0 deletions ci/docker-compose-backup.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
version: '3.4'
services:
weaviate-backup:
image: semitechnologies/weaviate:${WEAVIATE_VERSION}
restart: on-failure:0
ports:
- 8090:8080
- 50061:50051
environment:
QUERY_DEFAULTS_LIMIT: 20
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: "./weaviate-data"
BACKUP_FILESYSTEM_PATH: "/tmp/backups"
ENABLE_MODULES: backup-filesystem
CLUSTER_GOSSIP_BIND_PORT: "7100"
CLUSTER_DATA_BIND_PORT: "7101"
DISABLE_TELEMETRY: 'true'
...
2 changes: 1 addition & 1 deletion ci/docker-compose-cluster.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,4 +86,4 @@ services:
CONTEXTIONARY_URL: contextionary:9999
DEFAULT_VECTORIZER_MODULE: text2vec-contextionary
ENABLE_MODULES: text2vec-contextionary
...
...
1 change: 1 addition & 0 deletions ci/docker-compose-openai.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ services:
image: semitechnologies/weaviate:${WEAVIATE_VERSION}
ports:
- 8086:8086
- 50057:50051
restart: on-failure:0
environment:
QUERY_DEFAULTS_LIMIT: 25
Expand Down
53 changes: 53 additions & 0 deletions ci/docker-compose-proxy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
version: '3.4'
services:
weaviate-proxy:
command:
- --host
- 0.0.0.0
- --port
- '8020'
- --scheme
- http
- --write-timeout=600s
image: semitechnologies/weaviate:${WEAVIATE_VERSION}
restart: on-failure:0
environment:
CONTEXTIONARY_URL: contextionary:9999
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-contextionary'
ENABLE_MODULES: text2vec-contextionary
BACKUP_FILESYSTEM_PATH: "/tmp/backups"
CLUSTER_GOSSIP_BIND_PORT: "7100"
CLUSTER_DATA_BIND_PORT: "7101"
CLUSTER_HOSTNAME: "node1"
AUTOSCHEMA_ENABLED: 'false'
DISABLE_TELEMETRY: 'true'
GRPC_PORT: 8021
contextionary:
environment:
OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75
EXTENSIONS_STORAGE_MODE: weaviate
EXTENSIONS_STORAGE_ORIGIN: http://weaviate-proxy:8020
NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5
ENABLE_COMPOUND_SPLITTING: 'false'
image: semitechnologies/contextionary:en0.16.0-v1.2.0
ports:
- 9999:9999
proxy-http:
image: envoyproxy/envoy:v1.29-latest
command: envoy --config-path /etc/envoy/http.yaml
ports:
- 10000:10000
volumes:
- ./proxy:/etc/envoy
proxy-grpc:
image: envoyproxy/envoy:v1.29-latest
command: envoy --config-path /etc/envoy/grpc.yaml
ports:
- 10001:10000
volumes:
- ./proxy:/etc/envoy
...
30 changes: 30 additions & 0 deletions ci/docker-compose-rerank.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
version: '3.4'
services:
weaviate-reranker:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
image: semitechnologies/weaviate:${WEAVIATE_VERSION}
ports:
- 8079:8080
- 50050:50051
restart: on-failure:0
environment:
RERANKER_INFERENCE_API: http://reranker-transformers:8080
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: "./data"
DEFAULT_VECTORIZER_MODULE: 'text2vec-openai'
ENABLE_MODULES: 'text2vec-openai,reranker-transformers,generative-openai'
CLUSTER_HOSTNAME: 'node1'
DISABLE_TELEMETRY: 'true'
reranker-transformers:
image: semitechnologies/reranker-transformers:cross-encoder-ms-marco-MiniLM-L-6-v2
environment:
ENABLE_CUDA: '0'
...
30 changes: 22 additions & 8 deletions ci/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,43 @@
version: '3.4'
services:
weaviate:
command:
- --host
- 0.0.0.0
- --port
- '8080'
- --scheme
- http
- --write-timeout=600s
image: semitechnologies/weaviate:${WEAVIATE_VERSION}
restart: on-failure:0
ports:
- "8080:8080"
- "50051:50051"
restart: on-failure:0
environment:
CONTEXTIONARY_URL: contextionary:9999
QUERY_DEFAULTS_LIMIT: 20
QUERY_DEFAULTS_LIMIT: 25
AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED: 'true'
PERSISTENCE_DATA_PATH: "./weaviate-data"
DEFAULT_VECTORIZER_MODULE: text2vec-contextionary
ENABLE_MODULES: text2vec-contextionary,backup-filesystem
PERSISTENCE_DATA_PATH: '/var/lib/weaviate'
DEFAULT_VECTORIZER_MODULE: 'text2vec-contextionary'
ENABLE_MODULES: text2vec-contextionary,backup-filesystem,img2vec-neural
BACKUP_FILESYSTEM_PATH: "/tmp/backups"
CLUSTER_GOSSIP_BIND_PORT: "7100"
CLUSTER_DATA_BIND_PORT: "7101"
CLUSTER_HOSTNAME: "node1"
AUTOSCHEMA_ENABLED: 'false'
IMAGE_INFERENCE_API: "http://i2v-neural:8080"
DISABLE_TELEMETRY: 'true'
contextionary:
image: semitechnologies/contextionary:en0.16.0-v1.2.1
ports:
- "9999:9999"
environment:
OCCURRENCE_WEIGHT_LINEAR_FACTOR: 0.75
EXTENSIONS_STORAGE_MODE: weaviate
EXTENSIONS_STORAGE_ORIGIN: http://weaviate:8080
NEIGHBOR_OCCURRENCE_IGNORE_PERCENTILE: 5
ENABLE_COMPOUND_SPLITTING: 'false'
image: semitechnologies/contextionary:en0.16.0-v1.2.0
ports:
- 9999:9999
i2v-neural:
image: semitechnologies/img2vec-pytorch:resnet50
...
Loading

0 comments on commit 565fb7f

Please sign in to comment.