Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/merge dev minor main #1457

Merged
merged 3 commits into from
Oct 23, 2024
Merged

Conversation

emrgnt-cmplxty
Copy link
Contributor

@emrgnt-cmplxty emrgnt-cmplxty commented Oct 23, 2024

Important

This pull request enhances ingestion processes with a run_with_orchestration option, updates vector index management APIs, and refines testing and documentation.

  • Ingestion Process:
    • Added run_with_orchestration option to ingest_files, update_files, and ingest_chunks in ingestion_router.py, r2rClient.ts, and ingestion.py to allow synchronous execution without orchestration.
    • Updated CLI commands in ingestion.py to support --run-without-orchestration flag.
  • Vector Index Management:
    • Added new API endpoints for list_vector_indices and delete_vector_index in ingestion_router.py.
    • Updated mint.json to include new vector index endpoints.
  • Testing and Documentation:
    • Removed cli-graphrag and sdk-graphrag test categories from r2r-full-py-integration-tests.yml and r2r-light-py-integration-tests.yml.
    • Added new API reference files delete_vector_indices.mdx and list_vector_index.mdx.
    • Updated ingestion documentation in js-sdk/ingestion.mdx and python-sdk/ingestion.mdx to include run_with_orchestration parameter.

This description was created by Ellipsis for fda58eb. It will automatically update as commits are pushed.

emrgnt-cmplxty and others added 3 commits October 22, 2024 10:50
* add run without orchestration

* bump versions

* bump versions

* bump versions

* fix

* up
@emrgnt-cmplxty emrgnt-cmplxty marked this pull request as ready for review October 23, 2024 01:51
@emrgnt-cmplxty emrgnt-cmplxty merged commit 816a860 into dev-minor Oct 23, 2024
3 of 13 checks passed
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Reviewed everything up to fda58eb in 1 minute and 16 seconds

More details
  • Looked at 581 lines of code in 14 files
  • Skipped 1 files when reviewing.
  • Skipped posting 5 drafted comments based on config settings.
1. .github/actions/setup-python-light/action.yml:27
  • Draft comment:
    Ensure that the PATH modification persists across steps by using the 'GITHUB_PATH' environment file.
        echo "$env:APPDATA\Python\Scripts" | Out-File -Append -FilePath $env:GITHUB_PATH
  • Reason this comment was not posted:
    Comment did not seem useful.
2. .github/workflows/r2r-full-py-integration-tests.yml:74
  • Draft comment:
    Ensure that the removal of 'Run CLI GraphRAG Tests' and 'Run SDK GraphRAG Tests' is intentional and that these tests are not needed elsewhere.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR removes the 'Run CLI GraphRAG Tests' and 'Run SDK GraphRAG Tests' steps from the workflow. This might be intentional, but it's important to ensure that these tests are no longer needed or have been moved elsewhere. If they are still needed, their removal could lead to missing test coverage.
3. .github/workflows/r2r-light-py-integration-tests.yml:78
  • Draft comment:
    Ensure that the removal of 'Run CLI GraphRAG Tests' and 'Run SDK GraphRAG Tests' is intentional and consistent with the full integration tests workflow.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The same removal of 'Run CLI GraphRAG Tests' and 'Run SDK GraphRAG Tests' is present in this workflow file as well. It's important to ensure consistency and that these tests are not required.
4. docs/api-reference/endpoint/delete_vector_indices.mdx:2
  • Draft comment:
    The title and openapi path seem to be swapped with 'list_vector_index.mdx'. Ensure the title and openapi path are correct.
---
title: Delete Vector Index
openapi: 'DELETE /v2/delete_vector_index'
---
  • Reason this comment was not posted:
    Comment did not seem useful.
5. docs/api-reference/endpoint/list_vector_index.mdx:2
  • Draft comment:
    The title and openapi path seem to be swapped with 'delete_vector_indices.mdx'. Ensure the title and openapi path are correct.
---
title: List Vector Indices
openapi: 'GET /v2/list_vector_indices'
---
  • Reason this comment was not posted:
    Marked as duplicate.

Workflow ID: wflow_QnSuaiZkBsiU29PX


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

emrgnt-cmplxty added a commit that referenced this pull request Oct 23, 2024
* fix-actions (#1426)

* up

* modify

* add to github path

* Contextual Chunk Enrichment (#1433)

* add semantic chunking

* working

* precommit

* pre-commits

* Entity Deduplication (#1431)

* Modify graphrag prompt (#1421)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

* Feature/include vectors option document chunks (#1419)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

---------

Co-authored-by: Nolan Tremelling <[email protected]>

* Allow env var to set the default R2R deployment for the dashboard (#1417)

* modify community_summary_prompt function and corresponding prompt

* add tests

* up

* Feature/various documentation tweaks (#1422)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

* some various documentation tweaks

* some various documentation tweaks

---------

Co-authored-by: Nolan Tremelling <[email protected]>

* Graphrag tests (#1418)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* Modify graphrag tests timeouts (#1416)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* modify timeouts

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* feat: Make prompt provider methods asynchronous (comments below) (#1415)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* feat: Make prompt provider methods asynchronous

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* up

* up

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* Add routes, service level methods around conversations (#1420)

* Add routes, service level methods around conversations

* Slight refactor to match project conventions, add JS methods

* Updated JS methods

* JS docs

* Add python

* Update JS user tests

* add deduplication pipe, workflow, api, sdk, cli

* add summary workflow

* bug fixes

* pre-commit

* working

* search working

* adding dedup test files

* modify the update query

* precommit

* more testing

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* Refactor Python SDK for Intellisense, Thread Safety (#1430)

* Refactor Python SDK

* Fix CLI after SDK changes

* Add convo to agent

* Update conversation error handling, JS

* Remove unused, bad import

* Feature/cleanup refactor vector collection (#1432)

* Feature/include vectors option document chunks (#1419)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

---------

Co-authored-by: Nolan Tremelling <[email protected]>

* Allow env var to set the default R2R deployment for the dashboard (#1417)

* Feature/various documentation tweaks (#1422)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

* some various documentation tweaks

* some various documentation tweaks

---------

Co-authored-by: Nolan Tremelling <[email protected]>

* Graphrag tests (#1418)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* Modify graphrag tests timeouts (#1416)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* modify timeouts

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* feat: Make prompt provider methods asynchronous (comments below) (#1415)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* feat: Make prompt provider methods asynchronous

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>
Co-authored-by: emrgnt-cmplxty <[email protected]>

* bump pyproject version

* first commit

* towards slimmer vector implementation logic

* up

* iterate

* up

* checkin

* up

* work doc chunks

* working vector search

* working full text search

* remove asyncpg

* passing vector tests

* up

* merge

* rm pytest

* up

* up

* fix delete

* up

* up

---------

Co-authored-by: Nolan Tremelling <[email protected]>
Co-authored-by: Shreyas Pimpalgaonkar <[email protected]>

* Add tests + Cleanup (#1437)

* up

* add tests

* test rename to sdk

* up

* fix tests

* typo

* modify chunk enrichment prompt (#1438)

* modify prompt

* up

* Fix type error on port argument of CLI (#1439)

* finish (#1440)

* finish

* up

* fix

* fix

* up

* fix

* final cleanups

* fix naming convention

* fix schema error

* increase timeout

* split graphrag actions

* fix collection exists error

* up (#1442)

* Add error message (#1443)

* up

* sdk fix

* locally testing build

* up docs (#1445)

* checkin work (#1444)

* checkin work

* finish index functionality extension

* fix concurrency

* add alembic (#1446)

* Prompt Tuning (#1447)

* Check in

* Fix after merging dev-minor in

* Ensure to not cause int overflow with hatchet (#1454)

* Bump JS (#1456)

* Ensure to not cause int overflow with hatchet

* bump js

* improve migration implementation (#1452)

* improve migration implementation

* refine migrations to include kg

* add alembic cli

* extend documentation

* extend docs and all that

* Revert change of default behaviour of entities endpoint, docs, tests (#1455)

* change def behavior of entities + delete endpoint

* pre-commit

* add deduplication tests

* Delete graph (#1450)

* up docs

* up

* up

* rename to raw_chunks

* up

* add tests

* up

* up

* change default

* change cli

* separate out deduplication tests

* change run type in the test

* up

* up

* add test concurrency

* up

* rm concurrency groups

* rm dedup tests

* remove json

* tests

* up

* fix lock

* Update postgres.py

* Feature/merge dev minor main (#1457)

* add run without orchestration (#1448)

* add run without orchestration

* bump versions

* bump versions

* bump versions

* fix

* up

* add end points

* add run without orchestration (#1448) (#1458)

* add run without orchestration

* bump versions

* bump versions

* bump versions

* fix

* up

* sync migration changes

* Ensure that we await ingest files in ingest_files method (#1460)

* Nolan/await update files (#1461)

* Ensure that we await ingest files in ingest_files method

* Await update files as well

* Docs changes (#1462)

* up

* up

* up

* up

* fix failed find and replace (#1463)

* fix failed find and replace

* fix

* Fix JS Client for Ingest Chunks (#1464)

* Ensure that we await ingest files in ingest_files method

* Await update files as well

* Fix js client

---------

Co-authored-by: Shreyas Pimpalgaonkar <[email protected]>
Co-authored-by: Nolan Tremelling <[email protected]>
@emrgnt-cmplxty emrgnt-cmplxty deleted the feature/merge-dev-minor-main branch October 23, 2024 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant