Feature/v3 api (#1656)

* improve ci/cd runtime * update prompt tests * improve ci/cd runtime (#1535) * improve ci/cd runtime * update prompt tests * Support Python ^3.10 (#1534) * add azure * up * up * spec out v3 api * checkin document router * adding chunk abstr * add list chunks * add chunk search * up * add users routes * up * checkin progress * add collections annotations * add indices * add user * checkin work * up * complete conversations CRUD * fix type errors * add graph router * add graphs * Update JS (#1563) * Feature/add graph to v3 (#1565) * complete simple tests, cleanup routers * up * Harmonize Pagination across endpoints (#1564) * Pagination * Add fixmes * Fix nested deletion filter bug (#1567) * Remove Mintlify docs (#1569) * Nolan/list collection (#1568) * Check in * More * Fix nested transactions issue in sqlite logger * Fix update collection return type * JS V3 (#1571) * Sync collections JS * More documents * Clean up messy code * list not List * Users first pass * User tests and fixmes * More * typo * More prompts * Pre-commit improvements * Remove prints * Cleanups on conversations * Branches response * Chunks * More work on the return types * Jest config * Fix branch creation time * Fix lock * Nolan/v3 tests (#1578) * Add deprecated command back * Add warning * Fix GraphRAG tests (#1579) * More cleanup (#1580) * More cleanup * More * Fix test * More cleanups * More cleanups * More * Merge main * Python SDK V3 (#1585) * Python SDK V3 * Fix * First pass (#1586) * More V3 (#1587) * Validation errors * Update js test * more * Fix sync methods on v2 sdk, add check for download files (#1588) * More CLI (#1589) * Print logs on failing tests (#1590) * Print logs on failing tests * MOre * cleanup * Again * Again * More JS testing (#1591) * More JS testing * Cleanup * More refactors for tests (#1592) * System Routes (#1594) * Fix type errors, pass collection id (#1595) * Hotfix: dict * V3 graph implmentations (#1593) * complete simple tests, cleanup routers * up * up * checkin * up * up * response models * checkin * up * checkin * up * up * up * up * up * up * v2 * up * up * up * up --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * Allow passing of collection id at document ingestion (#1596) * KG Response sync (#1597) * fix * Fix Prompt Override (#1599) * Fix Prompt Override * print * Caching * Fix * Updated Graph Models, Drop SID (#1598) * New Graph Models * Fix * minor tweaks * fix summary model (#1604) * incr progress * Add /users/me (#1605) * Add /users/me * oops * Resolve Merge Conflicts (#1607) * Fix conflicts * Clean up * Nolan/conflicts (#1608) * expose reset data to admin (#1602) * up (#1603) * up * up * wtf github is a piece of garbage --------- Co-authored-by: emrgnt-cmplxty <[email protected]> * wrapup walkthrough * Add delete user method, sync JS to camel case (#1609) * V3 graph testing (#1606) * up * up * up * graph crud * up * community endpts * up * up * up * up * up * up * up * up * add back routers * up * pre-commit * Fix Broken V2 Graphs, Better Response Models (#1612) * Increase test coverage * Fix v2 graphs, better response models * Remaining types * Add types to Python SDK * Typo * update tests * revert test change * up * Add types to package export (#1613) * Graph refactor (#1611) * up * up * add back routers * up * pre-commit * update tests * revert test change * up * simplify * up * add the add/remove endpoints * up * include routers back * Create branch update (#1617) * Graph refactor (#1616) * up * up * add back routers * up * pre-commit * update tests * revert test change * up * simplify * up * add the add/remove endpoints * up * include routers back * List collections (#1619) * up * up * up * up * Graph refactor (#1620) * up * up * add back routers * up * pre-commit * update tests * revert test change * up * simplify * up * add the add/remove endpoints * up * include routers back * up * up * up * up * Nolan/update graph (#1621) * List collections * Update Graph JS SDK * up * up * cleanup * Graph refactor (#1622) * up * up * add back routers * up * pre-commit * update tests * revert test change * up * simplify * up * add the add/remove endpoints * up * include routers back * up * up * up * up * up * up * cleanup * up * up * up * remove unnecessary functions * up * up * complete document embedding workflow * working get command on graph * checkin progress * up * add entity and relationship deletions * no verif * up * up * up * up (#1636) * up * sync graph * up * up * fix relationship distance calc. * fix issue with faulty collection filter (#1637) * Patch/alternative fix logics 2 (#1638) * fix issue with faulty collection filter * further refinements, like fixing limits * up * fix logic around include metadata and scores * fix double collection assignment * up * fix communities * working clusters * up * add collection extraction * add collection extraction * up * prep for merge * Patch/alternative up with nolan (#1643) * SDK First pass * Add feature tracking * Typo * Check in * Rebase * Add Graph tests * Fix Agent empty message bug * Check in JS routes * More tests, examples * Sync python * Expose Entity/Relationship Params in Routes (#1640) * Expose Entity/Relationship Params * Descriptions * Modify create entities * Create relationships * set parent_id * Update entitiy * Update Relationships * Check in * Ellipsis fixes * More cleanup * Start CRUD on communities * Communities DB * Explicit working path * Once again * Fail fast false * Testing around community creation * Delete community test * Update community tests * Clean up type errors, cleaner code * More cleanup * More * remove chunk_entity * Delete bad, unused methods * More * fixup crud * rm pull --------- Co-authored-by: NolanTrem <[email protected]> * Feature/fix graph permissions (#1645) * update docs / collections * up * Feature/fix auth checks (#1647) * update docs / collections * up * fix super user and more * up * up (#1648) * Feature/rm v2 api (#1649) * SDK First pass * Add feature tracking * Typo * Check in * Rebase * Add Graph tests * Fix Agent empty message bug * Check in JS routes * More tests, examples * Sync python * Expose Entity/Relationship Params in Routes (#1640) * Expose Entity/Relationship Params * Descriptions * Modify create entities * Create relationships * set parent_id * Update entitiy * Update Relationships * Check in * Ellipsis fixes * More cleanup * Start CRUD on communities * Communities DB * Explicit working path * Once again * Fail fast false * Testing around community creation * Delete community test * Update community tests * Clean up type errors, cleaner code * More cleanup * More * remove chunk_entity * Delete bad, unused methods * More * remove v2 api * rm kg router * cleanups * fixup delete by filter * fixup delete by filter * fixes * up * up --------- Co-authored-by: NolanTrem <[email protected]> * Improved Data Structures (#1650) * Check in * Most tests fixed * fix tables * Once more * Move to a single community table * Don't modify existing migration script--keep them atomic * Migration * Migration, more clean up * All but deletion working * Up * Feature/tweaks for prod (#1651) * tweaks for prod * up * final tweaks * Nolan/deletion (#1652) * Check in * Most tests fixed * fix tables * Once more * Move to a single community table * Don't modify existing migration script--keep them atomic * Migration * Migration, more clean up * All but deletion working * Up * Fix deletion * Working migration (#1654) * Feature/production tweaks (#1653) * tweaks for prod * up * final tweaks * prod tweaks * fixed --------- Co-authored-by: NolanTrem <[email protected]> * sort --------- Co-authored-by: Nolan Tremelling <[email protected]> Co-authored-by: Shreyas Pimpalgaonkar <[email protected]>
SciPhi-AI · Dec 4, 2024 · d693f69 · d693f69
1 parent 17a34e6
commit d693f69
Show file tree

Hide file tree

Showing 638 changed files with 33,216 additions and 90,759 deletions.
diff --git a/.github/.codecov.yml b/.github/.codecov.yml
diff --git a/.github/actions/run-sdk-collections-tests/action.yml b/.github/actions/run-sdk-collections-tests/action.yml
@@ -97,3 +97,13 @@ runs:
       working-directory: ./py
       shell: bash
       run: poetry run python tests/integration/runner_sdk_basic.py test_delete_chunks
+
+    - name: Get all prompts
+      working-directory: ./py
+      shell: bash
+      run: poetry run python tests/integration/runner_sdk_basic.py test_get_all_prompts
+
+    - name: Get prompt
+      working-directory: ./py
+      shell: bash
+      run: poetry run python tests/integration/runner_sdk_basic.py test_get_prompt
diff --git a/.github/workflows/r2r-full-py-integration-tests-graphrag.yml b/.github/workflows/r2r-full-py-integration-tests-graphrag.yml
@@ -1,6 +1,14 @@
 name: R2R Full Python Integration Test (ubuntu)
 
 on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - dev
+      - dev-minor
+      - main
   workflow_dispatch:
 
 jobs:
@@ -47,3 +55,13 @@ jobs:
       - name: Run SDK GraphRAG Tests
         if: matrix.test_category == 'sdk-graphrag'
         uses: ./.github/actions/run-sdk-graphrag-tests
+
+      - name: Print Docker logs on failure
+        if: >
+          (matrix.test_category == 'cli-graphrag' && failure()) ||
+          (matrix.test_category == 'sdk-graphrag' && failure())
+        run: |
+          echo "::group::Docker Logs"
+          docker ps -a
+          docker logs r2r-full-r2r-1
+          echo "::endgroup::"
diff --git a/.github/workflows/r2r-full-py-integration-tests.yml b/.github/workflows/r2r-full-py-integration-tests.yml
@@ -1,6 +1,14 @@
 name: R2R Full Python Integration Test (ubuntu)
 
 on:
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - dev
+      - dev-minor
+      - main
   workflow_dispatch:
 
 jobs:
@@ -72,3 +80,18 @@ jobs:
       - name: Run SDK Prompt Tests
         if: matrix.test_category == 'sdk-prompts'
         uses: ./.github/actions/run-sdk-prompt-management-tests
+
+      - name: Print Docker logs on failure
+        if: >
+          (matrix.test_category == 'cli-ingestion' && failure()) ||
+          (matrix.test_category == 'sdk-ingestion' && failure()) ||
+          (matrix.test_category == 'cli-retrieval' && failure()) ||
+          (matrix.test_category == 'sdk-retrieval' && failure()) ||
+          (matrix.test_category == 'sdk-auth' && failure()) ||
+          (matrix.test_category == 'sdk-collections' && failure()) ||
+          (matrix.test_category == 'sdk-prompts' && failure())
+        run: |
+          echo "::group::Docker Logs"
+          docker ps -a
+          docker logs r2r-full-r2r-1
+          echo "::endgroup::"
diff --git a/.github/workflows/r2r-js-sdk-integration-tests.yml b/.github/workflows/r2r-js-sdk-integration-tests.yml
@@ -1,11 +1,59 @@
 name: R2R JS SDK Integration Tests
+
 on:
   push:
     branches:
-      - '**'  # Trigger on all branches
+      - '**'
+
 jobs:
-  test:
+  setup:
     runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python and install dependencies
+        uses: ./.github/actions/setup-python-light
+        with:
+          os: ubuntu-latest
+      - name: Setup and start PostgreSQL
+        uses: ./.github/actions/setup-postgres-ext
+        with:
+          os: ubuntu-latest
+      - name: Start R2R Light server
+        uses: ./.github/actions/start-r2r-light
+      - name: Use Node.js
+        uses: actions/setup-node@v2
+        with:
+          node-version: "20.x"
+      - name: Install pnpm
+        uses: pnpm/action-setup@v2
+        with:
+          version: 8.x
+          run_install: false
+      - name: Install JS SDK dependencies
+        working-directory: ./js/sdk
+        run: pnpm install
+      - name: Check if R2R server is running
+        run: |
+          curl http://localhost:7272/v2/health || echo "Server not responding"
+
+  v3-integration-tests:
+    needs: setup
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        test-group:
+          - ChunksIntegrationSuperUser.test.ts
+          - CollectionsIntegrationSuperUser.test.ts
+          - ConversationsIntegrationSuperUser.test.ts
+          - DocumentsAndCollectionsIntegrationUser.test.ts
+          - DocumentsIntegrationSuperUser.test.ts
+          - GraphsIntegrationSuperUser.test.ts
+          - PromptsIntegrationSuperUser.test.ts
+          - RetrievalIntegrationSuperUser.test.ts
+          - SystemIntegrationSuperUser.test.ts
+          - SystemIntegrationUser.test.ts
+          - UsersIntegrationSuperUser.test.ts
     env:
       OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
       AZURE_API_KEY: ${{ secrets.AZURE_API_KEY }}
@@ -42,9 +90,6 @@ jobs:
       - name: Install JS SDK dependencies
         working-directory: ./js/sdk
         run: pnpm install
-      - name: Check if R2R server is running
-        run: |
-          curl http://localhost:7272/v2/health || echo "Server not responding"
-      - name: Run integration tests
+      - name: Run remaining tests
         working-directory: ./js/sdk
-        run: pnpm test
+        run: pnpm jest ${{ matrix.test-group }}
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -8,6 +8,15 @@ repos:
       - id: check-ast
       - id: check-yaml
 
+  - repo: local
+    hooks:
+      - id: check-typing-imports
+        name: Check for Dict/List usage
+        entry: bash -c 'echo "Checking for typing imports..." && find . -name "*.py" | xargs grep -n "from typing.*import.*[^d]Dict\\|from typing.*import.*List" || exit 0 && echo "⚠️  Please import dict/list instead of Dict/List from typing" && exit 1'
+        language: system
+        types: [python]
+        pass_filenames: false
+
   - repo: local
     hooks:
       - id: isort

diff --git a/docs/README.md b/docs/README.md
diff --git a/docs/api-reference.mdx b/docs/api-reference.mdx
diff --git a/docs/api-reference/endpoint/add_prompt.mdx b/docs/api-reference/endpoint/add_prompt.mdx
diff --git a/docs/api-reference/endpoint/add_user_to_collection.mdx b/docs/api-reference/endpoint/add_user_to_collection.mdx
diff --git a/docs/api-reference/endpoint/agent.mdx b/docs/api-reference/endpoint/agent.mdx
diff --git a/docs/api-reference/endpoint/analytics.mdx b/docs/api-reference/endpoint/analytics.mdx
diff --git a/docs/api-reference/endpoint/app_settings.mdx b/docs/api-reference/endpoint/app_settings.mdx
diff --git a/docs/api-reference/endpoint/assign_document_to_collection.mdx b/docs/api-reference/endpoint/assign_document_to_collection.mdx
diff --git a/docs/api-reference/endpoint/change_password.mdx b/docs/api-reference/endpoint/change_password.mdx
diff --git a/docs/api-reference/endpoint/collections_overview.mdx b/docs/api-reference/endpoint/collections_overview.mdx
diff --git a/docs/api-reference/endpoint/communities.mdx b/docs/api-reference/endpoint/communities.mdx
diff --git a/docs/api-reference/endpoint/completion.mdx b/docs/api-reference/endpoint/completion.mdx
diff --git a/docs/api-reference/endpoint/create_collection.mdx b/docs/api-reference/endpoint/create_collection.mdx
diff --git a/docs/api-reference/endpoint/create_graph.mdx b/docs/api-reference/endpoint/create_graph.mdx
diff --git a/docs/api-reference/endpoint/create_vector_index.mdx b/docs/api-reference/endpoint/create_vector_index.mdx
diff --git a/docs/api-reference/endpoint/deduplicate_entities.mdx b/docs/api-reference/endpoint/deduplicate_entities.mdx
diff --git a/docs/api-reference/endpoint/delete.mdx b/docs/api-reference/endpoint/delete.mdx
diff --git a/docs/api-reference/endpoint/delete_collection.mdx b/docs/api-reference/endpoint/delete_collection.mdx
diff --git a/docs/api-reference/endpoint/delete_entities_and_triples.mdx b/docs/api-reference/endpoint/delete_entities_and_triples.mdx
diff --git a/docs/api-reference/endpoint/delete_prompt.mdx b/docs/api-reference/endpoint/delete_prompt.mdx
diff --git a/docs/api-reference/endpoint/delete_user_info.mdx b/docs/api-reference/endpoint/delete_user_info.mdx
diff --git a/docs/api-reference/endpoint/delete_vector_index.mdx b/docs/api-reference/endpoint/delete_vector_index.mdx
diff --git a/docs/api-reference/endpoint/document_chunks.mdx b/docs/api-reference/endpoint/document_chunks.mdx
diff --git a/docs/api-reference/endpoint/documents_overview.mdx b/docs/api-reference/endpoint/documents_overview.mdx
diff --git a/docs/api-reference/endpoint/enrich_graph.mdx b/docs/api-reference/endpoint/enrich_graph.mdx
diff --git a/docs/api-reference/endpoint/entities.mdx b/docs/api-reference/endpoint/entities.mdx
diff --git a/docs/api-reference/endpoint/get_all_prompts.mdx b/docs/api-reference/endpoint/get_all_prompts.mdx
diff --git a/docs/api-reference/endpoint/get_collection.mdx b/docs/api-reference/endpoint/get_collection.mdx
diff --git a/docs/api-reference/endpoint/get_collections_for_user.mdx b/docs/api-reference/endpoint/get_collections_for_user.mdx
diff --git a/docs/api-reference/endpoint/get_document_collections.mdx b/docs/api-reference/endpoint/get_document_collections.mdx
diff --git a/docs/api-reference/endpoint/get_prompt.mdx b/docs/api-reference/endpoint/get_prompt.mdx
diff --git a/docs/api-reference/endpoint/get_user_info.mdx b/docs/api-reference/endpoint/get_user_info.mdx
diff --git a/docs/api-reference/endpoint/get_users_in_collection.mdx b/docs/api-reference/endpoint/get_users_in_collection.mdx
diff --git a/docs/api-reference/endpoint/health.mdx b/docs/api-reference/endpoint/health.mdx
diff --git a/docs/api-reference/endpoint/ingest_chunks.mdx b/docs/api-reference/endpoint/ingest_chunks.mdx