Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add RecursiveSplitter component for Document preprocessing #8605

Open
wants to merge 82 commits into
base: main
Choose a base branch
from
Open

feat: add RecursiveSplitter component for Document preprocessing #8605

wants to merge 82 commits into from
+1,216 −2

Conversation

davidsbatista
Copy link
Contributor

Related Issues

Proposed Changes:

  • Adding a RecursiveSplitter, using a set of predefined separators to split text recursively - see issue for more details

How did you test it?

  • local unit tests and integration tests plus CI tests

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Dec 4, 2024
@coveralls
Copy link
Collaborator

coveralls commented Dec 4, 2024 •
edited
Loading

Pull Request Test Coverage Report for Build 12445051813

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage increased (+0.2%) to 91.119%
Files with Coverage Reduction New Missed Lines %
components/preprocessors/sentence_tokenizer.py 1 94.12%
Totals Coverage Status
Change from base Build 12443798462: 0.2%
Covered Lines: 8680
Relevant Lines: 9526
💛 - Coveralls

@davidsbatista davidsbatista marked this pull request as ready for review December 4, 2024 17:23
@davidsbatista davidsbatista requested review from a team as code owners December 4, 2024 17:23
@davidsbatista davidsbatista requested review from dfokina and julian-risch and removed request for a team December 4, 2024 17:23
@davidsbatista davidsbatista changed the title feat:: add recursive chunking strategy feat: add recursive chunking strategy Dec 4, 2024
@davidsbatista davidsbatista requested a review from sjrl December 4, 2024 17:24
@davidsbatista
Copy link
Contributor Author

@bglearning - mentioning you since I believe you were the one with most interest in this feature

davidsbatista and others added 10 commits December 19, 2024 10:23
@davidsbatista @sjrl
Update haystack/components/preprocessors/recursive_splitter.py
938b610 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@davidsbatista @sjrl
Update haystack/components/preprocessors/recursive_splitter.py
bc4dfbd 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@davidsbatista @sjrl
Update haystack/components/preprocessors/recursive_splitter.py
371028c 
Co-authored-by: Sebastian Husch Lee <[email protected]>
@davidsbatista
wip: adding more tests for word unit length
79cd8bd
@davidsbatista
fix
31c8412
@anakin87 @dfokina @davidsbatista
feat: Tool dataclass - unified abstraction to represent tools (#8652)
e1fed92 
* draft

* del HF token in tests

* adaptations

* progress

* fix type

* import sorting

* more control on deserialization

* release note

* improvements

* support name field

* fix chatpromptbuilder test

* port Tool from experimental

* release note

* docs upd

* Update tool.py

---------

Co-authored-by: Daria Fokina <[email protected]>
@wochinge @davidsbatista
fix: fix deserialization issues in multi-threading environments (#8651)
f71a22b
@davidsbatista
adding 'word' as default length
211c4ed
@davidsbatista
fixing types
0807902
@davidsbatista
handing both default strategies
460cc7d
@github-actions github-actions bot added topic:tests type:documentation Improvements on the docs labels Dec 20, 2024
@davidsbatista davidsbatista requested a review from sjrl December 23, 2024 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@julian-risch julian-risch julian-risch left review comments

@dfokina dfokina Awaiting requested review from dfokina dfokina is a code owner automatically assigned from deepset-ai/documentation

@anakin87 anakin87 Awaiting requested review from anakin87

@sjrl sjrl Awaiting requested review from sjrl

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
topic:tests type:documentation Improvements on the docs type:feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add a Recursive Chunking strategy
6 participants
@davidsbatista @coveralls @sjrl @julian-risch @anakin87 @wochinge