[FEAT] Add "auto" threshold configuration via Statistical analysis in SemanticChunker + minor fixes #79
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Introduced a warning message when no similarity threshold is specified, defaulting to the 80th percentile. - Removed the previous blocking warning and replaced it with a non-blocking warning to improve user experience. - Adjusted the calculation of similarity threshold to ensure proper handling of percentile values.
- Updated the method signatures of `_encode` and `_encode_batch` in the BaseChunker class to include return type hints, improving code clarity and type safety. - The `_encode` method now explicitly returns a List[int], while `_encode_batch` returns a List[List[int]].
…ing percentile mode - Updated SDPMChunker and SemanticChunker to replace similarity_threshold and similarity_percentile with a unified threshold parameter, enhancing clarity and usability. - Introduced new parameters: mode, min_sentences, min_characters_per_sentence, and threshold_step to provide more control over chunking behavior. - Refactored chunking logic to support both cumulative and window-based grouping of sentences, improving flexibility in semantic chunking. - Enhanced docstrings and method signatures for better documentation and understanding of class functionalities. - Updated tests to reflect changes in parameter names and ensure proper initialization and functionality of chunkers.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request includes significant changes to the
SDPMChunkerandSemanticChunkerclasses to improve the chunking logic and introduce new parameters for better control over the chunking process. Key updates include the addition of new methods, parameters, and refactoring of existing logic to enhance functionality and readability.Major Changes:
Enhancements to
SDPMChunker:SDPMChunkerclass, includingmode,threshold,similarity_window,min_sentences,min_characters_per_sentence, andthreshold_step, to provide more control over the chunking process. (src/chonkie/chunker/sdpm.py)src/chonkie/chunker/sdpm.py)Enhancements to
SemanticChunker:SemanticChunkerclass, similar toSDPMChunker, to provide better control over the chunking process. (src/chonkie/chunker/semantic.py) [1] [2]_prepare_sentencesmethod to compute embeddings based on a similarity window, enhancing the accuracy of semantic grouping. (src/chonkie/chunker/semantic.py)src/chonkie/chunker/semantic.py)Code Refactoring:
_encodeand_encode_batchmethods inBaseChunkerto include return type annotations for better type checking and readability. (src/chonkie/chunker/base.py) [1] [2]src/chonkie/chunker/sdpm.py,src/chonkie/chunker/semantic.py) [1] [2]These changes collectively enhance the functionality, flexibility, and readability of the chunking logic in the
chonkielibrary.