[FEAT] Add BaseRefinery and OverlapRefinery support #77
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I have added a small documentation on how to setup local env for testing. **Changes in pyproject.toml** `pytest` was not picking up local changes. It was giving `ModuleNotFoundError` error for `chonkie`. Adding `pythonpath` fixed that. **Automated Testing** I have also added a github action to run the tests automatically on each `git push`. I have used `uv` because of superfast dependency installation. I followed this guide for the GitHub Actions setup: https://docs.astral.sh/uv/guides/integration/github/
- Added ruff checks for import sorting and docstring formatting - Fixed docstrings across chunker and embeddings modules to comply with standards - Updated pyproject.toml to include ruff configuration
- Introduced a new Context class for managing contextual information during chunk refinement. - Added a new Refinery module with BaseRefinery and OverlapRefinery classes to enhance chunk processing. - Updated the __init__.py files to include new classes in the package exports. - Modified the Chunk class to incorporate context attributes. - Enhanced the pyproject.toml to include the new refinery package. - Added tests for OverlapRefinery to ensure functionality and correctness.
…updates - Added error handling for missing embedding model, prompting installation of the `semantic` extra. - Updated similarity threshold assignment to use the instance variable consistently. - Introduced a new test for SDPMChunker to validate functionality with percentile-based similarity, ensuring proper chunking behavior and attributes.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and new features across multiple files in the project. The most important changes include the addition of a new GitHub Actions workflow for automated Python testing, updates to the project dependencies, and enhancements to the chunking and embedding functionalities.
New Features:
.github/workflows/python-test-push.yml: Added a new GitHub Actions workflow for automated Python testing on push events. This workflow installs dependencies, sets up Python, and runs tests usingpytest.Dependency Updates:
pyproject.toml: Updated development dependencies, replacedblack,isort,flake8,mypy, andpylintwithruff, and addedpytest.ini_optionsandruffconfiguration sections.Code Enhancements:
src/chonkie/__init__.py: Refactored imports for better readability and added new imports forContextandRefineryclasses. Updated__all__to include new classes. [1] [2] [3]src/chonkie/chunker/base.py: Added docstrings, improved type annotations, and refactored methods for better error handling and readability. Introduced a newcontextattribute to theChunkclass. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10]src/chonkie/chunker/sdpm.py: Added a newmin_chunk_sizeparameter to theSDPMChunkerclass and updated the constructor and docstrings accordingly. [1] [2] [3] [4]src/chonkie/chunker/semantic.py: RefactoredSemanticSentenceandSemanticChunkclasses to usedataclassfields and added detailed docstrings to theSemanticChunkerclass. [1] [2] [3] [4]Documentation:
README.md: Added a new section for citations and a special thanks note.