Chonkie Files

Highlights

• Added LateChunker support! You can use LateChunker in the following manner:

  :::python from chonkie import LateChunker

  chunker = LateChunker( embedding_model="jinaai/jina-embeddings-v3", mode="sentence", trust_remote_code=True )

• Added Chonkie Discord to the repository~ Join now to connect with the community! Oh, btw, Chonkie is now on Twitter and Bluesky too!

• Bunch of bug fixes to improve chunkers' stability...

What's Changed

• [Fix] [#37]: Incorrect indexing when repetition is present in the text by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/87
• [Fix] [#88]: SemanticChunker raises UnboundLocalError: local variable 'threshold' referenced before assignment by @arpesenti in https://github.com/chonkie-ai/chonkie/pull/89
• [Fix] WordChunker chunk_batch fail by @sky-2002 in https://github.com/chonkie-ai/chonkie/pull/90
• [FIX] MEGA Bug Fix PR: Fix WordChunker batching, Fix SentenceChunker token counts, Initialization + more by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/96
• Add initial support for Late Chunking by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/97
• [FEAT] Add LateChunker by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/98
• [FIX] Update outdated package versions + set max limit to numpy to v2.2 (buggy) by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/99
• Update version to 0.3.0 in pyproject.toml and init.py by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/100
• [fix] Add LateChunker support to chunker and module exports by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/101
• [fix] Docstrings in SemanticChunker should include **kwargs by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/102
• [Minor] Add Discord badge to README for community engagement by @bhavnicksm in https://github.com/chonkie-ai/chonkie/pull/103

New Contributors

• @arpesenti made their first contribution in https://github.com/chonkie-ai/chonkie/pull/89

Full Changelog: https://github.com/chonkie-ai/chonkie/compare/v0.2.2...v0.3.0