Daft Files

What's Changed 🚀

We have a pretty crazy release this time around. Some especially notable features include: * Interactive DataFrames in Jupyter Notebooks, with special support for some multimodal types * An async API for LLM text generation, particularly with OpenAI * A new .into_batches DataFrame API, the modern alternative to .into_partitions * Adding support for .offset / OFFSET operator across the engine. Thanks @plotor for the great work! * Various Flotilla performance and reliability improvements * Various casting improvements

✨ Features

• feat: Async open ai llm generate @colin-ho (#4879)
• feat: Add offset syntax support to SQL @plotor (#4707)
• feat: adds support for SQL GROUP BY column position @rchowell (#4955)
• feat: better dtype type inference @universalmind303 (#4973)
• feat: Casting from Python into struct or list types @srilman (#4957)
• feat: support creating partitioned tables in Iceberg via the Catalog interface. @redpheonixx (#4951)
• feat: implement into_batches operator on flotilla distrubted engine @ohbh (#4958)
• feat: literal variants for (pretty much) all types @kevinzwang (#4947)
• feat: Add offset support to Flotilla Engine @plotor (#4918)
• feat: implement into_batches on the swordfish native daft runner @ohbh (#4935)
• feat: Flotilla broadcast join @colin-ho (#4867)

🐛 Bug Fixes

• fix: Always just use actor for flotilla scheduler @colin-ho (#4978)
• fix: Add handle for swordfish runtime stats manager @colin-ho (#4970)
• fix: Dudep lance read required columns @xloya (#4967)
• fix: Don't use wildcard for logical plan match in pushdown rules @colin-ho (#4945)
• fix: Coerce arrow schema for parquet decoding @colin-ho (#4948)
• fix: use associate type for swordfish into_batches operator state @ohbh (#4956)
• fix: raise error on invalid cross join parameters @rchowell (#4952)
• fix: interactive html fixes @colin-ho (#4943)

♻️ Refactor

• refactor: Use associate type for swordfish operator state @colin-ho (#4921)

📖 Documentation

• docs: update links in document processing example @ccmao1130 (#4946)
• docs: improve daft.func documentation and type inference @universalmind303 (#4942)
• docs: fix link for pandas @universalmind303 (#4941)

👷 CI

• ci: add 3.11 to matrix and always include typing_extensions @rchowell (#4950)

🔧 Maintenance

• chore: Remove Err Case for Series and RecordBatch size_bytes @srilman (#4954)

👋 New Contributors

• @redpheonixx made their first contribution in https://github.com/Eventual-Inc/Daft/pull/4951

Full Changelog: https://github.com/Eventual-Inc/Daft/compare/v0.5.18...v0.5.19