mastodon 4.7.3Delta Lake is an open-source project that enables building a Lakehouse architecture on top of existing storage system. Build your lakehouse with Delta Lake!
Learn about the upcoming enhancement in #Rust's crate capabilities in our upcoming Delta Lake Deep Dive with @rtyler@hacky.town, Delta-rs Maintainer and founder of Buoyant Data! ⭐
Say goodbye to the old DynamoDB locking mechanism and welcome the innovative Delta/Spark compatible concurrency support - the S3DynamoDbLogStore. This enhancement is a game-changer, thanks to the awesome efforts of Thomas Peiselt. 🦀
RSVP: https://www.meetup.com/data-ai-online/events/298469304/
TOMORROW, January 31st at 9AM PT, join us for #DeltaLake Deep Dive: Rust Crate with @rtyler@hacky.town, Delta-rs Maintainer and founder of Buoyant Data! 🦀 Learn about the upcoming enhancement in Rust's crate capabilities.
Say goodbye to the old DynamoDB locking mechanism and welcome the innovative Delta/Spark compatible concurrency support - the S3DynamoDbLogStore.
RSVP on LinkedIn 👉 https://hubs.la/Q02ghtvC0
RSVP on Meetup 👉 https://www.meetup.com/data-ai-online/events/298469304/
In two weeks, @rtyler@hacky.town, Delta-rs Maintainer and founder of Buoyant Data, will talk about the upcoming enhancement in #Rust's crate capabilities! 🦀
Say goodbye to the old DynamoDB locking mechanism and welcome the innovative Delta/Spark compatible concurrency support - the S3DynamoDbLogStore. 🚀 Bring your questions and get ready for an engaging Delta Lake Deep Dive on Wednesday, Jan 31st.
RSVP: https://www.meetup.com/data-ai-online/events/298469304/
How does Delta Lake quickly compute state? Here’s a quick primer on how #checkpoint files are used to speed up the process.
https://dennyglee.com/2024/01/09/computing-delta-lake-state-quickly-with-checkpoint-files/
We are excited to announce the release of python-v0.19.0! 🎉
Highlights include:
🌟 CDF support in write_deltalake, delete, and merge operation
🌟 Expired logs cleanup during post-commit. Can be disabled with 𝚍𝚎𝚕𝚝𝚊.𝚎𝚗𝚊𝚋𝚕𝚎𝙴𝚡𝚙𝚒𝚛𝚎𝚍𝙻𝚘𝚐𝙲𝚕𝚎𝚊𝚗𝚞𝚙 = 𝚏𝚊𝚕𝚜𝚎
🌟 Improved MERGE performance by using predicate non-partition columns min/max for prefiltering
🌟 𝙰𝙳𝙳 𝚌𝚘𝚕𝚞𝚖𝚗 option
🌟 Speed up log parsing
🔗 View the release notes: https://github.com/delta-io/delta-rs/releases/tag/python-v0.19.0
Delta UniForm is extremely easy to set up, and once it's enabled it works fast, seamlessly, and automatically!
Delta UniForm has negligible performance and resource overhead, ensuring optimal utilization of computational resources.
🔗Learn more: https://hubs.la/Q02f_W6p0
You can retrieve information on the operations, user, timestamp, and so on for each write to a Delta table by running the 𝐡𝐢𝐬𝐭𝐨𝐫𝐲 command! The operations are returned in reverse chronological order and by default, table history is retained for 30 days.
Learn more about retrieving Delta table history ➡ https://docs.delta.io/latest/delta-utility.html#retrieve-delta-table-history
The Delta Lake transaction log provides #ACID support, snapshot isolation for reads, scalability for millions of partitions or files, is designed for #streaming, and is self-describing. In this post, let's dive into the metadata! ✔️
https://dennyglee.com/2024/01/03/a-peek-into-the-delta-lake-transaction-log/
Happy New Year! 🚀 Are you interested in contributing to #DeltaLake in the new year? 🦀 Dive into the Delta Lake Roadmap and explore the highest priority issues across the Delta Lake ecosystem. Every contribution makes an impact! Let's continue to innovate and grow together in 2024.
View the roadmap here ➡ https://github.com/orgs/delta-io/projects/10/views/1
We are excited to announce the release of Delta Lake 3.2.0! 🎉 This release includes several exciting new features and improvements. Delta Spark 3.2.0 is built on Apache Spark™ 3.5. Similar to Apache Spark, we have released Maven artifacts for both Scala 2.12 and Scala 2.13.
View the full release notes for more info ➡ https://github.com/delta-io/delta/releases/tag/v3.2.0
⭐ Join us for the next #DeltaLake Community Meetup on Wednesday, May 22nd at 9AM PST / 12PM EST.
In this virtual event, we will cover the latest features and improvements in Delta Lake 3.2, walk through the Delta Lake + Pandas integration, discuss what's next for Delta Lake, and more. 🦀
Bring your #DeltaLake questions and get instant feedback from core contributors and maintainers of the project.
Register here 👉 https://us06web.zoom.us/webinar/register/WN_9yqi7z2wSuCtVqqcuAeA8g#/registration
We are excited to announce the first release candidate for #DeltaLake 3.2.0! 🚀
Some highlights include:
✅ Support for Liquid clustering to reduce write amplification using incremental clustering.
✅ Support for Type Widening (preview) to allow users to change the type of columns without having to rewrite data.
✅ Support for Apache Hudi (preview) in Delta UniForm tables.
✅ Support for Vacuum Inventory.
Please see the full release notes for more info: https://github.com/delta-io/delta/releases/tag/v3.2.0rc1
Delta Lake provides a convenient way to enforce data quality by adding constraints to a table, ensuring that only valid and consistent data can be added.
In the provided code, attempting to add new data with a negative salary violates the constraint of a positive salary, and thus, the data is not added to the table.
🚀 Link to delta-rs: https://bit.ly/3S3f7bS
⭐️ Bookmark this post: https://bit.ly/3SvvvDy
cc Khuyen Tran
📢 Attention Delta Lake community members! Join us for the #DeltaLake Community Meetup on Wednesday, February 7th at 9AM PST. From project updates to open discussions and collaboration opportunities, there's something for every community member.
How Delta Lake handles multiple concurrent reads and writes - Denny Lee dives into how #DeltaLake solves conflicts optimistically and utilizes multi-version concurrency control.
Read here: https://dennyglee.com/2024/01/16/how-delta-lake-handles-multiple-concurrent-reads-and-writes/
Learn how to remove files marked for deletion (aka “tombstoned files”) from storage with the Delta Lake vacuum command. ✅
#DeltaLake doesn't physically remove files from storage for operations that logically delete the files. You need to use the vacuum command to physically remove files from storage that have been marked for deletion and are older than the retention period.
Learn more ➡️ https://delta.io/blog/remove-files-delta-lake-vacuum-command/
We are excited to announce the release of python-v0.15.0! 🎊
In this release...
✅ Merge can now do partition pruning based on the predicate which tremendously improves execution performance on a table with many partitions.
✅ Files that have no changes will not be rewritten anymore reducing the merge execution.
✅ Add constraints to a table and write to a delta table with constraints if you use the #rust engine parameter.
View the release notes ➡ https://github.com/delta-io/delta-rs/releases/tag/python-v0.15.0
