DATA Pill #061 - Apache Celeborn, 8 Futuristic Databases to Watch in 2023

ARTICLES

Check out the DevGenius blog post that introduces Apache Celeborn Shuffle Service, a fresh addition to Apache Spark. The article dives into the advantages and capabilities of this feature, which focuses on enhancing shuffle performance and minimizing memory usage in Spark applications. By delegating shuffle operations to a dedicated service, Celeborn boosts the scalability and efficiency of Spark clusters, leading to optimized performance for data-intensive workloads.

This one explores the implementation of dbt incremental models with Teradata Vantage temporal tables, focusing on the sample ELT process of the fictional store Teddy Retailers. Dive into and learn more about achieved incremental ingestion of newly inserted or updated data into our dbt models.

Marie provides a comprehensive overview of data documentation, covering its importance, methods and intended audience. Dig into the significance of data documentation in enabling effective data management, collaboration and reproducibility. It offers insights and practical guidance on creating thorough and accessible data documentation for various stakeholders.

Let's dive into semi-supervised learning in the context of real-time data streams. It explores how this approach can be utilized to enhance machine learning models by incorporating unlabeled data alongside labeled data. By leveraging semi-supervised learning techniques, organizations can make the most of their real-time data streams and achieve improved accuracy and efficiency in their machine learning applications.

This article explores eight futuristic databases to keep an eye on in 2023. It discusses their advanced features and capabilities, highlighting the benefits and challenges of using distributed architectures. The databases mentioned include FaunaDB, PlanetScale, Dolt and Xata, along with others, each offering unique advantages and potential applications for users seeking cutting-edge data management solutions.

TUTORIALS

Delve into parallel task execution using parent and child workflows. This blog explores how this setup can enhance productivity and streamline processes, providing a practical guide on implementing this approach. By leveraging parent and child workflows, users can optimize task execution and improve overall efficiency in their application development workflows.

The databricks team discusses how to build a credit data platform that unifies data processes and people using lakehouse architecture. It highlights the benefits of leveraging the lakehouse model to streamline data operations, enable collaboration and enhance data quality in the credit industry. By adopting this approach, organizations can achieve a unified, scalable and efficient data platform tailored to the unique requirements of credit data management.

TOOLS

MLflow 2.4.2 is a patch release containing the following bug fixes and changes. Need some examples? Here they are:

• Add question-answering and summarization examples and docs with LLMs.
• Add compatibility for legacy transformers serialization.
• Add support for listing artifacts in UC model registry artifact repo.
• Include resources for recipes in mlflow-skinny.

and much more!

Another one release is from AirFlow, an open-source platform for developing, scheduling and monitoring batch-oriented workflows. Some bugs are fixed right now:

• Use linear time regular expressions
• Fix triggers alive check and add a new conf for triggerer heartbeat rate
• Sanitize DagRun.run_id and allow flexibility
• Add the trigger canceled log
• Fix behavior of LazyDictWithCache when import fails

Sweep is an AI junior developer that transforms bug reports & feature requests into code changes.

Describes bugs andsmall features, and refactors as you would to a junior developer and Sweep:

• reads your codebase
• plans the changes
• writes a pull request with code

DATA TUBE

Find out more about the convergence of DataOPS, MLOPS, and LLMs, exploring the growing relevance of data analytics and machine learning applications. Discover how Data and MLOps play a pivotal role in the success of machine learning, relying on a robust data ecosystem. Firat and Sören also showcase real-world examples and highlight the value that generative AI applications can bring to organizations within their data analytics ecosystem.

PODCAST

Let’s listen to a talk with Jerry, CEO and co-founder of LlamaIndex, a framework that empowers developers to tailor LLMs to their specific needs, improve performance and ensure data compliance, ultimately enhancing the use of AI technologies in various applications.

Interview highlights:

• Prototypical uses cases
• How does LlamaIndex integrate an LLM input prompt and provide context with my private data?
• The emerging stack around embedding-based lookups
• LlamaIndex and next-gen AI applications
• Metadata filters
• Revisiting the Modern Data Stack

and much more.

CONFS EVENTS AND MEETUPS

Join this insightful and engaging event about ING's journey to empower business use cases using Generative AI, such as Large Language Models (LLMs).

Why should you join?

• The webinar will provide a technical introduction to Generative AI and LLMs, exploring their capabilities and limitations in different levels of applications;
• You will become familiar with Google Cloud's Generative AI Studio;
• In addition, you will learn how ING is leveraging the power of LLMs to automate the ingestion of multi-language news articles and generate high-level insights for informed decision-making.

Don't miss out on this live, hands-on workshop. Discover the power of the data lakehouse, an innovative data architecture that combines data warehouses and data lakes. Join Databricks and Google to learn how to effectively implement a comprehensive data analytics, engineering and the science life cycle on the lakehouse using Databricks on Google Cloud.

This live, virtual hands-on lab will teach you how to:

• Access all your data — structured, semi-structured, unstructured — with a lakehouse
• Use Databricks SQL to query and visualize data in your lakehouse
• Train models and create predictions with Databricks on Google Cloud
• Track experiments and tune hyperparameters with MLflow
• Deploy and serve models with MLflow and other Google Cloud services