DATA Pill #042 - Reduce Amazon EMR cluster costs by up to 19%, Apache Flink Kubernetes Operator and more

ARTICLES

It is not common to talk about issues, but the Uber team in this blog post did it well. Many data issues are manually detected by users, weeks or even months after they start. Data regressions are hard to catch because the most impactful ones are generally silent. Check out the example, impact, detection strategies and way more.

Read about a Preset feature that allows the Plum-living team to define the data & metrics they want to expose to their users “as code” with dbt, without being tied to a closed solution or language specific to the visualization tool.

An overview of the key enhancement AWS team launched in EMR Managed Scaling. They observed that the cluster utilization improved by up to 15 percent, and cluster cost was reduced by up to 19 percent. Starting mid-December 2022, these enhancements were enabled by default for EMR clusters using Amazon EMR versions 5.34.0 and later, and Amazon EMR 6.4.0..

This one sounds delicious, but what does it have in common with hexagonal architecture? Hexagonal architecture is an architectural pattern for building software. Jeroen explains what hexagonal architecture is all about and what it has to do with cooking ravioli. Read how to transform ‘spaghetti code’ into highly cohesive, loosely coupled and highly testable code using the principles of hexagonal architecture and domain driven design.

This article discusses the challenges of using numeric data for machine learning in various sectors such as finance, medicine and social media. Although numeric data may seem ready for use, additional steps such as normalization and outlier removal are often necessary to obtain accurate results. However, computing statistics for the entire dataset can be difficult and time-consuming, especially for big data streams. Additionally, new data may not fit with previously computed statistics, which can impact model performance.

Onion Architecture is a software architectural style which promotes the separation of concerns between the most important part of a business application — the domain code — and its technical aspects like HTTP or the database. It does so with ideas similar to Hexagonal Architecture, Clean Architecture and other related architecture styles.

It can solve numerous of issues like:

• A more structured, layered layout of the code makes code navigation easier and makes the relationship between different parts of the codebase more visible at first glance
• Loose coupling between the domain and the infrastructure
• Coupling is towards the center of The Onion — expressed by the relationship between the layers

The Netflix media-focused ML infrastructure goal is to reduce the time from ideation to productization for media ML practitioners. They accomplish this by paving the path to:

• Accessing and processing media data (e.g. video, image, audio and text)
• Training large-scale models efficiently
• Productizing models in a self-serve fashion in order to execute on existing and newly arriving assets
• Storing and serving model outputs for consumption in promotional content creation

In this post you will find:

• challenges of applying machine learning to media assets
• the infrastructure components
• case study of using these components in order to optimize, scale and solidify an existing pipeline.

TUTORIAL

This post was made to compare the various existing streaming tools and get a feeling for their pros and cons. Let’s explore not only an analytical but also an operational (very low latency) data integration, which consists of the following steps in order to create a real-time core streaming data model:

• read (one or more) topic(s) from the streaming ledger
• arbitrary (stateful) computation
• (if necessary) write the results back into the streaming ledger (so that other operational analytical processes can subscribe and react to events)

Improving the experience of coding in Ruby is highly aligned with its goal of making the developer happy—and the future of Ruby tooling is exciting. If you want to know how the Shopify Team built the Ruby LSP, the features included within it and how you can install it, then your search is over.

In this one, firstly you will find the answer to the question: why may you need another distributed framework on top of Spark? There is also a simple introduction to Ray Architecture, and finally starting that on a Databricks Cluster.

NEWS

Apache has announced the latest stable release of the operator. In addition to the expected stability improvements and fixes, the 1.4.0 release introduces the first version of the long-awaited autoscaler module.

PODCAST

In this episode, Jean shares his experience on the application of data mesh, how to implement it in an organization and the combination of technical and organizational challenges that he encountered in the process. Some of the questions that were asked in this episode:

• What are the core problems that he addressed in this project?
• Is a data mesh ever "done"?
• What are the most interesting, unexpected, or challenging lessons that he learned while working on data mesh?

CONFS EVENTS AND MEETUPS

• You'll get to hear from some of the top experts in the field, with speakers from companies like Meta, Google, Airbnb and Xebia
• You'll learn about the latest trends and technologies in big data, AI, data engineering, real-time streaming, cloud, data science, MLOps and how they can be applied to real-world problems.
• You'll have the opportunity to network with other professionals and make valuable connections in the industry during discussions in 20 round tables.