DATA Pill #04 - Kubernetes Cluster scaling, Training ML Models and 1 clickbait

ARTICLES 

Ok, the title is clickbait, no need to change the job, but it still has some valid points. 

Like:

Data for the middle class - if Big Data is commonly adopted, the tools must be simpler and focus on basic problems. If everyone wanted to use Big Data for complex stuff, there wouldn't be enough qualified people who would like to do such a job. 

The process of training, validating and deploying a machine learning model using BigQuery ML (BQML) explained. The example model predicts the probability of adding an item to the shopping cart on an e-commerce website. GetInData used the Google Analytics 4 export to BigQuery to train a logistic regression classifier and then predicted the probability of a sample event (addedToCart) on the new sessions.

Over the past few years, Airbnb has shifted almost all online services from manually orchestrated EC2 instances to Kubernetes. Today, it runs thousands of nodes across nearly a hundred clusters to accommodate these workloads. In the blog post there are three stages of setting up a Kubernetes cluster. 

Stage 1: Homogenous Clusters, Manual Scaling

Stage 2: Multiple Cluster Types, Independently Autoscaled

Stage 3: Heterogeneous Clusters, Autoscaled

Discover how Lyft built a robust system for identifying and preventing model degradation and the variety of model monitoring approaches they developed. Plus conclusions:

• Automate as much as possible. An even better solution to provide a seamless onboarding experience is to remove it entirely.
• Monitoring will reduce shipping velocity slightly. By enforcing monitoring for all models, there is an additional prerequisite in the productionzation of models. 

This post focuses on the metrics that are concentrated on expanding on these metrics and provides more insights into identifying resource bottlenecks and sources of errors.

You will also find some details about the Grafana dashboard.

NEWS

Another option to run Spark serverless (and many more Big Data tools).

PODCAST

Peeyush Agarwal explains 2 key pieces of the ML infrastructure at Chime. He goes into detail about the current feature store design and feature monitoring process along with the ML monitoring setup.

SVP of engineering Shireesh Thota describes the impact on overall system architecture that Singlestore can have and the benefits of using a cloud-native database engine.

CONFS AND MEETUPS

What’s gonna happen?

• Keynote by Fiona Coath, who will talk about opposition to surveillance capitalism and our responsibilities as technologists.
• Nick Burch, who will explore what Wordle can teach us about Information Retrieval, Search and AI/ML.
• Anshum Gupta will tell us what's new in Apache Solr 9.0

Conferences focus areas:

• Machine Learning & Deep Learning
• Machine Learning for finance
• NLP
• Data Engineering & MLOps

Reach agenda of Machine Learning, Data Mining & Exploration