DATA Pill #027 - what the SpaceX rocket has to do with ML models, and CampNou with Data Science

ARTICLES 

Tomás works at FC Barcelona’s Venue Business department which is responsible for selling match day and season tickets to football matches at Spotify Camp Nou.

The goal is to optimize match day revenue and fan experience, maintaining a baseline attendance. In this article Tomás shares:

• how they approach data science challenges at Ticketing
• how the team is organized
• how they prioritize tasks
• real-world examples of both fan-faced and internal projects they have tackled
• who to blame if you don’t manage to buy a ticket before they sell out (just kidding ;))

It's in this article that you'll learn how similar the work on ML models is to the work on SpaceX rocket launches. Of course, this is just a pretext for a more important topic: how to enable monitoring for launched ML Models, and for this purpose we will use Vertex AI.

Vertex AI Model Monitoring can help to detect:

• Training-serving skew — which occurs when the input data distributions after production launch are different from the input data distribution used to train the model. 
• Prediction drift occurs when input data distribution after production launch changes significantly over time. Here Vertex AI does not need to know anything about the training dataset. Instead it collects statistics about the input data from requests for predictions sent during previous monitoring windows.

The Uber platform architecture uses Kafka, Flink and Pinot. The Kafka events generated by backend services, are aggregated by Flink.

Uber noticed a statistically significant boost across all key metrics since it started providing the information on performances to Freight drivers: -0.4% of late cancellations, +0.6% of on-time pick-ups, +1% of on-time drop-off and +1% of auto tracking performances.

In today’s blog, Piotr Chaberski considers the topic of the Machine Learning prototype.

After reading you will realize that it does not have to resemble a mix of spaghetti code.

You can find out how GetInData created a complete blueprint, which gives you a clear example of how to:

• build reproducible, containerized working environments
• track experiments and version our model prototypes
• develop clean, production-quality machine learning code starting from the PoC phase
• transfer our local sample-based implementations into a full-scale cloud environment

As of 2022 it’s estimated that less than 20% of machine learning models are brought into production. Why do so few companies bring ML models to production, and even fewer do so reliably and efficiently? Why do you need data science product teams to do MLOps? 

• Short communication lines between data scientists and engineers.
• Easy to formulate common grounds between all experts.
• Easy to monitor, maintain and iterate on a model once it has been “shipped” to production.

Also,  what are the requirements?

1. The right roles & skills to take ownership over ML products
2. A platform team that can enable the data science product team
3. The right organizational environment to make data science in production a success.

Nice lessons learned about Apache Airflow from Coinbase.

How Coinbase has revamped Apache Airflow-based the orchestration platform to ensure operational efficiency and development velocity. Experience gained in migrating pipelines and onboarding users to the revamped platform via crowdsourcing.

Many people think you need CPU limits on Kubernetes but this isn't true. In most cases, Kubernetes CPU limits do more harm than help.

Natan explains why CPU limits are harmful with three analogies between CPU starved pods and thirsty explorers lost in a desert. In this article, CPU will be water and CPU starvation will be… death.

At last, native integration with GCP. This is quite handy to manage code pipelines in SQL in BigQuery. It could be a useful addition, especially for Data Engineers and Data Analysts who only work with BigQuery.

A quick categorization of all the mentioned tools in this article:

• Infrastructure-as-Code: Pulumi
• Security: SOPS, Trivy
• K8s/multi-cluster: Cluster API, Linkerd
• CI/CD: GitHub Actions, Tekton, HashiCorp Harness
• Monitoring: Thanos
• Policy-as-Code: HashiCorp Sentinel

NEWS

It's great that they are moving forward with this - the combination of Snowpark + Kedro (an open-source Python framework for creating reproducible, maintainable, and modular data science code) will really be a powerful tool for MLOps, both from the perspective of ease of data processing and model training.

OK, fine, we may have missed it a bit, but it's still pretty fresh news. 

​​Datahub got column-level lineage!

PODCAST

Arunabh Singh works as a director of data science at Willa - FinTech that helps professional freelancers, influencers, and social media content creators get paid immediately by brands for their freelance work and paid collaborations. 

Topics of podcast:

• Data that is used at Willa and business use cases that are developed using this data
• The most important ML models implemented at Willa
• The ML(Ops) stack at Willa and a decision to build ML & Analytics capability very early
• The most important skills and competencies that data scientists should have these days
• The main trends and predictions for ML/AI for the next decades
• Plans for 2023 at Willa.

A very interesting podcast episode with Kevin Goldsmith (CTO at Anaconda, Inc.) about the technical and social challenges that data science is facing, plus the impact that open-source software has on this industry. Recorded a year ago, but it's still worth listening to.

The relationship between data-centric AI and knowledge-first AI.

CONFS EVENTS AND MEETUPS

During this meeting Rafal Stryjek, Data Superhero, will demo Snowflake's streaming API, and how to use it to write data directly to Snowflake. There will be dedicated time to Q&A and networking.

A chance to speak in front of an audience of Big Data professionals. 

More than 500 professionals will attend the conference to hear dozens of technical presentations. One of them could be yours ;)

The Snowflake Summit is still a while away, but now it's time for a call to arms. If you want to take part in Vegas: a story of migration, transformation or innovation, you can submit your presentation.