DATA Pill #018 - Data Mesh everywhere, dynamic pricing in a real-time app and shocking news

ARTICLES 

In 2020, Capital One moved the entire data workload to the cloud. That came with a challenge in managing the exponential growth in data that was previously stored in different places and meeting the demands of users with a variety of use cases. They had to change the complicated Data Ecosystem which was hard to scale. In the article, they described how they managed it by federated data management and a centralized tooling and policy. 

Takeaway from Jacinta Waak, Filip Allard and Hannes Kindbom (Hedvig) presentation.

Key insights:

• That was not the first time we observed that companies consider replacing Fivetran with Airbyte to be more cost-effective.
• The ML world is still separate, next step from the Data world, but can be well integrated. ML on Vertex AI steps in after data is being ingested with Fivetran and modeled with dbt.

Technology to avoid high fines for GDPR issues - how to comply with regulations while also maintaining data integrity. Technology developed by Robert Sahlin, the Data Engineering Lead at Mathem.

They can simply delete the row for the member in the operational token vault, and then that person can’t be re-identified. They also have the option of deleting certain fields if only parts of the data needs to be deleted.

A couple of interesting takeaways from one of the recent data conferences by Prukalpa:

• Concept of MVD (minimum viable dataset) - the shift from big data to small data, going on a "data diet", avoiding data FOMO and stopping ingesting everything
• The replacement of real by synthetic data for AI - addressing weaknesses of real data (quality, privacy, etc.)
• Active metadata - ferrying metadata back and forth among all the tools in the stack

PayPal is building a Data Mesh, the next generation of data platforms. This post details the evolution of data platforms, highlights their problems and why they decided to build a Data Mesh.

This article reveals the architecture and engineering choices behind the various components that Safe Deploys comprise.

Designing a near real-time experimentation system required making explicit tradeoffs among speed, precision, cost, and resiliency. An early decision was to limit near real-time results to only the first 24 hours of an experiment — enough time to catch any major issues and transition to using comprehensive results from the batch pipeline. The idea being once batch results were available, experimenters would no longer need real time results.

Data Warehouse, Data Lake and now Data Lakehouse and Data Mesh, what is what and where are the differences, especially the question of how do they relate to each other?

The Data Mesh Operability Pattern helps us understand the operating characteristics of an enterprise Data Mesh. 

• How does this pattern work?
• Why is this so important for the stability, resilience, and performance of enterprise Data Mesh?

NEWS

Lightbend is changing Akka’s licensing to “Business Source License (BSL)”. 

A stab in the back for many projects.

Discussion on GitHub: https://github.com/akka/akka/pull/31561#issuecomment-1239473395

PODCAST

Topic:

• Alessandro's journey to FREE NOW
• Techniques, signals, and KPIs used to develop the dynamic pricing ML model for a real-time mobile app
• Working with stakeholders to understand the changing priorities to adapt and optimize ML models
• The technology stack used by data scientists and ML engineers at FREE NOW

If you find this interesting, check out the Data Mass Gdańsk Summit (September 30th), where Alessandro will be a speaker. Also, the agenda is worth checking.

If someone prefers (quite long) interviews, here is a high-quality interview about Data Mesh - with Zhamak Dehghani who defined Data Mesh while working for Thoughworks.

CONFS AND MEETUPS

• Practical application of language models in the legal domain 
• AI Economics & delivering production level machine learning components with ModelOps +Q&A

Gamification is on it's way to google certification. Registration for the event wili give you free access to Cloud Skills Boost for 3 calendar months.