While working to centralise all of their data in the cloud, the Virgin Media O2 team had a problem with providing a standardised, streamlined way for stakeholders to access this data in real-time. Now they are working on a data ingestion framework that unifies operational and analytical pipelines while handling the real-time events and batch ingestion method. In this article you will find the architecture of this solution and a description of the analytical pipeline.
An interesting read on some of the lessons learned with dbt - especially regarding the conscious management of technical debt. It seems like it's not too hard to make a mess when scaling up...
At LinkedIn, the model for predicting CTR was changed for a model with a deep learning-based system. Here’s how it was done, the architecture behind it and how it brought large relevance lifts (+8.5% CTR). Lessons shared with the LinkedIn team:
Madison shares how she set up the first Airbyte connector, ingesting raw data from Mailchimp into Snowflake. A few observations:
A few takeaways:
Amazon is announcing that now its time-series machine learning based forecasting service Amazon Forecast, can run what-if assessments to determine how different business scenarios can affect demand estimates. (...) It can be easily imported into common business and supply chain applications, such as the SAP and Oracle Supply Chain. This makes it easy to integrate more accurate forecasting into the existing business processes with little to no change.
Facebook has released Velox: an abstracted, unified, implemented in C++ (FTW!) execution engine that accelerates the query execution of Presto, Spark, PyTorch and possibly other systems. They have just made the general announcement, but the system has been public since Dec 2021.
So, on 26 August, Databricks announced the General Availability (GA) of Delta Sharing on AWS and Azure.
By announcing this release, Databricks proclaims the highest level of stability, support and enterprise readiness from Databricks for mission-critical workloads on the Databricks Lakehouse Platform.
When you create a feature table with the Databricks Feature Store, the data sources used to create the feature table are saved and accessible. For each feature in a feature table, you can also access the models, notebooks, jobs and endpoints that use the feature.
Snowflake Data Cloud gives you the option of deploying Lakehouse architecture, along with Data Mesh and other popular architecture. Lakehouse architecture is a subset of the Snowflake Data Cloud.
Have you discovered the hottie of the month yet? With us it's a hit!
Animate Stable Diffusion by interpolating between two prompts.
This is a bit off topic, but I think the results are just amazing and worth sharing in the community.
As the video author says: “remember that science is a process. Don’t think about what was achieved in this paper, think about what the results will be a few papers down the line!”
A "backstage peek" at Backstage, an open platform for building developer portals. A great insight.