DATA Pill #077 - Snowflake + Snowpark + Streamlit + Vanna AI, How to reduced docker build times by 40%

ARTICLES

The journey toward a Zero-touch network parallels the evolution from cars to self-driving vehicles. In the context of 5G complexities, automating Radio Access Networks (RANs) becomes vital. Intent-based Automation in RANs is pivotal in refining operations, boosting network performance, and enabling innovative services. This approach is a pioneer for achieving a Zero-touch network, revolutionizing network management and introducing new service offerings.

LinkedIn relies heavily on Apache Beam to process over 4 trillion events daily through thousands of pipelines, enabling real-time data processing across crucial services for their 950 million members. The case study details how Apache Beam's unified framework has significantly impacted streaming processing, reducing latency, optimizing costs, enabling real-time ML feature generation, improving user experiences through personalized services and improving abuse detection.

In this blog, you’ll get to discover some of the problems, challenges and solutions facing the field of AI/ML-driven business analytics, focused on what is at its core—data. You’ll walk away with a better understanding of what you’re up against when it comes to doing real-time business analytics with AI/ML, optimizing your data infrastructure, and preparing analyst teams and technologies to use the highest quality data for the job.

This blog post delves into strategies for accelerating CI pipelines and improving Docker build times. It showcases two impactful changes: storing build cache information remotely and utilizing the link option while adding and copying files into Docker images, ultimately leading to a 40% reduction in overall build times.

TUTORIALS

Discover how Jacco developed an 80-line Python code to automate the process, providing comprehensive insights into services, actions and more for policy validation and potential tool development.

This tutorial will show you how to use Snowpark Stored Procedures to let you chat with your Snowflake data using Snowflake’s new built-in Streamlit integration.

Read an overview of orchestrating a data pipeline by employing Snowflake operators within an Amazon MWAA environment. It outlines the necessary steps to establish integration between Amazon MWAA and Snowflake. The solution presents an automated, comprehensive workflow, encompassing data ingestion, transformation, analytics and consumption.

NEWS

Databricks Lakehouse Monitoring allows teams to oversee their entire data pipelines, from data and features to ML models, all without needing extra tools. Using Unity Catalog, it ensures data and AI assets are high-quality, accurate and dependable by providing detailed insights into their lineage.

TOOLS

K9s is a terminal based UI designed to interact with your Kubernetes clusters. The aim of this project is to make it easier to navigate, observe and manage your deployed applications in the wild. K9s continually watches Kubernetes for changes and offers subsequent commands to interact with your observed resources.

Kor is a tool to discover unused Kubernetes resources. Currently, Kor can identify and list unused:

• ConfigMaps
• Secrets
• Services
• ServiceAccounts
• Deployments

and more.

DATA TUBE

Learn about BigQuery's built-in machine learning capabilities and how to use Vertex AI's LLMs directly within BigQuery. You'll see how these tools make tasks like sentiment analysis and entity extraction easier. Plus, there'll be a live demo showing how Faraday uses these powerful models in BigQuery to predict customer behavior.

PODCAST

Topics that we talk about include:

• What Pleo is, who uses it and how it works
• How data is collected and utilized by Pleo
• Pleo's data tech stack, Including GCP, BigQuery, Kafka, Metabase and Looker
• Pleo's approach to Generative AI
• Differences in working with data at Pleo, Acast, and Skyscanner

and more…

CONFS EVENTS AND MEETUPS

We invite you to go through the process of speaking proposal submission – you will find the necessary details in the link. The 10th edition of the Big Data Technology Warsaw Summit takes place on 10th-11th April 2024. More than 500 professionals will attend the conference to listen to technical presentations given by dozens of speakers from top data-driven companies.