DATA Pill #046 - Is the Data Engineer dead? And how Fivetran + dbt fail?

ARTICLES

Do you remember this saga that was mentioned in DATA Pill #23? The last article argues that Fivetran's business practices are part of a more significant trend in the tech industry of prioritizing growth and revenue over long-term customer relationships and sustainable business practices.

Let’s discuss the opportunities, risks, and precautions associated with generative AI in the banking industry. Some possibilities include improving customer experience, detecting fraud, and automating repetitive tasks. On the other hand, risks include data security and privacy concerns, biased algorithms, and potential job displacement. To mitigate these risks, Jesper suggests implementing ethical AI principles, ensuring transparency and explainability of AI models, and involving multiple stakeholders in the decision-making process.

Robert discusses the evolving role of data engineers and how the traditional responsibilities of a data engineer have shifted. He argues that the traditional data engineer role has become outdated with the advancements in technology and the increasing need for data-driven decision-making. In his opinion, data engineers should focus on developing and implementing scalable data architectures while also possessing a strong understanding of business needs and a wider range of technical skills. He also touches on the importance of collaboration between data engineers and data scientists.

TUTORIAL

A quick tutorial on how to replace AirFlow. It is time to introduce you to Mage. Mage is an open-source data pipeline tool for transforming and integrating data.

What does it give you?

• Integrate and synchronize data from 3rd party sources.
• Build real-time and batch pipelines using Python, SQL, and R to transform data.
• Run, monitor, and orchestrate thousands of pipelines without losing sleep.

DATA LIBRARY

Data leaders should measure five pillars of data quality: freshness, volume, schema, (lack of) anomalies, and distribution.

Obtaining a higher degree of Data Observability can help improve these five pillars of data quality, but not all Data Observability tooling is created equal.

We distinguish between “Shallow” Data Observability and “Deep” Data Observability, and data leaders should aim for the latter in order to fully measure the five pillars of data quality and to get full confidence in their data.

Deep Data Observability is different from Shallow, because it is fully comprehensive in terms of data sources, data formats, data granularity, validator configuration, cadence, and user focus.

NEWS

Tired of people telling you that you cannot build a data warehouse on top of Excel? Look no further! dbt-excel seamlessly integrates Excel into dbt, so you can take advantage of the dbt's rigor and Excel's flexibility. MUST WATCH about this release here.

Flink announces Apache Flink 1.17. It had 172 contributors enthusiastically participating and saw the completion of 7 FLIPs and 600+ issues, bringing many exciting new features and improvements to the community. What’s new about that?

For batch processing:

• Streaming Warehouse API
• Batch Execution Improvements
• SQL Client/Gateway

For stream processing:

• Streaming SQL Semantics
• Checkpoint Improvements
• Watermark Alignment Enhancement
• StateBackend Upgrade

Google will make your work in Google Docs, Gmail, Sheets, and Slides easier with the help of AI. For example, Google says will be coming to Workspace apps in the future:

• Draft, reply, summarize, and prioritize your Gmail
• Brainstorm, proofread, write, and rewrite in Docs
• Bring your creative vision to life with auto-generated images, audio, and video in Slides
• Go from raw data to insights and analysis via auto-completion, formula generation, and contextual categorization in Sheets
• …and a way more!

Firstly Google announced an AI bot in their suite, now Microsoft. Revolution to our work is coming. Copilot is a modern AI assistant that will help Microsoft 365 users create Office documents. It allows Office users to summon it to generate text in documents, create PowerPoint presentations based on Word documents, or even help use features like PivotTables in Excel.

DATA TUBE

Listen to talk with Dave, who has been an active member of the Kafka community for many years and noticed that there were a lot of Kafka resources for Java but few for Python. Kris and Dave discuss all things Kafka and Python: the libraries, the tools, pros & cons. What’s more?

• What are the Python clients for Apache Kafka?
• What are the use cases for using Python with Kafka?
• Tips for getting started in Python with Kafka
• Are there options similar to Kafka Streams for Python?

PODCASTS

Liudmyla is the Head of Data Science at Metadata.io, a company building the first Marketing OS for B2B.
Mentioned topics:

• Data sources collected and used by Metadata.io
• Data-driven features and product analytics at Metadata.io
• ML algorithms at Metadata.io e.g. Neural Networks
• Tech stack used at Metadata.io e.g. AWS, Databricks, (Py)Spark, MLflow

Let’s listen to talk with Sebastian, who developed and grew a successful Data Product Management team at Vista. He explains what that process was like and what he learned. Sebastian shares valuable insights on how he implemented a data product orientation at Vista, what makes a good data product manager, and why technology usage isn’t the only metric that matters when measuring success. He also shares what he would do differently if he had to do it all over again.

CONFS EVENTS AND MEETUPS

The ultimate conference for data and AI leaders, innovators and enthusiasts. Get ready for an immersive experience with industry experts and thought leaders. Don’t miss out on this exciting opportunity to lead the way in data and AI innovation.

Learn from leading global organizations on lakehouse best practices, and how the platform has become the destination for businesses of every size, in every industry and region around the world. There will also be opportunities to participate in technical quickstart training, product walkthroughs, and in-depth customer breakout sessions.