Data Quality in the Databricks Lakehouse
Liebe Hauptstadt-Monster,
wir haben für euch 2 der Besten schlechthin am Start. Mit einem Topaktuellen Thema. Das Kommen lohnt sich also dieses Mal besonders.
ACHTUNG, wir treffen uns dieses Mal bei Ceteris in der Linienstraße, da Berlin Bytes leider nicht verfügbar ist.
Auf euer Kommen und unsere Sprecher freuen sich
Markus & Ben
Vortrag
Data Quality in the Databricks Lakehouse
As modern enterprises adopt the Lakehouse architecture to unify data engineering, analytics, and AI, maintaining high data quality becomes a foundational requirement. In this session, we explore how Databricks enables scalable, automated data quality management using a combination of open-source and native tools.
First, we will uncover Databricks Labs DQX. DQX is a simple validations framework for assesing data quality of PySpark dataframes. The main benefit is as it allows real-time quality validation during data processing rather than post-fact monitoring as Databricks Data Quality Monitoring does. This alllows quicker identification and resolution of data quality problems. As it allows to quarantine invalid data and investigate data quality issues before they actually are written to the table.
Then we will dive into Databricks Data Quality Monitoring. Databricks Data Quality Monitoring is a native feature which allows monitor the statistical properties and quality of the data in all of the tables in your account. Monitoring your data provides quantitative measures that help you track and confirm the quality and consistency of your data over time.
Vitalija Bartusevičiūtė
Vitalija Bartuseviciute is a data professional with a passion for transforming complex data into meaningful, actionable insights. With a PhD rooted in years of scientific research and quantitative analysis, she developed a strong analytical foundation through collecting and interpreting large volumes of biological data. Today, she works as a Senior Consultant and Data Engineer, specializing in Azure Databricks and modern data platforms. She enjoys sharing knowledge, discussing data strategies, and exchanging best practices with fellow professionals. Outside of work, you'll likely find her exploring the world of champagne - because great conversations pair well with great bubbles. 🥂
Alexander Klein
Alexander Klein is a senior Business Intelligence consultant with more than 20 years of experience. He focuses on Business Intelligence and Data Warehouse projects with Microsoft technologies like SQL Server, Power BI, Azure ML or Cognitive Services.
Since 2008, he has been a self-employed consultant in large and medium-sized projects in all sectors across Europe. He has been visiting SQL Saturdays all over Europe since 2013.Since 2017 Alexander speaks at national and international conferences / user groups about Data Warehouse, BI, Azure and AI.
Ort
Zur Gruppe
Markus Raatz
Benjamin Kettner
Die Deutsche Data Plattform Community
Seit 2004.
Monster-Events
| Mo | Di | Mi | Do | Fr | Sa | So |
|---|---|---|---|---|---|---|
|
28
|
29
|
30
|
1
|
2
|
3
|
4
|
|
|
|
|
|
|
|
|
|
5
|
6
|
7
|
8
|
9
|
10
|
11
|
|
|
|
|
|
|
|
|
|
12
|
13
|
14
|
15
|
16
|
17
|
18
|
|
|
|
|
|
|
|
|
|
19
|
20
|
21
|
22
|
23
|
24
|
25
|
|
|
|
|
|
|
|
|
|
26
|
27
|
28
|
29
|
30
|
31
|
1
|
|
|
|
|
|
|
|
|