Data Quality: Transactions, Ingestions, and Storage
44mAdvanced2025-11-07
Authors
Mark Freeman
Course details
Upskill as a technical data engineer. Explore the essentials of data collection, transactions, ingestion, and storage in this hands-on advanced-level course. Along the way, get tips from instructor, data engineer, and tech lead Mark Freeman as he shows you how to identify and fix common data quality issues.
Skills covered
Cloud StorageData Resource ManagementData EngineeringDatabase ManagementCloud ComputingData ScienceOne-Off
Concepts
Introduction
- Introduction
Data Ingestion
- Initial ingestion into PostgreSQL
- Ingesting dirty data
- Implementing CHECK constraints
- Ingestion code exploration
Transactions
- Database transaction pitfalls
- Simulating the lost update problem
- Preventing concurrency issues with isolation
- Transaction code exploration
Replication
- A brief overview of data lakehouses
- Intentionally breaking our replication pipeline
- Introduction to the Write-Audit-Publish (WAP) pattern
- Implementing the Write-Audit-Publish (WAP) pattern
- Replication code exploration
Data Quality Product Requirement Document
- Final project - Writing a product requirement document
- Final project - Overview
Conclusion
- Next steps