Automating Data Quality in Dev Environments
1h 4mAdvanced2024-08-07
Authors

Lauren Maffeo
Course details
Data quality is the backbone of successful AI, yet most leaders lack quality standards they can automate in production. This course teaches you how to create quality standards for the data in your domains, then automate those standards in production environments.
Most organizations produce and ingest more data than they can effectively manage, with insufficient standards to measure quality. As leaders face increasing pressure to leverage AI, companies that don't adopt and implement better standards will fall behind. Instructor Lauren Maffeo explains how to define data quality standards per domain, who should set these quality standards, which tools you should use to scale and automate these standards, and how to ensure that any new data is measured against these standards. Gain an understanding of the people, processes, and tools needed to know what data quality looks like and integrate those standards into your data architecture.
Most organizations produce and ingest more data than they can effectively manage, with insufficient standards to measure quality. As leaders face increasing pressure to leverage AI, companies that don't adopt and implement better standards will fall behind. Instructor Lauren Maffeo explains how to define data quality standards per domain, who should set these quality standards, which tools you should use to scale and automate these standards, and how to ensure that any new data is measured against these standards. Gain an understanding of the people, processes, and tools needed to know what data quality looks like and integrate those standards into your data architecture.
Skills covered
Data GovernanceData Resource ManagementData EngineeringDatabase ManagementData ScienceOne-Off
Concepts
0. Introduction
- 01 - Why data quality is crucial
1. Write Your Roadmap for Data QA
- 02 - Manage your data as a product
- 03 - Choose a high-priority project
- 04 - Do a data audit
- 05 - Create a current-state process map
- 06 - Define data quality
- 07 - Write a roadmap for data product delivery
2. Practice Governance-Driven Development
- 08 - Write data requirements for your roadmap
- 09 - Confirm your data's source system(s)
- 10 - Establish the right data system integrations
- 11 - Define your source data's minimum acceptance criteria (MAC)
- 12 - Set up data lineage tracking
- 13 - Define levels of access per user
- 14 - Draft a to-be process map
- 15 - Define areas of data transformation
- 16 - Choose some super users to validate your product
- 17 - Give your data team room to fail
3. Monitor Data in Production
- 18 - Make a plan to govern data throughout the full lifecycle
- 19 - Practice data mesh design principles
- 20 - Automate federated data QA standards
- 21 - Execute data security standards
- 22 - Make a traceability matrix
- 23 - Scale and automate your data QA standards
- 24 - Use feature stores to prevent data drift
- 25 - Ship new data as deployable units
- 26 - Track ongoing regulation changes
Conclusion
- 27 - Continuing your learning journey in the data quality