Automating Data Quality in Dev Environments

Automating Data Quality in Dev Environments

1h 4mAdvanced2024-08-07

Authors

Lauren Maffeo

Lauren Maffeo

Course details

Data quality is the backbone of successful AI, yet most leaders lack quality standards they can automate in production. This course teaches you how to create quality standards for the data in your domains, then automate those standards in production environments.

Most organizations produce and ingest more data than they can effectively manage, with insufficient standards to measure quality. As leaders face increasing pressure to leverage AI, companies that don't adopt and implement better standards will fall behind. Instructor Lauren Maffeo explains how to define data quality standards per domain, who should set these quality standards, which tools you should use to scale and automate these standards, and how to ensure that any new data is measured against these standards. Gain an understanding of the people, processes, and tools needed to know what data quality looks like and integrate those standards into your data architecture.

Skills covered

Data GovernanceData Resource ManagementData EngineeringDatabase ManagementData ScienceOne-Off

Concepts

Introduction

  • Why data quality is crucial

Write Your Roadmap for Data QA

  • Manage your data as a product
  • Choose a high-priority project
  • Do a data audit
  • Create a current-state process map
  • Define data quality
  • Write a roadmap for data product delivery

Practice Governance-Driven Development

  • Write data requirements for your roadmap
  • Confirm your data's source system(s)
  • Establish the right data system integrations
  • Define your source data's minimum acceptance criteria (MAC)
  • Set up data lineage tracking
  • Define levels of access per user
  • Draft a to-be process map
  • Define areas of data transformation
  • Choose some super users to validate your product
  • Give your data team room to fail

Monitor Data in Production

  • Make a plan to govern data throughout the full lifecycle
  • Practice data mesh design principles
  • Automate federated data QA standards
  • Execute data security standards
  • Make a traceability matrix
  • Scale and automate your data QA standards
  • Use feature stores to prevent data drift
  • Ship new data as deployable units
  • Track ongoing regulation changes

Conclusion

  • Continuing your learning journey in the data quality
40,000 Toman