One thing that saved me *so* much time—data validation checks at each stage of the ETL pipeline.
Like, before loading, I run a quick check to see if the schema matches or if there are nulls where they shouldn’t be. Tools like Great Expectations or dbt tests are clutch for this.
Also, if your pipeline’s in Python, pdb or ipdb for stepping through transforms is a lifesaver. No more guessing where it broke!
---
Like, before loading, I run a quick check to see if the schema matches or if there are nulls where they shouldn’t be. Tools like Great Expectations or dbt tests are clutch for this.
Also, if your pipeline’s in Python, pdb or ipdb for stepping through transforms is a lifesaver. No more guessing where it broke!
---