[We would love to have] A maturity curve of an end-to-end dbt implementation for each version of dbt .... There are so many features in dbt now but it'd be great to understand, "what is the minimum set of dbt features/components that need to go into a base-level dbt implementation?...and then what are the things that are extra credit?"
-Will Weld on dbt Community Slack
One question we hear time and time again is this - what does it look like to progress through the different stages of maturity on a dbt project?
When Will posed this question on Slack, it got me thinking about what it would take to create a framework for dbt project maturity.
Doing analytics is hard. Doing analytics right is even harder.
There are a massive number of factors to consider: Is data missing? How do we make this insight discoverable? Why is my database locked? Are we even asking the right questions?
Compounding this is the fact that analytics can sometimes feel like a lonely pursuit.
Sure, our data is generally proprietary and therefore we can’t talk much about it. But we certainly can share what we’ve learned about working with that data.
So let’s all commit to sharing our hard won knowledge with each other—and in doing so pave the path for the next generations of analytics practitioners.
Airflow and dbt are often framed as either / or:
You either build SQL transformations using Airflow’s SQL database operators (like SnowflakeOperator), or develop them in a dbt project.
You either orchestrate dbt models in Airflow, or you deploy them using dbt Cloud.
In my experience, these are false dichotomies, that sound great as hot takes but don’t really help us do our jobs as data people.
As we get closer to dbt v1.0 shipping in December, it's a perfect time to get your installation up to scratch. dbt 1.0 represents the culmination of over five years of development and refinement to the analytics engineering experience - smoothing off sharp edges, speeding up workflows and enabling whole new classes of work.
Even with all the new shinies on offer, upgrading can be daunting – you rely on dbt to power your analytics workflow and can’t afford to change things just to discover that your daily run doesn’t work anymore. I’ve been there. This is the checklist I wish I had when I owned my last company’s dbt project.