More generally, what we are solving for here is the case of ad-hoc analysis versus creating new data. When you are performing analysis you care about the insight, but less so about how you got there – when you create data the output is the dataset itself, so the process is key.
In other words, that I like more 😀, it is data exploration versus asset creation. When you are performing ad-hoc analysis you are exploring the data, when you are building dbt models, or similar, you are creating assets – things that (in theory) will live forever in the data layer and be used by others downstream.
Therefore when you build a dbt model, you should aim for a slower process, a la software engineering, of testing, reviewing, versioning and the like. When it comes to asset creation we want to run a tight ship, make sure the outputs are kosher and keep the processes strict and scaleable.
To me, the test when building a dbt model should be to ask yourself: would I be happy to have someone else use this in its current state? Because once you put something in a pipeline, and from there into the data layer, you should expect whoever to pick it up and start using it for whatever mad Frankenstein application downstream that you never even imagined. So it needs to be versioned, well-tested, reviewed, maintained and everything else that you would expect of a critical piece of software.
This split between exploration and assets also extend to dashboards (and data apps) themselves. Some dashboards are assets – they are widely shared and used, and a critical part of the business – others are only used for a specific piece of analysis and as a means to share the analysis with a small group of stakeholders.
