How can I consolidate projects in dbt Cloud?
Consolidating your dbt projects can be an enormous task, and there is no universal solution. But, there are some common approaches to project consolidation in dbt Cloud that you can follow, depending on the scope of the work that needs to be done.
If you have multiple projects that contain production-worthy code, there are rarely straightforward solutions to merging them. Let's suppose you have
Main Project and
Smaller Subset Project.
Files and Folders
Git and the local directory
Reference the merge git commands to help complete the migration plan. Using the commands will help retain git commit history, but you might result in duplicate folders called
tests, etc. You will most likely still have to move files around manually.
Another option would be to use an external code editor (for example, VS Code) to move files from the
Smaller Subset Project to the
Main Project. This is what internal dbt Labs experts recommend to stay informed about what comes over to the main project and also allows you to be more aware of the incoming files, with the ability to make any minor tweaks to folder hierarchy that you might want to do at the same time.
Manual migration with multiple browser tabs
If you only have a couple of models or macros that you want to consolidate, copy the raw file contents from your git provider in
Smaller Subset Project. Then, in the dbt Cloud IDE, paste the contents into a new file in your
Alternatively, you can download those files from your git provider (
Smaller Subset Project repo) and upload them back to your other repository (
Main Project repo). This doesn’t scale well and could bypass change controls, so it might only be a viable solution for organizations with only a few files.
If you have multiple projects with deployment environments deploying jobs, this poses another challenge. Assuming all the models from
Smaller Subset Project can be consolidated into
Main Project, your commands within your jobs will take on a new meaning. In lieu of refactoring your global job strategy at the same time, you can add tags to the incoming project models and utilize that in your job command syntax, with the help of node selection syntax.
Main Project job command example:
dbt build --exclude tag:smaller_subset_project
Smaller Subset Project commands:
dbt build --select tag:smaller_subset_project