rpc

Overview

The dbt rpc command runs a Remote Procedure Call dbt Server. This server can compile and run queries in the context of a dbt project. Additionally, it provides methods that can be used to list and terminate running processes. The rpc server should be run from a directory which contains a dbt project. The server will compile the project into memory, then accept requests to operate against that project's dbt context.

Running on Windows

The rpc server is not supported on Windows, due to historic reliability issues. A docker container may be a useful workaround if required.

Running the server:

$ dbt rpc
Running with dbt=0.15.0


16:34:31 | Concurrency: 8 threads (target='dev')
16:34:31 |
16:34:31 | Done.
Serving RPC server at 0.0.0.0:8580
Send requests to http://localhost:8580/jsonrpc

Configuring the server

  • --host: Specify the host to listen on (default=0.0.0.0)
  • --port: Specify the port to listen on (default=8580)

Submitting queries to the server: The rpc server expects requests in the following format:

rpc-spec.json
{
    "jsonrpc": "2.0",
    "method": "{ a valid rpc server command }",
    "id": "{ a unique identifier for this query }",
    "params": {
        "timeout": { timeout for the query in seconds, optional },
    }
}

Built-in Methods

status

The status method will return the status of the rpc server. This method response includes a high-level, like ready, compiling, or error, as well the set of logs that accumulated during the initial compilation of the project. When the rpc server is in the compiling or error state, all non-builtin methods of the RPC server will be rejected.

Example request

{
    "jsonrpc": "2.0",
    "method": "status",
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d"
}

Example response

{
    "result": {
        "status": "ready",
        "error": null,
        "logs": [..],
        "timestamp": "2019-10-07T16:30:09.875534Z",
        "pid": 76715
    },
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "jsonrpc": "2.0"
}

poll

The poll endpoint will return the status, logs, and results (if available) for a running or completed task. The poll method requires a request_token parameter which indicates the task to poll a response for. The request_token is returned in the response of dbt tasks like compile, run and test.

Parameters:

  • request_token: The token to poll responses for
  • logs: A boolean flag indicating if logs should be returned in the response (default=false)
  • logs_start: The zero-indexed log line to fetch logs from (default=0)

Example request

{
    "jsonrpc": "2.0",
    "method": "poll",
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "params": {
        "request_token": "f86926fa-6535-4891-8d24-2cfc65d2a347",
        "logs": true,
        "logs_start": 0
    }
}

Example Response

{
    "result": {
        "results": [],
        "generated_at": "2019-10-11T18:25:22.477203Z",
        "elapsed_time": 0.8381369113922119,
        "logs": [],
        "tags": {
            "command": "run --models my_model",
            "branch": "abc123"
        },
        "status": "success"
    },
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "jsonrpc": "2.0"
}

ps

The ps methods lists running and completed processes executed by the RPC server.

Parameters

  • completed: If true, also return completed tasks (default=false)

Example request:

{
    "jsonrpc": "2.0",
    "method": "ps",
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "params": {
        "completed": true
    }
}

Example response:

{
    "result": {
        "rows": [
            {
                "task_id": "561d4a02-18a9-40d1-9f01-cd875c3ec56d",
                "request_id": "3db9a2fe-9a39-41ef-828c-25e04dd6b07d",
                "request_source": "127.0.0.1",
                "method": "run",
                "state": "success",
                "start": "2019-10-07T17:09:49.865976Z",
                "end": null,
                "elapsed": 1.107261,
                "timeout": null,
                "tags": {
                    "command": "run --models my_model",
                    "branch": "feature/add-models"
                }
            }
        ]
    },
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "jsonrpc": "2.0"
}

kill

The kill method will terminate a running task. You can find a task_id for a running task either in the original response which invoked that task, or in the results of the ps method.

Example request

{
    "jsonrpc": "2.0",
    "method": "kill",
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "params": {
        "task_id": "{ the task id to terminate }"
    }
}

Running dbt projects

The following methods make it possible to run dbt projects via the RPC server.

Common parameters

All RPC requests accept the following parameters in addition to the parameters listed:

  • timeout: The max amount of time to wait before cancelling the request.
  • task_tags: Arbitrary key/value pairs to attach to this task. These tags will be returned in the output of the poll and ps methods (optional).

Running a task with CLI syntax

Parameters:

  • cli: A dbt command (eg. run --models abc+ --exclude +def) to run (required)
{
    "jsonrpc": "2.0",
    "method": "cli_args",
    "id": "<request id>",
    "params": {
        "cli": "run --models abc+ --exclude +def",
        "task_tags": {
            "branch": "feature/my-branch",
            "commit": "c0ff33b01"
        }
    }
}

Several of the following request types accept these additional parameters:

  • threads: The number of threads to use when compiling (optional)
  • models: The space-delimited set of models to compile, run, or test (optional)
  • select: The space-delimeted set of resources to seed or snapshot (optional)
  • selector: The name of a predefined YAML selector that defines the set of resources to execute (optional)
  • exclude: The space-delimited set of resources to exclude from compiling, running, testing, seeding, or snapshotting (optional)
  • state: The filepath of artifacts to use when establishing state (optional)

Compile a project (docs)

{
    "jsonrpc": "2.0",
    "method": "compile",
    "id": "<request id>",
    "params": {
            "threads": "<int> (optional)",
            "models": "<str> (optional)",
            "exclude": "<str> (optional)",
            "selector": "<str> (optional)",
            "state": "<str> (optional)"
        }
}

Run models (docs)

Additional parameters:

  • defer: Whether to defer references to upstream, unselected resources (optional, requires state)
{
    "jsonrpc": "2.0",
    "method": "run",
    "id": "<request id>",
    "params": {
            "threads": "<int> (optional)",
            "models": "<str> (optional)",
            "exclude": "<str> (optional)",
            "selector": "<str> (optional)",
            "state": "<str> (optional)",
            "defer": "<bool> (optional)"
        }
}

Run tests (docs)

Additional parameters:

  • data: If True, run data tests (optional, default=true)
  • schema: If True, run schema tests (optional, default=true)
{
    "jsonrpc": "2.0",
    "method": "test",
    "id": "<request id>",
    "params": {
            "threads": "<int> (optional)",
            "models": "<str> (optional)",
            "exclude": "<str> (optional)",
            "selector": "<str> (optional)",
            "state": "<str> (optional)",
            "data": "<bool> (optional)",
            "schema": "<bool> (optional)"
        }
}

Run seeds (docs)

Parameters:

  • show: If True, show a sample of the seeded data in the response (optional, default=false)
{
    "jsonrpc": "2.0",
    "method": "seed",
    "id": "<request id>",
    "params": {
            "threads": "<int> (optional)",
            "select": "<str> (optional)",
            "exclude": "<str> (optional)",
            "selector": "<str> (optional)",
            "show": "<bool> (optional)",
            "state": "<str> (optional)"
        }
}

Run snapshots (docs)

{
    "jsonrpc": "2.0",
    "method": "snapshot",
    "id": "<request id>",
    "params": {
            "threads": "<int> (optional)",
            "select": "<str> (optional)",
            "exclude": "<str> (optional)",
            "selector": "<str> (optional)",
            "state": "<str> (optional)"
        }
}

Generate docs (docs)

Additional parameters:

  • compile: If True, compile the project before generating a catalog (optional, default=false)
{
    "jsonrpc": "2.0",
    "method": "docs.generate",
    "id": "<request id>",
    "params": {
            "compile": "<bool> (optional)",
            "state": "<str> (optional)"
        }
}

Compiling and running SQL statements

Compiling a query

This query compiles the sql select {{ 1 + 1 }} as id (base64-encoded) against the rpc server:

rpc-spec.json
{
    "jsonrpc": "2.0",
    "method": "compile_sql",
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "params": {
        "timeout": 60,
        "sql": "c2VsZWN0IHt7IDEgKyAxIH19IGFzIGlk",
        "name": "my_first_query"
    }
}

The resulting response will include a key called compiled_sql with a value of 'select 2'.

Executing a query

This query executes the sql select {{ 1 + 1 }} as id (bas64-encoded) against the rpc server:

rpc-run.json
{
    "jsonrpc": "2.0",
    "method": "run_sql",
    "id": "2db9a2fe-9a39-41ef-828c-25e04dd6b07d",
    "params": {
        "timeout": 60,
        "sql": "c2VsZWN0IHt7IDEgKyAxIH19IGFzIGlk",
        "name": "my_first_query"
    }
}

The resulting response will include a key called table with a value of {'column_names': ['?column?'], 'rows': [[2.0]]}

Reloading the Server

When the dbt Server starts, it will load the dbt project into memory using the files present on disk at startup. If the files in the dbt project should change (either during development or in a deployment), the dbt Server can be updated live without cycling the server process. To reload the files present on disk, send a "hangup" signal to the running server process using the Process ID (pid) of the running process.

Finding the server PID

To find the server PID, either fetch the .result.pid value from the status method response on the server, or use ps:

# Find the server PID using `ps`:
ps aux | grep 'dbt rpc' | grep -v grep

After finding the PID for the process (eg. 12345), send a signal to the running server using the kill command:

kill -HUP 12345

When the server receives the HUP (hangup) signal, it will re-parse the files on disk and use the updated project code when handling subsequent requests.

Edit this page