(Applies to dbt v2.0 and later)

Connect Apache Spark to dbt v2 Beta

dbt v2 supports Apache Spark, enabling faster compilation and execution for your Spark-based dbt projects. Currently, dbt v2 only supports Apache Spark 3.0.

dbt v2 and Spark

dbt v2 uses the Databricks SQL dialect for static analysis when working with Spark. Databricks SQL is a superset of Spark SQL, so your SQL is validated with Databricks semantics. This provides comprehensive error checking and SQL comprehension features. A dedicated Spark SQL dialect for static analysis is planned for a future release.

Authentication

The Spark adapter in dbt v2 supports:

• Thrift
  • Simple Authentication and Security Layer (SASL) PLAIN
  • No SASL (NOSASL)
• Livy
  • Basic authentication (username and password)
  • When deployed on Amazon Web Services (AWS): AWS Signature Version 4
    • Supports authentication using single sign-on, service accounts, or user tokens

Configure dbt v2

Configure your Spark connection in profiles.yml:

~/.dbt/profiles.yml

your_profile_name:
  target: dev
  outputs:
    dev:
      type: spark
      method: CONNECTION_METHOD # thrift, http, or livy
      port: PORT_NUMBER
      auth: AUTH_METHOD
      schema: SCHEMA_NAME
      host: HOSTNAME
      platform_hint: PLATFORM_HINT # optional; aws_emr_serverless or aws_emr_eks
      server_side_parameters: # optional; required keys depend on platform_hint
        livy.server.session.ttl: SESSION_TTL_SECONDS # optional when using method: livy

Profile field	Required	Description	Example
method	Yes	Connection method. Accepted values: thrift, http, or livy.	thrift
port	Yes	Port number for the connection.	443
auth	Yes	Authentication method.	SASL PLAIN, NOSASL, AWS_SIGV4
schema	Yes	The database or schema name where dbt will create and query objects.	analytics
host	Yes	Hostname of the Spark cluster or Databricks workspace.	yourorg.sparkhost.com
platform_hint	No	Hints to dbt v2 which Spark platform you use. dbt v2 uses this to validate required server_side_parameters. Accepted values: aws_emr_serverless, aws_emr_eks. If omitted, dbt v2 assumes a generic Spark cluster.	aws_emr_serverless
server_side_parameters	No	Spark session parameters passed to the cluster. Required keys when using platform_hint: - For aws_emr_serverless, use emr-serverless.session.executionRoleArn. - For aws_emr_eks, use spark.kubernetes.namespace. When using method: livy, you can set livy.server.session.ttl to configure how long a session can remain idle before it is terminated.	Refer to example profiles.

Example profiles

Thrift (binary)

~/.dbt/profiles.yml

spark-local-thrift-binary:
  target: spark
  outputs:
    spark:
      type: spark
      method: thrift
      port: 10000
      auth: NOSASL
      host: localhost
      schema: my_schema

Thrift (HTTP)

~/.dbt/profiles.yml

spark-local-thrift-http:
  target: spark
  outputs:
    spark:
      type: spark
      method: http
      port: 443
      auth: NOSASL
      # Omit the protocol scheme to use HTTPS. For HTTP, use a host like http://localhost
      host: localhost
      schema: my_schema

AWS EMR Serverless

~/.dbt/profiles.yml

spark-emr-serverless:
  target: spark
  outputs:
    spark:
      type: spark
      method: livy
      auth: AWS_SIGV4
      port: 443
      host: "YOUR_APPLICATION_ID.livy.emr-serverless-services.us-east-1.amazonaws.com"
      schema: my_schema
      platform_hint: aws_emr_serverless
      server_side_parameters:
        "spark.driver.memory": "1g"
        "spark.driver.cores": 1
        "spark.executor.memory": "1g"
        "spark.executor.cores": 1
        # Required by EMR Serverless when using platform_hint: aws_emr_serverless
        "emr-serverless.session.executionRoleArn": "arn:aws:iam::YOUR_AWS_ACCOUNT:role/YOUR_ROLE"

AWS EMR on EKS

~/.dbt/profiles.yml

spark-emr-eks:
  target: spark
  outputs:
    spark:
      type: spark
      method: livy
      auth: AWS_SIGV4
      port: 443
      host: "https://my-spark-cluster.com"
      schema: my_schema
      platform_hint: aws_emr_eks
      server_side_parameters:
        # Required by EMR on EKS when using platform_hint: aws_emr_eks
        "spark.kubernetes.namespace": "emr-jobs"
        "spark.driver.memory": "1g"
        "spark.driver.cores": 1
        "spark.executor.memory": "8g"
        "spark.executor.cores": 4

For detailed configuration options, refer to the Spark configuration page.

Was this page helpful?

Privacy policyCreate a GitHub issue

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.