Hightouch lets you pull data stored in Databricks and push it to downstream destinations. Hightouch connects to Databricks using Open Database Connectivity (ODBC). This guide walks you through getting your ODBC URL for your Databricks cluster and using your credentials to connect to Hightouch.
You may need to allowlist Hightouch's IP addresses to let our systems connect to your Databricks warehouse. Reference our networking docs to determine which IP addresses you need to allowlist.
You can also securely connect to your Databricks warehouse using AWS PrivateLink. AWS PrivateLink is a Business Tier feature.
In your Databricks Account console, go the Workspaces page. Select the relevant workspace and then click Open workspace.
In your workspace, go the Compute page and click the relevant cluster. This brings you to its Configuration tab.
At the bottom of the page, expand the Advanced Options toggle and open the JDBC/ODBC tab.
This tab displays your cluster's Server Hostname, Port, and HTTP Path, which you need to connect to Hightouch. Keep the tab open, or save these to a secure location.
Enter the following required fields into Hightouch:
Server hostname: To find your server hostname, visit the Databricks web console and locate your cluster. Then, click to reveal Advanced options and navigate to the JDBC/ODBC tab.
Port: The default port number is 443, but yours may be different. To find your port, visit the Databricks web console and locate your cluster. Then, click to reveal Advanced options and navigate to the JDBC/ODBC tab.
HTTP path: To find your HTTP path, visit the Databricks web console and locate your cluster. Then, click to reveal Advanced options and navigate to the JDBC/ODBC tab.
(Optional) Catalog: You can optionally include a Databricks catalog. Catalogs are the first level of Unity Catalog's three-level namespace. If Lightning engine is enabled, the 'hightouch_audit' and 'hightouch_planner' schemas need to be created in the catalog specified here. If this field is left empty, these schemas should be created in the default catalog. Note that data in other catalogs can still be accessed for model creation by specifying the full name (catalog.schema.table) in the Hightouch SQL interface.
Schema: The initial schema to use for the connection.
SQL dialect: By default, Hightouch assumes that your queries use the Databricks SQL dialect. You may wish to override this behavior if your queries use legacy ANSI SQL-92 syntax. Some features are not available when legacy syntax is used.
For optimal performance, Hightouch tracks incremental changes in your data model—such as added, changed, or removed rows—and only syncs those records. You can choose between two different sync engines for this work.
The Basic engine requires read-only access to Databricks. Hightouch executes a query in your database, reads all query results, and then determines incremental changes using Hightouch's infrastructure. This engine is easier to set up since it requires read—not write—access to Databricks.
The Lightning engine requires read and write access to Databricks. The engine stores previously synced data in a separate schema in Databricks managed by Hightouch. In other words, the engine uses Databricks to track incremental changes to your data rather than performing these calculations in Hightouch. Therefore, these computations are completed more quickly.
Hightouch recommends using the Lightning sync engine when syncing more than 100 thousand rows of data.
If you select the Basic engine, you can switch to the Lightning engine later. Once you've configured the Lightning engine, you can't move back to the Basic engine without recreating Databricks as a source.
The Lightning sync engine requires granting write access to your data warehouse, which makes its setup more involved than the Basic sync engine. However, it is more performant and reliable than the Basic engine. This makes it the ideal choice to guarantee faster syncs, especially with large data models. It also supports more features, such as Warehouse Sync Logs, Match Booster, and Identity Resolution.
Criteria
Basic sync engine
Lightning sync engine
Performance
Slower
Quicker
Ideal for large data models (over 100 thousand rows)
No
Yes
Reliability
Normal
High
Resilience to sync interruptions
Normal
High
Extra features
None
Warehouse Sync Logs, Match Booster, Identity Resolution
Ease of setup
Simpler
More involved
Location of change data capture
Hightouch infrastructure
Databricks schemas managed by Hightouch
Required permissions in Databricks
Read-only
Read and write
Ability to switch
You can move to the Lightning engine at any time
You can't move to the Basic engine once Lightning is configured
To set up the Lightning engine, ensure that your Databricks user or service principal has the appropriate permissions. You can do so by running the following SQL snippet.
Before running the snippet, make sure to include your own username, which can be found in the top right corner of the Databricks web console. Alternatively, you can create an API-only service principal by following these instructions.
CREATE SCHEMA IF NOTEXISTS hightouch_audit;
CREATE SCHEMA IF NOTEXISTS hightouch_planner;
GRANTALL PRIVILEGES ON SCHEMA hightouch_audit TO<YOUR_USER>;
GRANTALL PRIVILEGES ON SCHEMA hightouch_planner TO<YOUR_USER>;
When setting up a source for the first time, Hightouch validates the following:
Network connectivity
Databricks credentials
Permission to list schemas and tables
Permission to write to hightouch_planner schema
Permission to write to hightouch_audit schema
All configurations must pass the first three, while those with the Lightning engine must pass all of them.
Some sources may initially fail connection tests due to timeouts. Once a connection is established, subsequent API requests should happen more quickly, so it's best to retry tests if they first fail. You can do this by clicking Test again.
If you've retried the tests and verified your credentials are correct but the tests are still failing, don't hesitate to .
Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up models to define which data you want to pull from Databricks.
The Databricks source supports these modeling methods:
You may also want to consider storing sync logs in Databricks. Like using the Lightning sync engine versus the standard one, this feature lets you use Databricks instead of Hightouch infrastructure. Rather than performance gains, it makes your sync log data available for more complex analysis. Refer to the warehouse sync logs docs to learn more.
The best way to resolve the error is to rewrite your model query to remove subqueries. Common table expressions (CTE) are supported. If you require assistance, don't hesitate to .
This error can occur when testing the source connection or when running a sync that uses a Databricks model.
It is typically caused by an expired Databricks access token.
To solve this, generate a new token and insert it in the source configuration.
Ready to get started?
Jump right in or a book a demo. Your first destination is always free.