Databricks

Connect your Databricks data lakehouse to Bloomreach for automated data imports without manual file transfers or third-party middleware. This integration supports importing customers, events, and catalogs, with automatic synchronization as often as every 15 minutes.

Why use Databricks integration

Automated data imports: Set up scheduled, near-real-time data transfers from Databricks to Bloomreach. Your campaigns and analytics always use up-to-date information without manual intervention.
Reduced complexity: Eliminate time-consuming and error-prone SFTP file transfers or custom connectors. The integration handles data movement automatically.
Lower integration costs: Avoid extra expenses for middleware or manual processes. Reduce setup time and ongoing maintenance requirements.
Better support for AI features: Ensure Bloomreach's AI-powered tools, such as Loomi, have access to the latest customer and product data for more accurate insights and automation.
Enterprise-ready: The integration supports organizations already using Databricks or migrating to it, accommodating advanced analytics and marketing use cases.

Compute and cost

Each import runs SQL queries against your Databricks SQL warehouse, which consumes compute resources (DBUs) on your Databricks account. You control costs by choosing the import schedule and the scope of data queried. Bloomreach covers the cost of storing and processing data on its side. There's no additional per-query charge from Bloomreach.

📘
Note
Bloomreach also runs automatic health checks on this integration to confirm your connection and credentials are still valid. These checks run about every 3 hours, independent of your import or webhook activity. They don't send customer, campaign, or other business data, and they don't run an import.

Depending on your Databricks SQL warehouse configuration, including auto-suspend settings, health checks can add to your compute costs even when no import is running. Review your SQL warehouse's auto-suspend settings to manage costs between imports.

How Databricks integration works

The integration connects to Databricks and requires a one-time connection setup. Once configured, it can import from tables, views, and user-defined query results.

When importing from tables with change tracking enabled, the system automatically imports subsequent changes from Databricks as often as every 15 minutes to keep your data current.

Connection architecture

Bloomreach connects to your Databricks SQL warehouse using the native Databricks SQL connector over HTTPS. This isn't a JDBC, ODBC, or Kafka connection. Here's what the connection uses:

Transport: HTTPS (TLS-encrypted) to your Databricks SQL warehouse endpoint.
Authentication: OAuth machine-to-machine (M2M) via a Databricks service principal (Client ID and Client Secret).
Addressing: Server hostname and HTTP path identify your SQL warehouse; optional catalog and schema scope which data is accessible.

Bloomreach doesn't require middleware, reverse ETL tooling, or third-party connectors.

Security

Transport encryption: All communication between Bloomreach and your Databricks SQL warehouse uses TLS-encrypted HTTPS. No data is transmitted in plaintext.
Authentication and credential storage: The integration authenticates using a Databricks service principal (Client ID and Client Secret) via OAuth M2M. Credentials are stored encrypted at rest within Bloomreach's infrastructure.
Access control: Bloomreach only accesses data that the service principal is authorized to read. Your Databricks Unity Catalog governance and table-level ACLs remain authoritative. We recommend creating a dedicated service principal with read-only access to the specific catalog, schema, and tables needed.
Network controls: Bloomreach import workers connect from a fixed set of outbound IP addresses. To restrict access to your Databricks SQL warehouse by IP, add these addresses to your Databricks IP access list. Find your outbound IPs in Project settings > Security or Workspace settings > Security. See Outbound IP addresses.

When your Databricks SQL warehouse uses Cloud Fetch, Bloomreach downloads query results from cloud storage. If access to that storage is restricted by IP address, allowlist Bloomreach’s outbound IP addresses in both your Databricks IP access list and the storage account’s firewall or network rules.

Related resources

Updated 18 days ago

Did this page help you?