Databricks integration requirements

This page lists what your Databricks account, source tables, and delta setup need to satisfy for the integration to work.

Access requirements

Before you set up the integration, make sure you have:

A Databricks service principal (Client ID and Client Secret) with access to the relevant catalog, schema, and tables. You might need account admin privileges to create the service principal.
Bloomreach account with Data hub module enabled. Contact your Customer Success Manager to enable Databricks support for catalog imports in your project.

Data format requirements

Bloomreach has flexible data format requirements. During import setup, you map the source data format to the relevant data structure.

The following columns must be present in source tables or views:

Data category	Required columns	Optional columns
Customers	ID to be mapped to the customer ID in Bloomreach (typically `registered`)	Timestamp to be mapped to `update_timestamp`
Events	ID to be mapped to the customer ID in Bloomreach (typically `registered`); timestamp
Catalog	ID to be mapped to the catalog item ID in Bloomreach (typically `item_id`)

Bloomreach supports the following attribute data types: text, long text, number, Boolean, date, datetime, list, URL, and JSON.

Delta update requirements

Delta updates are supported for tables only. They aren't supported for views.

This feature requires delta.enableChangeDataFeed to be enabled on the source table in Databricks. Enable the Change Data Feed using the following command:

ALTER TABLE myDeltaTable SET TBLPROPERTIES (delta.enableChangeDataFeed = true)

For more details, see the Change Data Feed documentation by Databricks.

Limitations

Delta updates don't support delete operations. If you delete a record in Databricks that was previously imported, Bloomreach doesn't delete it on the next sync.
Deleting a record in Bloomreach doesn't affect Databricks. The record remains in Databricks and is only recreated in Bloomreach if it's updated in Databricks, which triggers the import job to re-import it.