Imports best practices

Importing data keeps your Bloomreach Engagement project running smoothly and helps you avoid common issues. This guide covers essential practices for importing customers, events, catalogs, and other data.

For instructions on creating imports, see the Imports article.

Plan your imports

• Test your imports in a test project before running them in production. This helps you avoid accidental deletion, duplication, or the need to reimport data.
• Split large files and schedule imports during off-peak times for better performance and stability.
• Review the Size limitations section in Data Manager to understand attribute and event storage limits before you run large imports.
• Import the oldest events first. If an error occurs, you can use "events expiration" in Data Manager to remove new information.

❗️

Warning

Don't run daily imports for all events. Importing the same events over and over (including new ones each day) causes issues with automation flows and increases your monthly processed events (MPE).

Individual import limits

• Individual imports can't exceed 1 million rows. Split larger files into smaller ones.
• Maximum file size is 1 GB.
• Contact the Product team if you need to exceed these limits.

Set up your data source

• Preview your data: Click Preview data after selecting your data source to verify the format looks correct. If something's off, check that you've selected the correct delimiter and encoding.
• Check for file issues: Check for errors or missing elements before importing.
• Watch for formatting issues: Check whether commas, dots, and spaces in the values are used correctly. When working with CSV files, one misplaced comma can move values to the wrong column.

Handle regional differences

• Remember regional differences with various Excel settings. Excel can interpret the value "6.1" as a price in one language setting but as a date in another.
• When using the "Text to Columns" function, you can specify the formatting of each column and set its decimal point to avoid this problem.
• When locating and changing values in Excel, be aware that the Excel dropdown filter lists the first 10,000 rows.

Map your columns

Make sure you select the right columns for your import. If you miss a column or mark an extra one that isn't required, you'll need to go back into the import process to fix it.

In the dropdown menu for each column name, you can map that column to existing attributes.

• When mapped to an existing attribute, the column name changes to blue font.
• You can create a new attribute by typing the name and clicking the + sign.

❗️

Warning

Watch for duplicate headers when importing the same event from multiple CSV files. The import wizard treats 2 headers with the same name as one column and, within Bloomreach Engagement, as one attribute.

Select the attribute checkbox after adding the column and data type to import the column as an attribute.

Handle customer IDs

Link IDs without importing them as attributes

During every import, pair at least one of your columns with the identifiers in the yellow boxes (located in the customer IDs section). These identifiers tell Bloomreach Engagement which customer profile to update with the imported values.

🚧

Important

Don't check the customer ID box. If you select the checkbox for the customer ID column, it imports the ID as an attribute. This causes duplication of information and can create issues with data volume in your project.

You can map multiple ID columns (registered, cookie). Bloomreach Engagement uses whichever one is available for a customer to pair the event. The registered hard ID is the top priority

Keep IDs consistent and clean

Make sure customer IDs in your import files match the IDs of existing customers. Some lists might use auto-generated IDs, while others use emails.

🚧

Important

Avoid misspellings. Misspelled IDs create duplicate customer profiles in the system.

Trim and lowercase important values like IDs to avoid wrong identification. Use the TRIM(LOWER(xx)) function in Excel.

Fix "No valid customer ID" errors

The "No valid customer ID" error occurs when the customer is missing an ID or when the ID is longer than 256 bytes. To resolve this, check customer IDs in each row of your source document.

Set correct data types

Set the correct data type for each column (text, number, date, and more). Click on a column's name and select one of the existing attributes to match it with.

Why data types matter

The data type affects how Bloomreach Engagement treats values. When filtering data, the data type determines which operators appear by default. It also affects how values display in the customer profile.

For example, the value "28.9.1996 13:45" displays in different ways depending on whether it's imported as text, number, or date format.

Avoid common data type mistakes

• Importing a timestamp as an integer causes display issues. Timestamps must be imported in the date type format.
• You can change the data type for each attribute later in Data Manager, but be careful with list values. They're processed in a different way when imported under an incorrect data type.

Each data type is explained in detail in the Data Manager article.

Track your import sources

• Add a source column with a static value by clicking the + sign on the right of the preview table. Add a "source" column with a value describing the source of the data, such as "Import_11 June 2019". This helps with troubleshooting and organizing your data later.

• Track import dates by adding an attribute like import_date=20190715. The total number of events with this attribute should match the number of lines in your CSV file.

Import catalogs

Set searchable fields during first import

Set the right columns as searchable when you first import a catalog. If columns aren't searchable, you'll have trouble creating recommendations because the data won't be indexed in the backend.

Searchable fields are available in Bloomreach Engagement search, and recommendations are based on these columns. For example, if you want to make product recommendations based on brand, the brand column must be set as a searchable field.

❗️

Warning

You can assign searchable columns while creating the import. Once the catalog structure is created, you can't change it. You can only add more items to the catalog.

Searchable field limits

• Maximum of 40 columns can be set as searchable and available for recommendations.
• List and JSON fields can't be set as searchable.

Understand catalog IDs

At least one ID tag must be assigned. There are 2 IDs for products: item_id and product_id.

The difference: One product with a single product_id can have multiple variants with different item_ids. For example, "iPhone X Case" might have product_id "123", but the black, white, and green variants each have their own item_id while sharing the same product_id.

For general catalogs, item_id is used as the identifier.

Replace an existing catalog

If you need to replace a catalog with a new one that has different searchable fields or data types, follow these steps:

1. Make sure data types are properly set and all searchable fields are correctly marked in your new catalog. You can't remove or rename columns, change data types, or alter searchability later.
2. If you want to migrate existing emails and imports to use the new catalog, keep existing column names the same. Otherwise, you'll need to update the Jinja code in the emails.
3. Create a new catalog from a fresh import. The import may take a while, even after the UI's progress bar reaches 100%. Verify it's complete and correct based on what's visible in the UI.
4. Make a note of the new catalog ID from the URL.
5. Rename the old catalog, and rename the new catalog to the same name as the original. This can be done in the UI and ensures that any emails referencing the catalog in Jinja will continue to work, as Jinja references catalogs by name.
6. Any existing imports should still work against the new catalog, provided column names and data types are the same as before. Otherwise, you need to modify them according to the catalog name.
7. Modify any existing API updates to reference the new catalog ID. If columns are the same, then the ID should be the only thing that needs changing, which is in the API endpoint URL.
8. Edit any existing recommendations manually to reference the new catalog, since this relationship is fixed on the catalog ID.
9. Delete the old catalog after all the recommendations are re-pointed to the new one.

Import events

Include required columns

Two columns are required for event imports:

• Customer IDs to pair each event with the correct customer (assign the yellow tag).
• Timestamp for each event (assign the blue tag).

Use the right timestamp format

Transform all time formats into UNIX timestamps (the number of seconds since January 1, 1970 00:00:00 UTC). Using UNIX timestamps avoids problems with different time formats where, for example, "02/04/2017" could be interpreted as either April or February.

UNIX timestamp format is flexible and recommended for all imports. Check the timezone of your timestamp before importing.

Accepted formats

• Datetime format: "2020-08-05T10:22:00".
• Numeric format: number of seconds since 1970 (for example, "1596615705"—note this works in seconds).

If you provide a date, Bloomreach Engagement adds midnight as the event's time.

Avoid future timestamps

Events with future timestamps are processed, but become visible only after that future time has passed. Tampering with the event could create duplicates and create extra load on the platform.

To make the data visible today, import it again with a valid timestamp. When you make this change and the import runs, the data should appear in the platform without problems.

❗️

Warning

Imported events with the same type, timestamp, and attributes for the same customer are deduplicated. However, these events still trigger scenarios before being discarded. If you import the same events multiple times with the trigger option enabled, scenarios trigger each time.

Handle purchase events

Make sure all purchase_item events have a purchase_id attribute that's tracked in a matching purchase event. This allows multiple purchase_item events from a single purchase to be treated as such.

When combining two sets of events, make sure the price attribute in the purchase event uses the same VAT, shipping, and discount settings in both sets. Check whether both sets include refunded purchases.

Manage repeated imports

When you set a repeated import and it gets stuck, you can edit the repeated import setting and save it again without changing anything. This kicks off the repeated imports again.

Time your reimports

After deleting events, don't start reimporting right away. Events and customers may appear deleted in the Bloomreach Engagement app, but the deletion process on the backend may still be in progress. The deletion process has low priority on the backend, so it takes longer than the import.

Understand import processing

The import system is designed for throughput instead of reactivity. All projects on an instance use the same queue, which means many large imports in one project may delay imports in other projects.

When you create an import, it's first scheduled into the queue and starts once a worker is available. This scheduled phase takes a few seconds, but if the instance is busy processing many other parallel imports, one import must finish before the next one in the queue can start.

First-come, first-served

Imports are processed on a first-come, first-served basis. Sometimes there's a delay between importing and visibility in the UI. This can occur when larger imports are queued in the system, consuming more system resources.
You can follow the import and processing phases using the import progress bar. After the processing phase finishes, your data is available in the system.

Handle errors

When there's an error, the rows with errors won't be imported. The rest of the file is imported.

For more troubleshooting, refer to the Tips for importing on Engagement article, which covers the most common import issues and their fixes.