Send Your Data (Product)

❗️

Please ignore the "Accept" header (Accept: application/json) in the request examples. This is just a quirk of the example module. Also, make sure to use the correct "Content-Type" header from here.

There are two modes available to update your catalog data:

• PUT (full): replace the catalog’s data entirely
• PATCH (delta): make partial updates to the catalog's data

PATCH mode

When you PATCH products into a catalog, you can make partial updates to the catalog’s data. If you need to send remove patch operations, you should use PATCH mode.

📘

You can index multiple PATCH operations with a single Index API call.

In PATCH mode, if there are malformed products, the entire modification will be considered invalid and not applied.

Before you begin

Before sending your content catalog data to Bloomreach via the Catalog Data Management API, ensure that you have the following:

1. Formatted catalog data - Create your product catalog data according to the Format your Data page. After formatting your data, you must deliver it to Bloomreach using one of two methods:
   • Direct API payload: send your catalog data directly via an API payload
   • SFTP file : send your catalog data via a file on SFTP
2. Transfer Key - You need an API key for authentication that is unique to your merchant account. If you do not already have it, speak to your Bloomreach representative.
3. Catalog names - Catalog names used in the Catalog Data Management API endpoints. Use the same value as your domain key in your search API requests.
4. Authentication key - Required header for Catalog Data Management API requests. You should have received this during kickoff.

❗️

No malformed items allowed in catalog data

if there are any malformed items in your catalog data, the whole operation will be invalid and no changes will be applied.

Endpoints

• Production: https://api.connect.bloomreach.com/dataconnect/api/v1/
• Staging: https://api-staging.connect.bloomreach.com/dataconnect/api/v1/

Authentication key

An API key will be provided for each environment in your account. This access key must accompany all API requests as a request header. Example value: merchantname-1066facc-94fb-450e-9f9c-b66d0b155fd2

Send Data as API payload

To send your catalog data directly via an API payload, you must convert all of the operations into a single JSON Patch, which is an array of catalog records. In your Catalog Data Management API request body, include the catalog records.

The max size of a single API payload is 5MB.

[{
  "op": "add",
  "path": "/products/pid-123",
  "value": {
    "attributes": {
      "title": "Example 123",
      "price": 100,
      "description": "Example product description here",
      "url": "http://www.example.com/example-product-url.html",
      "availability": true
    }
  }
},
{
  "op": "add",
  "path": "/products/pid-456",
  "value": {
    "attributes": {
      "title": "Example 456",
      "tags": ["", ""]
    }
  }
},
{
  "op": "add",
  "path": "/products/pid-789",
  "value": {
    "attributes": {
      "title": "Example 789",
      "category": "Seafood",
      "tags": ["", ""]
    }
  }
},
{
  "op": "add",
  "path": "/products/pid-101112/attributes/availability",
  "value": false
}
]

Sending up to 1 GB Data using JSON Lines

You can send more than 5 MB (and up to 1 GB) of data by formatting your JSON Patch as JSON Lines. For this, each valid JSON Patch operation object should be on a single line.

Here’s a sample request body in the JSON lines format:

{"op":"add", "path":"/products/123", "value":{...}}
{"op":"add", "path":"/products/456", "value":{...}}
{"op":"add", "path":"/products/789", "value":{...}}
{"op":"add", "path":"/products/abc", "value":{...}}
{"op":"add", "path":"/products/def", "value":{...}}

curl --request PATCH \
     --url https://api.connect.bloomreach.com/dataconnect/api/v1/accounts/6702/catalogs/catalog_name/products \
     --header 'Content-Type: application/json-patch+jsonlines' \
     --data '
     {"op":"add", "path":"/products/123", "value":{...}}
     {"op":"add", "path":"/products/456", "value":{...}}
     {"op":"add", "path":"/products/789", "value":{...}}
     {"op":"add", "path":"/products/abc", "value":{...}}
     {"op":"add", "path":"/products/def", "value":{...}}    
'

Additionally, you must use the following Content-Type header:

Content-Type: application/json-patch+jsonlines

You may gzip compress the request payload; however, the uncompressed payload may not be larger than 1GB. A gzip compressed request payload must also include the Content-Encoding: gzip HTTP header.

Good to Remember

• When the request is being made via cURL, the data should be sent as binary (“--data-binary”) so that the newlines remain intact.

Send Data as a file through SFTP

To send your catalog data via a file on SFTP, you must convert all of the patch operations into a file of JSONLines. Each line should only contain one patch operation. In your Catalog Data Management API request body, include the relative path to the file location.

Feed files may be sent uncompressed, or be compressed using gzip. The max size of a single API payload is 5MB.

In the request body, provide the paths to your catalog files as relative paths. For example, if your catalog files are in a folder named "content_en", then the request body should look like the following:

[ 
  "content_en/20200410-090909-products1_staging.jsonl", 
  "content_en/20200410-090909-products2_staging.jsonl" 
]

{"op": "add", "path": "/products/pid-123", "value": {...}} //First product record
{"op": "add", "path": "/products/pid-456", "value": {...}} //Second product record
{"op": "add", "path": "/products/pid-789", "value": {...}} //Third product record, etc.
…

📘

When to use SFTP transfer

Other than the case that you have already been using SFTP File Transfer in your organization's workflow and that it's convenient for you, there are other reasons why you can consider using it:

• If you want to send more than 1 GB data, which is not possible through a direct payload.
• If you're sending sensitive data, SFTP is a highly secure option.
• SFTP transfer lets you independently control when you want to send the data and when to ingest it (or even send it to an intermediary server).

SFTP Endpoints

• Production: sftp.connect.bloomreach.com (3.82.164.133, port 22)
• Staging: sftp-staging.connect.bloomreach.com (54.211.108.247, port 22)

Your sftp command should look like the following:

sftp -i ~/path/to/private_key @sftp-staging.connect.bloomreach.com

Provide Bloomreach with public keys that will need access to each environment. More than one can be provided for each environment, and we advise using different public keys for staging and production.

Steps

For Delta updates

1. If sending data through SFTP, upload your catalog data to your Bloomreach SFTP location.
2. Send the Catalog Data Management PATCH API request.
3. If the order of data is important, wait for the PATCH job to finish.
4. Run the index API.

For Full updates

1. If sending data through SFTP, upload your catalog data to your Bloomreach SFTP location.
2. Send the Catalog Data Management PUT API request.
3. Run the index API.

📘

SFTP tips

SFTP directory structure

Any catalog data files should be sent via the SFTP location provided in the SFTP Endpoints section. We suggest organizing catalogs into their own folders to help with debugging. For example, you could create a folder called <catalog_name> and then place files related to that catalog in this folder.

File naming

We suggest naming your catalog data files to include the catalog name, the type of modification (put/full, patch/delta) and a datetime version identifier. This will help with debugging. For example, if your catalog is named <catalog_name>, you could name your file 20200410-090909-<catalog_name>_full.jsonl

Storage length

Files sent to the Bloomreach SFTP are stored for 30 days. Files that are older than 30 days will be automatically deleted and cleared.

Job ID

The response contains a job ID that can be used to query for status.

Sample PATCH request

Given below is a sample patch operation request: 

Best practices when working with Delta feeds (PATCH on /products or /items)

Catalog data changes that impact a small portion of the products

When you have changes in your catalog data that impact a portion of the products or items in the catalog, we recommend using our delta feed system. Our delta feed system allows you to make fine-grained changes with individual operations.

All operations must be valid when applying these changes to your catalog in a delta; otherwise, the entire feed will fail, and no changes will be applied. This ensures the consistency of your data.

Bloomreach’s Recommendation: For implementing the delta feed, we recommend that you batch up your catalog changes into small jobs and send them more frequently instead of sending many changes in a single delta operation all at once.

An error in a single large delta operation will block the application of all your changes. Batching up the changes into small jobs ensures that your delta feed changes are processed and implemented efficiently.

A small change would be on the order of 5-1000 operations per delta job.

Catalog data changes that impact a large portion of the products

When there is a change in your system that impacts a large percentage of your entire catalog at once, then a full feed may be more suitable than a delta feed job.

The percentage will vary from catalog to catalog, but a good rule of thumb is any delta change modifying more than 20% of products or items in a catalog at once.

Bloomreach’s Recommendation: In this scenario, we recommend that you regenerate a full feed and send that. If you're sensitive to processing times and plan on sending large changes (even if it is spread out across many smaller delta feeds), you may benefit from sending a full feed if your source system allows for that.

By default, we place an adjustable limit of 10k operations in a single delta feed job. Please contact Bloomreach support if your use case needs to generate a delta feed with more than 10k operations.

Pick the most efficient operation type and path for your change

When modifying products, choose the appropriate operation and path combination for the best performance.

Generally speaking, you should modify the smallest part of the product that has changed as opposed to modifying a larger component. The path indicates the component level you are operating on.

When you modify a larger component than is necessary, you will modify data that is unrelated to the specific change in your product that you care about. This extra work increases the time it takes to process. This is particularly important if your products have many variants or views.

For instance, if you have a product that is going in and out of availability throughout the day, update only that product's availability instead of removing the entire product (sample below). If the products are truly being removed, then your next full feed job can remove those products.

✅ Do this

{
    "op": "add",
    "path": "/products/pid1/attributes/availability",
    "value": false
}

❌ Instead of this

{
    "op": "remove",
    "path": "/products/pid1"
}

The reason this is more efficient is that the product you are modifying may have many variants or views that will also need to be removed from our system instead of a single attribute of the product being modified. If you had 100 variants on this product, you would be doing 100x more work than is necessary.

Parameters Reference