File storage imports

This guide explains how to connect to SFTP, Amazon S3, or Google Cloud Storage to import data files.

Prerequisites

• Configured connection: Under Data hub > Integrations, set up your SFTP, Amazon S3, or Google Cloud Storage connection before you can select it as a file storage option.
• Storage access permissions: Make sure you have the appropriate permissions to access the storage location and read files from the specified directories.
• Data hub permissions: Ensure you have the required permissions.
• File preparation: Prepare your data file in CSV or XML format.

Configure file storage import

After creating a new import and select the import type.

Select file storage option

1. Navigate to the File storage tab.

2. Click the File storage dropdown menu.

3. Select your preferred storage option (SFTP, Amazon S3, or Google Cloud Storage). If your storage option doesn't appear, configure the connection first.

   

Configure connection details

Specify where your data file lives and how the system should read it:

• Path to parent directory: Enter the static path to the folder that contains your data file. The import only checks files directly in this folder, not in its subfolders.
• File matching: Choose how the import finds your file:
  • Fixed: The import uses one file with an exact name. Enter the full file name, including the extension, for example product-catalog.xml.
  • Dynamic: The import finds the file by pattern. Use this option when the file name changes with each export, for example when it includes a date. Enter a regular expression without enclosing slashes, for example ^catalog_\d{4}-\d{2}-\d{2}\.csv$. For syntax and examples, see File name matching.
• Selection rule: Available with Dynamic file matching. If more than one file matches the pattern, choose which one to import:
  • Last modified: Imports the file that was most recently modified on the server. Recommended for full feeds.
  • Last alphabetically: Imports the file that comes last when the matching file names are sorted alphabetically.

    📘

    Note

    Each run imports one file. If no file matches, the run fails, and you get a failure notification. If the same file is still the newest match on the next run, it's imported again, so move imported files out of the parent directory.

• File format: Select your file format (CSV or XML) from the dropdown, or choose Autodetect to let the system identify the format automatically.

• Encoding: Select the character encoding used in your file. Use UTF-8, or select a different encoding if your source system uses another format.

Import limits

Maximum columns: 260 columns per file.

Performance recommendations

• File size: Maximum of 1 GB per file.
• Row count: Maximum of 1,000,000 rows per import file.

Preview and validate data

1. Click Preview data to validate that your file appears correctly and the system can read it as expected. With Dynamic file matching, the preview shows the name of the file that the selection rule selected.
2. Click Next and specify the data mapping.

Troubleshooting