Import dynamic URLs

You should set the date format to match how it appears in your URL. If the date is 05-19-20, the format is M-d-y.

For example, the date 2020-05-19 is in the format of Y-m-d, and Twig displays it as {{ date|date('Y-m-d') }}.

See PHP DateTime to learn more about changing the date format.

Importing a dynamic filename from Google Cloud Storage can be a challenge due to its naming convention of objects. In Google Cloud, an object is specified by its file path and filename. You must designate them and the date in Twig.

For example, the product file has the following parameters:

• file path: item/full

• filename: target_item_full_2022-09-20.csv.gz

Set the following Twig the Object field in the settings of the Google Cloud Storage data source in Productsup: {{ [ 'item/full/target_item_full_' , "now"|date("Y-m-d"), '.csv.gz' ] | join}}

If necessary, you can also set the timezone of the date format.

To set the timezone as Berlin/Europe, modify your Twig as {{ date |date("Ymd", "Europe/Berlin") }}.

See List of Supported Timezones for more information.

You can also specify that you want to try and import a file from the past and not today’s file.

Use the Twig {{ date |date_modify("-1 day")|date("Ymd") }}.

It searches for a file where the dynamic date element matches yesterday’s date.

You can adapt this time range by editing the -1 day element of Twig. For example, set -7 day to import the file where the dynamic date element matches the date from seven (7) days ago.

You should avoid adding hours, minutes, or seconds in the URL when generating files because it can cause import failures due to timezone differences.

For example, your data source finished loading new data into your file on September 19, 2022, at 00:55:12 in the Berlin timezone, with the file name displaying this time as follows: http://www.yourdomain.com/feed/product/fullcatalog-20220919-005512.txt. A schedule runs your site at 1:00:00 Berlin time to import a file with the dynamic URL http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date('Ymd-his') }}.txt. But instead of the newest updates, the platform ends up importing a file from September 18, 23:00:00 because Productsup servers use the UTC timezone to unify processes across timezones.

However, there are situations where it's impossible to avoid including hours, minutes, and seconds into the URL of your file. If so, see the following:

To import files with dynamic aspects from an HTTP link or an FTP Server, you can use wildcards.

The wildcard symbol is an asterisk (*), which you use to replace dynamic elements. The wildcard ignores everything that comes in its place. It has no character match limit, so you only need to add one wildcard.

For example, if you're trying to import the following file, where the end of the address, adfc3424, is dynamic: ftp://my.ftpsource.io/files/datafeed-adfc3424.csv you should use a wildcard in the following manner: ftp://my.ftpsource.io/files/datafeed-*.

Importing from an SFTP server doesn't support the wildcard search.

For this reason, use the Dynamic SFTP Download data source. When setting up the data source, make the following change:

To add a prefix to the names of attributes that the Dynamic SFTP Download data source imports into Productsup, follow these steps: