Skip to main content

Import dynamic URLs

Import dynamic URLs for importing the latest file containing the current date in Productsup.

Introduction

You may have a dynamic element in your URL, for example, a date in the filename that changes each time the platform creates a file: http://www.yourdomain.com/feed/product/fullcatalog-20210408.txt.

The dynamic date is necessary and is automatically included in the URL of your import file. This guarantees that the platform always imports the latest file containing the current date. It then inserts the current date in the filename when the platform updates a feed.

See Import a file from a URL for more information on importing data from a URL.

Import files with dynamic date in the filename using Twig

To import a file with a dynamic date element in the URL, you can use a date filter from Twig. This forces the URL to display the current date. For more information, see the date Twig filters.

For the following example, http://www.yourdomain.com/feed/product/fullcatalog-20210408.txt, Twig modifies the date 20210408 to display in the following format using the date filter: {{ "now" |date('Y-m-d') }}.

The final URL form is: http://www.yourdomain.com/feed/product/fullcatalog-{{ "now" |date('Y-m-d') }}.txt.

This means the platform always looks for a filename containing the current date and only imports this file.

Set the date format in a Twig

You should set the date format to match how it appears in your URL.

For example, 2020-05-19 has a date format of Y-m-d, which Twig displays as: {{ date|date('Y-m-d') }}.

In this example, where the date is 05-19-20, the format is M-d-y.

To learn more about changing the date format, see PHP DateTime.

Set the timezone in the format of your date

If necessary, you can also set the timezone of the date format.

To set the timezone as Berlin/Europe, modify your Twig to the following: {{ date |date("Ymd", "Europe/Berlin") }}.

For more information, see List of Supported Timezones.

Import yesterday’s file

You can also specify that you want to try and import a file from the past and not today’s file.

To do so, use the following Twig: {{ date |date_modify("-1 day")|date("Ymd") }}.

This searches for a file where the dynamic date element matches yesterday’s date.

You can adapt this time range by editing the -1 day element of Twig. Setting this at -7 day attempts to import the file where the dynamic date element matches the date from seven (7) days ago.

Avoid hours, minutes, and seconds in the URL

You should avoid adding hours, minutes, or seconds in the URL when generating files because it can cause import failures due to timezone differences.

For example, your data source finished loading new data into your file on September 19, 2022, at 00:55:12 in the Berlin timezone, with the file name displaying this time as follows: http://www.yourdomain.com/feed/product/fullcatalog-20220919-005512.txt. A schedule runs your site at 1:00:00 Berlin time to import a file with the dynamic URL http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date('Ymd-his') }}.txt. But instead of the newest updates, the platform ends up importing a file from September 18, 23:00:00 because Productsup servers use the UTC timezone to unify processes across timezones.

However, there are situations where it is impossible to avoid including hours, minutes, and seconds into the URL of your file. If so, see the following:

  1. You can use a Twig to define the time format used in the URL. For example, you can use the Twig {{ date |date('Ymd-his') }} to import a file featuring the hours, minutes, and seconds of the upload time as 20220919-005512.

  2. You should refer to the timezone of your file upload timestamp within the Twig that specifies your upload time format. For example, {{ date |date("Ymd-his", "Europe/Berlin") }}.

    This means the file name you use to import data should be similar to http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date("Ymd-his", "Europe/Berlin") }}.txt.

Import files with other dynamic aspects in the name

If the dynamic aspect of your filename is not a date, you can use one of the following methods. Each method is dependent on where you host your files:

  • HTTP link or FTP server

  • SFTP server

Import dynamic files from a link or FTP server using wildcards

To import files with dynamic aspects from an HTTP link or an FTP Server, you can use wildcards.

The wildcard symbol is an asterisk (*), which you use to replace dynamic elements. The wildcard ignores everything that comes in its place. It has no character match limit, so you only need to add one wildcard.

For example, if you're trying to import the following file, where the end of the address, adfc3424, is dynamic: ftp://my.ftpsource.io/files/datafeed-adfc3424.csv you should use a wildcard in the following manner: ftp://my.ftpsource.io/files/datafeed-*.

Import dynamic files from an SFTP server

Importing from an SFTP server does not support the wildcard search.

For this reason, you should use the Dynamic SFTP Download data source.

In the filename field, you can replace the dynamic element with {?}.

For example, to match the filename datafeed-adfc3424.csv, set the filename in the Dynamic SFTP Download data source to datafeed-{?}.

Note

To add a prefix to the names of attributes that the Dynamic SFTP Download data source imports into Productsup, follow these steps:

  1. Go to Data Sources from your site's main menu and select the cogwheel icon (⚙) next to the relevant data source.

  2. Find the Content Options panel at the bottom of the data source setup page and select CSV Settings or XML Settings. If your data source produces files other than CSV or XML files, you can still select either of the tabs to use this option.

  3. In Prepend a String to the Header Columns, enter the prefix you want to add to the names of the attributes imported from this data source.

  4. Select Save all settings.

  5. On the Data Sources page, open the Settings tab and modify the name of your product identifier in ID Column according to the new prefix of the relevant attribute name.

  6. If your site has additional data sources, you should go back to the Overview tab on the Data Sources page to:

    1. Select the cogwheel icon (⚙) next to an additional data source.

    2. Add a relevant prefix to the column name stated in ID Column In This Feed if you added this prefix to the columns imported from this additional data source.

    3. Choose the updated column name in ID Column From Your Main Feed if you added a prefix to the columns imported from a relevant main data source.

In this section: