Skip to main content

Import dynamic URLs

Import dynamic URLs for importing the latest file containing the current date in Productsup.

Introduction

You may have a dynamic element in your URL, for example, a date in the filename that changes each time the platform creates a file: http://www.yourdomain.com/feed/product/fullcatalog-20210408.txt.

The dynamic date is necessary and is automatically included in the URL of your import file. This guarantees that the platform always imports the latest file containing the current date. It then inserts the current date in the filename when the platform updates a feed.

See Import a file from a URL for more information on importing data from a URL.

Import files with dynamic date in the filename using Twig

To import a file with a dynamic date element in the URL, you can use a date filter from Twig. This forces the URL to display the current date. For more information, see the date Twig filters.

For the following example, http://www.yourdomain.com/feed/product/fullcatalog-20210408.txt, Twig modifies the date 20210408 to display in the following format using the date filter: {{ "now" |date('Y-m-d') }}.

The final URL form is: http://www.yourdomain.com/feed/product/fullcatalog-{{ "now" |date('Y-m-d') }}.txt.

This means the platform always looks for a filename containing the current date and only imports this file.

Note

You should avoid adding hours, minutes, or seconds in the URL when generating files. The file upload time most likely varies every day, which can cause import failures.

For example, your data source finished loading new data into your file on September 19, 2022, at 8:55:12, with the file name displaying this time as follows: http://www.yourdomain.com/feed/product/fullcatalog-20220919-085512.txt. The next day, your file update finished at 8:47:38 and produced a file with the URL http://www.yourdomain.com/feed/product/fullcatalog-20220920-084738.txt. While you can use Twig to include a dynamic date in your filename to substitute 20220919 and 20220920, it may be challenging to predict the hour, minute, and second when your file update occurs and describe this info with another Twig.

Set the date format in a Twig

You should set the date format to match how it appears in your URL.

For example, 2020-05-19 has a date format of Y-m-d, which Twig displays as: {{ date|date('Y-m-d') }}.

In this example, where the date is 05-19-20, the format is M-d-y.

To learn more about changing the date format, see PHP DateTime.

Set the timezone in the format of your date

If necessary, you can also set the timezone of the date format.

To set the timezone as Berlin/Europe, modify your Twig to the following: {{ date |date("Ymd", "Europe/Berlin") }}.

For more information, see List of Supported Timezones.

Import yesterday’s file

You can also specify that you want to try and import a file from the past and not today’s file.

To do so, use the following Twig: {{ date |date_modify("-1 day")|date("Ymd") }}.

This searches for a file where the dynamic date element matches yesterday’s date.

You can adapt this time range by editing the -1 day element of Twig. Setting this at -7 day attempts to import the file where the dynamic date element matches the date from seven (7) days ago.

Import files with other dynamic aspects in the name

If the dynamic aspect of your filename is not a date, you can use one of the following methods. Each method is dependent on where you host your files:

  • HTTP link or FTP server

  • SFTP server

Import dynamic files from a link or FTP server using wildcards

To import files with dynamic aspects from an HTTP link or an FTP Server, you can use wildcards.

The wildcard symbol is an asterisk (*), which you use to replace dynamic elements. The wildcard ignores everything that comes in its place. It has no character match limit, so you only need to add one wildcard.

For example, if you're trying to import the following file, where the end of the address, adfc3424, is dynamic: ftp://my.ftpsource.io/files/datafeed-adfc3424.csv you should use a wildcard in the following manner: ftp://my.ftpsource.io/files/datafeed-*.

Import dynamic files from an SFTP server

Importing from an SFTP server does not support the wildcard search.

For this reason, you should use the Dynamic SFTP Download data source.

In the filename field, you can replace the dynamic element with {?}.

For example, to match the filename datafeed-adfc3424.csv, set the filename in the Dynamic SFTP Download data source to datafeed-{?}.

Note

To add a prefix to the names of attributes that the Dynamic SFTP Download data source imports into Productsup, follow these steps:

  1. Go to Data Sources from your site's main menu and select the cogwheel icon (⚙) next to the relevant data source.

  2. Find the Content Options panel at the bottom of the data source setup page and select CSV Settings or XML Settings. If your data source produces files other than CSV or XML files, you can still select either of the tabs to use this option.

  3. In Prepend a String to the Header Columns, enter the prefix you want to add to the names of the attributes imported from this data source.

  4. Select Save all settings.

  5. On the Data Sources page, open the Settings tab and modify the name of your product identifier in ID Column according to the new prefix of the relevant attribute name.

  6. If your site has additional data sources, you should go back to the Overview tab on the Data Sources page to:

    1. Select the cogwheel icon (⚙) next to an additional data source.

    2. Add a relevant prefix to the column name stated in ID Column In This Feed if you added this prefix to the columns imported from this additional data source.

    3. Choose the updated column name in ID Column From Your Main Feed if you added a prefix to the columns imported from a relevant main data source.

In this section: