Import dynamic URLs

Import dynamic URLs for importing the latest file containing the current date in Productsup.

Introduction

While importing your file from a URL, you may experience having a dynamic element in your URL. A dynamic element can be a date in the filename that changes each time the platform creates a file, for example, http://www.yourdomain.com/feed/product/fullcatalog-20210408.txt.

The dynamic date is necessary and is automatically included in the URL of your import file. This guarantees that the platform always imports the latest file containing the current date. It then inserts the current date in the filename when the platform updates a feed.

Import files with a dynamic date in the filename using Twig

To import a file with a dynamic date in the URL, you can use a date filter from Twig. This forces the URL to display the current date. For more information, see the date Twig filters.

For the following example, http://www.yourdomain.com/feed/product/fullcatalog-20210408.txt, Twig modifies the date 20210408 to display in the following format using the date filter: {{ "now" |date('Y-m-d') }}.

The final URL form is: http://www.yourdomain.com/feed/product/fullcatalog-{{ "now" |date('Y-m-d') }}.txt

This means the platform always looks for a filename containing the current date and only import this file.

Note

You should avoid adding hours, minutes, or seconds in your URL when generating files. The file upload time most likely varies every day. This means it may be a challenge to import the file, as you can't predict the URL when it contains minutes and seconds.

Set the date format in a Twig

You should set the date format to match how it appears in your URL.

For example, 2020-05-19 has a date format of Y-m-d, which Twig displays as: {{ date|date('Y-m-d') }}.

In this example, where the date is 05-19-20, the format is M-d-y.

To learn more on changing the date format, see PHP DateTime.

Set the timezone in the format of your date

If necessary, you can also set the timezone of the date format.

To set the timezone as Berlin/Europe, modify your Twig to the following: {{ date |date("Ymd", "Europe/Berlin") }}.

For more information, see List of Supported Timezones.

Import yesterday’s file

You can also specify that you want to try and import a file from the past, and not today’s file.

To do so, use the following Twig: {{ date |date_modify("-1 day")|date("Ymd") }}

This searches for a file where the dynamic date element matches yesterday’s date.

You can adapt this time range by editing the -1 day element of Twig. Setting this as -7 day attempts to import the file where the dynamic date element matches the date from seven (7) days ago.

Import files with other dynamic aspects in the name

If the dynamic aspect of your filename is not a date, you can use one of the following methods. Each method is dependent on where you host your files:

  • HTTP link or FTP server

  • SFTP server

Import dynamic files from a link or FTP server using wildcards

To import files with dynamic aspects from an HTTP link or an FTP Server, you can use wildcards.

The wildcard symbol is an asterisk (*), which you use to replace dynamic elements. The wildcard ignores everything that comes in its place. It has no character match limit, so you only need to add one wildcard.

For example, if you're trying to import the following file, where the end of the address, adfc3424, is dynamic: ftp://my.ftpsource.io/files/datafeed-adfc3424.csv you should use a wildcard in the following manner: ftp://my.ftpsource.io/files/datafeed-*

Import dynamic files from an SFTP server

Importing from an SFTP server does not support the wildcard search.

For this reason, you should use the Dynamic SFTP Download data source. In the filename field, you can replace the dynamic element with {?}.

For example, to match the filename datafeed-adfc3424.csv, set the filename in the Dynamic SFTP Download data source to datafeed-{?}.

dynamics_ftp_source.png