Manage dynamic date components in URL data sources

If you import by URL and the contain a dynamic date section, you can import dynamic per data via a twig to the URL in the Data Sources.

Assigning the twig the current date enables it to search for the file with that date in the directory the Productsup platform pulls your feed from during import. As a result, the Productsup platform only pulls the most recent file, even if the directory contains multiple files.

Amend the URL with today's date

The target URL may already contain a date component such as in this example:

http://www.yourdomain.com/feed/product/fullcatalog-20150729.txt

To pull a file from this URL dynamically, replace the date section 20150729 with "{{ date |date("Ymd") }}. "

Making the final URL :

http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date("Ymd") }}.txt

For the twig to function correctly, ensure the following:

  • There is no empty space before or after the function.

  • There is empty space inside the brackets, as in the example.

  • The scheduled import time in the Productsup platform matches the filename in your feed. If the Productsup platform attempts to import a feed early, the import fails as the file does not yet exist.

Add a date to import past or future data

Following the same syntax, you can pull a file with a date in the past or future:

  • For yesterday : "{{ date |date_modify("-1 day")|date("Ymd") }}"

  • For tomorrow: "{{ date |date_modify("+1 day")|date("Ymd") }}"

Example:

Today is July 27th:

http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date_modify("-1 day")|date("Ymd") }}.txt

The platform imports this URL:

http://www.yourdomain.com/feed/product/fullcatalog-20150726.txt

Import alternate date formats

If your data uses a different convention for time-date format than the other examples, you can adjust the twig accordingly:

Example: Today is July 27th. To import a file with "07-27" in the URL, use "m-d"

http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date_modify("-1 day")|date("m-d") }}.txt

The platform imports this URL:

http://www.yourdomain.com/feed/product/fullcatalog-07-27.txt

Important: Do not add hours, minutes, or seconds to the import URL. Since the exact time the Productsup platform initiates a pull from a URL varies, it is impossible to specify such a granular URL.

Include timezone information

If your data source operates at your local time, a conflict can occur during import since the Productsup platform's servers operate at UTC time. This conflict results from a failure to match the current day in the Productsup platform and that of your servers.

For example, if your data source produces files saved to the Berlin/Europe timezone, you need to adjust the twig to ensure the scheduled pull happens today at 00:30 Berlin time and not yesterday at 23:30 UTC).

Example:

http://www.yourdomain.com/feed/product/fullcatalog-20150729.txt

Replace the date part (20150729) with "{{ date |date("Ymd", "Europe/Berlin") }}".

The final URL is :

http://www.yourdomain.com/feed/product/fullcatalog-{{ date |date("Ymd", "Europe/Berlin") }}.txt

Add additional dynamic information

URLs can contain dynamic information beyond the date. For example, URLs can have hashes, job-ids, or more precise times in seconds. Pulling such dynamic information requires a different technique.

In the case of FTP and HTTP, replace the relevant elements with an asterisk (*) to act as a wildcard.

For example, the file link: ftp://my.ftpsource.io/files/datafeed-adfc3424.csv should have the following syntax in Data Sources: ftp://my.ftpsource.io/files/datafeed-*

This technique does not work with SFTP since SFTPs do not support wildcard search.

To pull dynamic information from SFTPs, use Dynamic SFTP Import in Data Source.

6149d5c60f7d9.png

Fill in the fields as follows:

- Enter your SFTP host address

- Enter the path containing the folder and target file, replacing the dynamic section of the path with {?}. For example: datafeed-{?}.

- Add the authentication credentials required to access your SFTP.

The Productsup platform accesses the SFTP and downloads the latest file matching the specified pattern on the next import.