Keep track of the data processing status at any stage
Discover best practices for getting notifications about data processing errors in Productsup.
Best practice
The platform may fail to process your data during import or export for multiple reasons, such as an empty import feed, a decreased or increased number of imported items, or an unreachable export destination.
Productsup can notify you if the data processing error occurs and show its exact reason to let you fix it.
We recommend the following actions to let you track data processing and let the platform automatically stop the processing in case of errors:
Open the Notifications panel whenever you run your site or start an import or export process. The panel shows the progress, errors, warnings, and data processing completion messages.
Set up necessary monitoring events in Monitor. The platform will stop the data processing and notify you via email, Slack, or Microsoft Teams whenever an error occurs.
Create a separate group email in your company to receive the monitoring notifications so that all who need to know of issues during the data processing can receive the monitoring emails. This best practice lies outside of Productsup, and you need to set up the group email with your email provider.
What features to use
To keep track of data processing, you can use the following features:
The Notifications panel available in the top right-hand corner of your view.
The Monitor feature available from the main menu.
You can select from over 25 monitoring events for the import, intermediate, and export stages, as well as general events that cover all stages. See the Monitor events overview to learn about all possible monitoring events.
How to set it up
Track data processing in the Notification panel
During an import, export, or run, select the bell icon in the top-right corner of your view. The Notifications panel is available on most pages in the platform next to the Run button.
The Notifications panel unfolds at the bottom of the page.
Set up the monitoring events
You can set up the monitoring events on the organization, project, or site level in Monitor.
See Set up Monitor for your feeds to learn how to set up a monitoring event.
As a common recommendation, you can start with the following events and settings on the organization level. After setting up monitoring events on the organization level, you can add more granular events on the project and site levels.
Monitoring events for the Import stage
High-priority events:
Import failed for Data Source – A data source couldn't import products, or the import feed was empty and contained no records.
Set up the event as follows:
Enable Stop Processing.
Enable and add an email in Email Notification or activate Slack Notification for your email group.
Import file failed to download or is empty (first main feed) – The platform couldn't import the feed from your first main data source, or the feed contained no data. There could be an invalid URL, or an error could have occurred during the file creation.
Set up the event as follows:
Enable Stop Processing.
Enable and add an email in Email Notification or activate Slack Notification for your email group.
Mapped attribute has been removed from Data Sources – A mapped attribute has disappeared from a data source. This may indicate errors on the data source side caused by file creation during the processes. This can also mean an issue with appending an additional data source to your main feed.
Set up the event as follows:
Enable Stop Processing.
Enable and add an email in Email Notification or activate Slack Notification for your email group.
Number of imported items decreased – The number of imported items has decreased compared to the last import.
Set up the event as follows:
Set Threshold(%) to
20%.Enable Stop Processing.
Enable and add an email in Email Notification or activate Slack Notification for your email group.
Medium-priority events:
Number of Imports without a change in Data Sources – The contents of a data source haven't changed in a given number of runs. This may indicate that the file updating process doesn't work or the URL is outdated.
Set up the event as follows:
Set Threshold(runs) to
3.Enable and add an email in Email Notification or activate Slack Notification for your email group.
Number of days without a change in Data Sources – The content of a data source hasn't changed within a specified number of days. This can indicate that the file updating process doesn't work or the URL is outdated.
Set up the event as follows:
Set Threshold(days) to
2.Enable and add an email in Email Notification or activate Slack Notification for your email group.
Low-priority events:
New attribute added to Data Sources – A new attribute appeared in one of the data sources, or you set up a data source and added new attributes to the feed.
Set up the event as follows:
Enable and add an email in Email Notification or activate Slack Notification for your email group.
Number of Import items increased – The number of imported items has increased compared to the last import.
Set up the event as follows:
Set Threshold(%) to
20%.Enable and add an email in Email Notification or activate Slack Notification for your email group.
Number of import file download retries – The number of retries the platform performs to import a file if the previous import attempts failed. You can use this monitoring event if the connection to a data source is unstable.
Set up the event as follows:
Set Threshold(attempts) to
2.
Monitoring events for the Export stage
High-priority events:
Feed upload failed – The export failed. For example, if an export destination is unreachable or the credentials are wrong.
Set up the event as follows:
Enable and add an email in Email Notification or activate Slack Notification for your email group.
Fewer items exported – The platform exported fewer items than during the last run. This could be due to fewer items in the feed, skip rules, or ROI rules.
Set up the event as follows:
Select On All Export Channels.
Set Threshold(%) to
25%.Enable Stop Processing.
Enable and add an email in Email Notification or activate Slack Notification for your email group.
More Products Exported – The platform exported more items than during the last run. This could indicate adding more items to the feed or deactivating skip or ROI rules active in the previous run.
Set up the event as follows:
Select On All Export Channels.
Set Threshold(%) to
25%.Enable and add an email in Email Notification or activate Slack Notification for your email group.
Monitoring events - General
Site run exceeds given number of hours – A site run exceeds a given time limit. For example, this could be due to applying new rule boxes in Dataflow or Data View or a slow speed while downloading data from a data source or uploading data to an export destination. The given number of hours equals the previous durations. Consider your previous run durations to set an adequate threshold. Apply this event to data sets of over 10 million products.
Set up the event as follows:
Select On All Export Channels.
Set Threshold(hours) to
10. The number may vary depending on the number of products in your site.Enable and add an email in Email Notification or activate Slack Notification to your email group.