Uploading Data

How to upload product data to the Stream API, including request formats, stream status, and attribute requirements.

See the chapter on Stream types for more information on the differences between both types and also payload recommendations.

In general, our main recommendation is to go with the Chunked Stream type. This type is optimized for high throughput and speed.

There is a single endpoint to upload data to Productsup: /streams/{streamId}/products

The endpoint to upload data is not REST compatible, therefore the methods POST, PATCH, and PUT all produce the same behavior.

We maintain strict rate limits on all our upload endpoints. See Rate limits for more information. We recommend you implement a retry and exponential backoff mechanism.

Request body

Example 1: Chunked Stream - NDJSON with one product

curl --location --request POST 'https://stream-api.productsup.com/streams/124773/products' \
--header 'Content-Type: application/x-ndjson' \
--header 'Accept: application/vnd.api+json' \
--data-raw '{"id":"34SKJDF42DF","name":"Product 1","company":"My Company"}'

Example 2: Chunked Stream - NDJSON with multiple products

curl --location --request POST 'https://stream-api.productsup.com/streams/124773/products' \
--header 'Content-Type: application/x-ndjson' \
--header 'Accept: application/vnd.api+json' \
--data-binary @- <<EOF
  {"id":"34SKJDF42DF","name":"Product 1","company":"MyCompany"}
  {"id":"475-SHIRT-XL","name":"Product 2","company":"My Company","size":"XL"}
  {"id":7824796324,"name":"Product 3","company":"My Company","price":5,"sale_price":4.5}
EOF

Example 3: Referenced Stream - NDJSON with multiple products

curl --location --request POST 'https://stream-api.productsup.com/streams/224773/products' \
--header 'Content-Type: application/x-ndjson' \
--header 'Accept: application/vnd.api+json' \
--data-binary @- <<EOF
  {"id":"34SKJDF42DF","name":"Product 1","company":"MyCompany"}
  {"id":"475-SHIRT-XL","name":"Product 2","company":"My Company","size":"XL"}
  {"id":7824796324,"name":"Product 3","company":"My Company","price":5,"sale_price":4.5}
EOF

Example 4: Referenced Stream - JSON with one product

curl --location --request POST 'https://stream-api.productsup.com/streams/224773/products' \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.api+json' \
--data-raw '[{"id":"34SKJDF42DF","name":"Product1","company":"My Company"}]'

Example 5: Referenced Stream - JSON with 3 products

curl --location --request POST 'https://stream-api.productsup.com/streams/224773/products' \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.api+json' \
--data-binary @- <<EOF [
  {"id":"34SKJDF42DF","name":"Product 1","company":"MyCompany"},
  {"id":"475-SHIRT-XL","name":"Product 2","company":"My Company","size":"XL"},
  {"id":7824796324,"name":"Product 3","company":"My Company","price":5,"sale_price":4.5}
]
EOF

Example 6: Referenced Stream - JSON with compressed products

curl --location --request POST 'https://stream-api.productsup.com/streams/224773/products' \
--header 'Content-Type: application/json' \
--header 'Accept: application/vnd.api+json' \
--header 'Authorization: Bearer pup_token' \
--header 'Content-Encoding: gzip' \
--data-binary '@/path/to/compressed_file'

Example 7: Chunked Stream - JSON with compressed products

curl --location --request POST 'https://stream-api.productsup.com/streams/224773/products' \
--header 'Content-Type: application/x-ndjson' \
--header 'Accept: application/vnd.api+json' \
--header 'Authorization: Bearer pup_token' \
--header 'Content-Encoding: gzip' \
--data-binary '@/path/to/compressed_file'

The type of the Stream dictates the allowed payload format:

Chunked Streams accept application/x-ndjson

Referenced Streams accept application/json or application/x-ndjson

Accompany every request with the correct Content-Type header. The platform does not support the mixing of different content types in a single Stream.

Productsup supports gzip compression for both types of Streams to improve performance further. Use the Content-Encoding header to specify gzip compression for request payloads.

Nested structures or non-scalar values are not supported in neither NDJSON nor JSON content-types; we recommend:

Flattening any nested structure into a single level object
Non-scalar should be concatenated or spread over multiple attributes

Response depending on stream status

The stream status feature is essential for maintaining and protecting our system's performance. It lets our data source actively monitor stream status, enabling administrators and automated processes to manage streams and prevent potential system overload responsibly.

Status	Description
active	Default value
disabled	Stream is disabled and rejects uploads. HTTP app returns 403 Forbidden status, and the platform doesn't trigger import.
locked	Stream is temporarily locked. HTTP app returns 423 Locked status, and the platform prohibits data upload and import.
locked_upload	Stream is temporarily locked for uploading. HTTP app returns 423 Locked status.
locked_processing	Stream is temporarily locked for importing.

Product attribute requirements

Incompatible nested JSON payload:

{
  "name": "Alice",
  "age": 30,
  "address": {
    "street": "123 Main St",
    "city": "Springfield",
    "zip": "12345"
  },
  "contact_details": {
    "email": "alice.springfield@example.com",
    "phone_numbers": [
      {
        "type": "mobile",
        "number": "+1-555-555-5555"
      },
      {
        "type": "home",
        "number": "+1-555-555-1155"
      }
    ]
  },
  "hobbies": ["reading", "hiking", "chess"]
}

Compatible flattened JSON payload:

{
  "name": "Alice",
  "age": 30,
  "address_street": "123 Main St",
  "address_city": "Springfield",
  "address_zip": "12345",
  "contact_email": "alice.springfield@example.com",
  "contact_phone_mobile": "+1-555-555-5555",
  "contact_phone_home": "+1-555-555-1155",
  "hobbies": "reading,hiking,chess"
}

Compatible flattened NDJSON payload:

{"name":"Alice","age":30,"address_street":"123 Main St","address_city":"Springfield","address_zip":"12345","contact_email":"alice.springfield@example.com","contact_phone_mobile":"+1-555-555-5555","contact_phone_home":"+1-555-555-1155","hobbies":"reading,hiking,chess"}

To accommodate to our clients data sets, we don't enforce a specific standard set of attributes or naming conventions.

Stream API accepts only flat JSON objects and attributes with scalar values.

We recommend the following rules for naming and handling attributes:

Ideally attributes are lowercase and do not contain any spaces or special characters.

Nested structures or non-scalar values are not supported in neither NDJSON nor JSON content-types; we recommend:

Flattening any nested structure into a single level object
Non-scalar should be concatenated or spread over multiple attributes

Internally, we rely on the SQLite format so we are bound to its technical limits. All data is stored in the SQLite TEXT data type.

When you add new attributes in future requests:

New attributes horizontally expand existing data with empty values

When attributes are not part of the upload anymore:

If any existing row has a value for that attribute, the attribute remains in existence.
Any new row uploaded without the attribute will have an empty value

If an attribute clean up is needed, we recommend you to remove all rows and start with a clean dataset.

ID-attribute tips

The id-attribute is the only mandatory attribute.

We always require that an id-attribute is present. This is our golden rule. The id-attribute is a unique identifier to your specific data row and the platform uses the id-attribute to create new rows, apply permutations, and delete specific rows.

Boolean and null value handling

Boolean and null values undergo automatic conversion when stored:

Value	Type	Value after conversion
true	boolean	`1`
false	boolean	`''` (empty string)
null	null	`''` (empty string)

To avoid unexpected conversions and maintain data integrity, we strongly recommend sending boolean values as string literals 'true' or 'false'.

Recommended boolean handling:

{
  "id": "123",
  "name": "Product Name",
  "in_stock": "true",
  "on_sale": "false"
}

Site Stream Data Sources

The data source management endpoint allows you to link your streams to sites programmatically. You can create Stream API datasources and link them to sites, update them, delete or get more information about existing links.

The authentication for this endpoint is unified, i.e. the new Stream API Personal Access Token authentication can be used to access functionality from the Platform API.