Streams

This section introduces a concept called Streams. Streams are spaces where we can receive ongoing flows of structured data. Using Streams gives our clients the following possibilities:

Importing and merging data into a single site from different systems using multiple Streams.

Importing data from a single Stream into multiple sites in order to reuse the data without uploading it multiple times.

Allowing the clients to create and merge Streams by themselves.

Stream creation

You can create a maximum of 100 Streams per account. If you need more Streams, reach out to your Client Success Manager.

Stream naming

Stream names should be unique, and their length can be from 3 to 255 characters.

If you delete a Stream via API, it remains in our system for 7 days. During this period, you can no longer access the data of the deleted Stream, but its name remains reserved. When 7 days have passed, the system completely deletes the Stream, and its name becomes available for new Streams.

Stream formats

Streams support different structured data types. For now, you can use the following formats for the input, but we may add more formats in the future:

Classic JSON

Newline Delimited JSON (NDJSON)

Stream types: Chunked & Referenced

The Stream API supports two Stream types to handle data:

Chunked

Referenced

Chunked type

The chunked type provides high throughput and upload speeds.

We maintain the following hard-limits on payloads:

The maximum size of a single product is 1 MB.

The platform automatically rejects any product exceeding 1 MB.
You must split large data into chunks of up to 1 MB.

The total payload size shouldn't exceed 200 MB, either uncompressed or compressed.

The chunked type supports the NDJSON format, which structure lets the platform keep the object deserialization simple and lets process more data.

Referenced type

The referenced type fits the clients who send extremely large quantities of data.

Your request payload sent compressed or uncompressed, can't be greater than 200 MB.

The referenced type supports both JSON and NDJSON format, though it's not as optimized as the chunked type for throughput and speed.

Which is the best type for your use case?

Answer the following questions to determine which Stream type is best for you.

Do you plan to send single product updates? Single product updates mean the platform will send requests for individual products. If yes, use the chunked type.

Do you plan to send frequent product updates? Frequent product updates mean that the platform pushes multiple updates per minute: single-product or batched updates. If yes, use the chunked type.

Do you work with delta-supporting destinations? Some destinations, such as GMC or Facebook Marketplaces, support delta updates. If yes, use the chunked type. As we continuously improve our systems, our latest end-to-end delta functionality lets you achieve a near-real-time pipeline and a much smoother integration.

Do you have a large catalog of over 10 million products and send delta updates? If yes, use the chunked type. The platform imports delta changes faster from the chunked Streams.

Are you going to send only full catalog updates? If yes, continue with the following questions to get more precise recommendations:

Is your catalog size over 10 million products?

If yes, use the referenced type.
Do you import your data no more than 1-2 times per day?

If yes, use the referenced type.

If you don't have a definite answer, please use the chunked type.