Buckets

Persistent key-value storage your connector can read and write across runs, shared by every site running the connector.

Buckets give your connector storage that outlives a single run. Run input and output files exist only for the run that produced them; a bucket object stays until you overwrite or delete it.

What a bucket is

A bucket is a key-value store your connector reads and writes through the Container API. You choose the keys, and the value is whatever bytes you send — JSON, CSV, a serialized cursor, a token.

Two properties shape almost every decision you make with them:

Objects persist between runs. A value written in one run is readable in the next, days or weeks later.

One bucket serves the whole connector environment. Every site running your connector in that environment reads and writes the same bucket. Buckets are not scoped per site.

That second property cuts both ways, and which way it cuts is entirely down to how you name your keys.

Because all sites share one bucket, an unqualified key like state.json is a single value that every site overwrites in turn. The run environment exposes the current site in the SITE environment variable — include it in the key whenever a value belongs to one site:

# Per-site state: each site keeps its own value
state-${SITE}.json

# Shared state: deliberately visible to every site
catalog-version.json

Both patterns are legitimate. Per-site keys are what you want for a cursor, a watermark, or a last-synchronized timestamp. A shared key is useful when sites genuinely need to coordinate — a reference dataset fetched once and reused, or a value one site publishes for the others to read.

Decide which of the two you are using before you write the first key. Discovering later that several sites have been overwriting one value is difficult to untangle, because the earlier values are gone.

Keys accept letters, digits, underscores, hyphens, and dots, up to 512 characters. A key cannot begin with a dot and cannot end with a dot. Keys passed to the streaming endpoints cannot contain a forward slash — use a separator such as - or . to structure them.

Link a bucket to your connector

A connector has no bucket until one is created and linked to it. Until then, calls to the bucket endpoints fail.

Create the bucket

Call Create bucket on the CDE API. The response contains the bucket identifier.

Link it to a connector version

Call Link Bucket with Connector with that identifier. A link applies to one connector type and one environment, so a connector that needs a bucket in more than one environment needs a link for each.

Verify the link

Call List buckets to confirm the bucket exists and is linked. Runs started after linking can use it; a run already in progress cannot.

Read and write objects

The Container API exposes a bucket as two streaming endpoints that carry raw bytes in and raw bytes out. There is no request or response envelope — serialization is your choice.

Operation	Endpoint
Write a value	`PUT /buckets/stream/{key}`
Read a value	`GET /buckets/stream/{key}`

# Write
curl -X PUT "$CONTAINER_API/buckets/stream/state-${SITE}.json" \
  --data-binary '{"last_synchronized":"2026-09-09T10:00:00Z"}'

# Read
curl "$CONTAINER_API/buckets/stream/state-${SITE}.json"

In PHP, the SDK wraps both on ContainerApiInterface:

$key = sprintf('state-%s.json', getenv('SITE'));

// Write a string
$containerApi->streamDataToInternalStorageKey(
    json_encode(['last_synchronized' => $now->format(DATE_ATOM)]),
    $key
);

// Write a stream, for values too large to hold in memory
$containerApi->streamToInternalStorageKey($stream, $key);

// Read
$previous = json_decode(
    (string) $containerApi->streamFromInternalStorageKey($key),
    true
);

The SDK names these methods after internal storage rather than buckets — streamDataToInternalStorageKey, streamToInternalStorageKey, and streamFromInternalStorageKey are the bucket methods. Searching the SDK for "bucket" finds the deprecated methods instead.

Handle a key that does not exist

Reading a key your connector has never written returns an error response, not an empty body and not a 404 you can treat as a clean miss. Any unsuccessful read means "no value yet", and your connector needs a path for it:

try {
    $previous = json_decode((string) $containerApi->streamFromInternalStorageKey($key), true);
} catch (\Throwable) {
    $previous = null; // First run for this site, or the value was removed
}

Design that branch as a real behavior rather than an error case. A connector that cannot read its previous state usually needs to fall back to a full synchronization, and it will take that path on its first run for every site.

Deprecated endpoints

Three older bucket endpoints remain in the API and should not be used in new connectors. They move files through the run exchange folder instead of streaming, and the streaming endpoints replace them.

Deprecated	Use instead
`PUT /buckets/exchange/upload/{remoteFile}`	`PUT /buckets/stream/{key}`
`GET /buckets/exchange/download/{remoteFile}`	`GET /buckets/stream/{key}`
`DELETE /buckets/delete/{remoteFile}`	No streaming equivalent

In the PHP SDK these correspond to uploadObjectFromExchangeFolderToBucket, downloadObjectFromBucketToExchangeFolder, and deleteObjectFromBucket.

Deleting has no streaming replacement. From inside a run, write the key again to replace its value. To remove an object outright, use Delete file from bucket on the CDE API.