Amazon S3

Store session data only in your own S3 bucket, with the console reading it back from there.

With S3 connected, new sessions are written to your bucket and nowhere else. When someone opens a session in the console, Surface reads it from your bucket. How long the data is kept becomes your bucket’s lifecycle policy.

Before you begin

An S3 bucket.
An access key and secret key that can GetObject, PutObject, DeleteObject, and ListBucket on it.
For MinIO, Ceph, or another S3-compatible service: its endpoint URL.

Connect

Open the setup panel

Hub, then the Amazon S3 card under Data Sources.

Choose the provider

Amazon Web Services for Amazon S3. S3-compatible (MinIO, Ceph) for MinIO, Ceph, and similar.

Fill in the required fields

Field	Notes
Bucket	Bucket name
Access key ID	Secret
Secret access key	Secret
Endpoint URL	S3-compatible only, for example `https://minio.internal:9000`

Test and save

Click Test Connection and wait for “Amazon S3 connection works”. Then click Save. The first save turns S3 on, and you see “Amazon S3 connected”.

Advanced options

Field	Default	Change it when
Prefix	`prompts`	You want a different top-level folder
Region	`us-east-1`	AWS only, and your bucket is elsewhere
Query engine	`parquet`	Set `athena` to query session data with SQL through Amazon Athena
Athena workgroup	`primary`	Athena engine only. Needs Athena engine version 3
Glue database	`surface_prompts`	Athena engine only. Created on first configure
Athena results location	`athena-results/` under the prefix	Athena engine only
Path-style addressing	on	S3-compatible only

Athena is Amazon’s service for running SQL over files in S3. If you are unsure, leave the engine as parquet and ask whoever owns your AWS account.

What lands in the bucket

<prefix>/transcripts/org_id=<your org>/<session id>/part-00000-00099-<digest>.json.gz
<prefix>/sessions/org_id=<your org>/<session id>/...parquet
<prefix>/messages/org_id=<your org>/<session id>/...parquet
<prefix>/usage/org_id=<your org>/<session id>/...parquet

transcripts holds the conversations. The other three folders hold summaries the console uses for lists and search. Set lifecycle rules on the prefix as a whole.

What happens next

The next session recorded on any endpoint is written to your bucket. Open it on the Sessions page: if the trace loads, reads from the bucket work. Sessions recorded before you enabled S3 stay in SuperAlign storage until they expire.

Next steps

Splunk to also send a copy to your index
Data handling for what retention now depends on