# Consuming Parquet pipeline files

List and download near-real-time ledger change data in Parquet format.


Source: https://www.twisp.com/docs/tutorials/advanced/parquet-pipeline

When enabled for your environment, Twisp writes committed ledger changes to
tenant-isolated Parquet files and exposes them through the Files API. No managed
Redshift warehouse is required.

## File layout

Files use hourly UTC partitions:

```text
warehouse/parquet/YYYY/MM/DD/HH/<entity>/<batch>-NNNN.parquet
```

Available entities are `journal`, `account`, `account_context`, `account_set`,
`account_set_member`, `tran_code`, `transaction`, `entry`, `balance`,
`calculation`, `velocity_control`, `velocity_limit`, `transaction_exception`,
and `workflow_execution`.

List a narrow, preferably hourly, prefix:

```graphql
query ListParquetFiles($pageToken: String) {
  files {
    listPage(
      keyPrefix: "warehouse/parquet/2026/07/29/10/entry/"
      pageSize: 250
      pageToken: $pageToken
    ) {
      keys {
        key
        download {
          downloadURL
        }
      }
      nextPageToken
    }
  }
}
```

Repeat the query with `nextPageToken` until it is `null`. The older `files.list`
field is deprecated because it loads every matching object into one response.
Select `download` when you need a five-minute presigned URL for each listed key;
omit it when you only need the file inventory.

You can also create a fresh download URL for any returned key with a mutation:

```graphql
mutation DownloadParquetFile {
  files {
    createDownload(key: "warehouse/parquet/2026/07/29/10/entry/seed-000003-0000.parquet") {
      downloadURL
      downloadURLExpiration
      downloadHeaders
      contentType
    }
  }
}
```

## CDC semantics

The files contain change data, not latest-state snapshots. Every committed record
version carries the warehouse metadata columns `record_begin`, `record_rowid`,
`record_status`, `record_tenantid`, and `record_version`. Process deletes according
to `record_status` and deduplicate on `(record_rowid, record_version)`.

A historical seed represents records exported at time T. The live pipeline should
be enabled before creating the seed export. Seed and live files will then overlap
slightly instead of leaving a gap; the normal version deduplication removes that
overlap.

The hour in a seed key is the export time. Live file hours are their delivery
partitions. Consumers should checkpoint successfully processed object keys and
continue polling new hourly prefixes.

Raw delivery-stream inputs expire after 30 days. Parquet retention is configured
separately by agreement with the customer.