Consuming Parquet pipeline files

When enabled for your environment, Twisp writes committed ledger changes to tenant-isolated Parquet files and exposes them through the Files API. No managed Redshift warehouse is required.

File layout

Files use hourly UTC partitions:

Copywarehouse/parquet/YYYY/MM/DD/HH/<entity>/<batch>-NNNN.parquet

Available entities are journal, account, account_context, account_set, account_set_member, tran_code, transaction, entry, balance, calculation, velocity_control, velocity_limit, transaction_exception, and workflow_execution.

List a narrow, preferably hourly, prefix:

Copyquery ListParquetFiles($pageToken: String) {
  files {
    listPage(
      keyPrefix: "warehouse/parquet/2026/07/29/10/entry/"
      pageSize: 250
      pageToken: $pageToken
    ) {
      keys {
        key
        download {
          downloadURL
        }
      }
      nextPageToken
    }
  }
}

Repeat the query with nextPageToken until it is null. The older files.list field is deprecated because it loads every matching object into one response. Select download when you need a five-minute presigned URL for each listed key; omit it when you only need the file inventory.

You can also create a fresh download URL for any returned key with a mutation:

Copymutation DownloadParquetFile {
  files {
    createDownload(key: "warehouse/parquet/2026/07/29/10/entry/seed-000003-0000.parquet") {
      downloadURL
      downloadURLExpiration
      downloadHeaders
      contentType
    }
  }
}

CDC semantics

The files contain change data, not latest-state snapshots. Every committed record version carries the warehouse metadata columns record_begin, record_rowid, record_status, record_tenantid, and record_version. Process deletes according to record_status and deduplicate on (record_rowid, record_version).

A historical seed represents records exported at time T. The live pipeline should be enabled before creating the seed export. Seed and live files will then overlap slightly instead of leaving a gap; the normal version deduplication removes that overlap.

The hour in a seed key is the export time. Live file hours are their delivery partitions. Consumers should checkpoint successfully processed object keys and continue polling new hourly prefixes.

Raw delivery-stream inputs expire after 30 days. Parquet retention is configured separately by agreement with the customer.