Interface IngestionConfig

The ID of the Databricks Job that performs the historical backfill of the ingestion Delta table.

A user-provided source for backfilling data. Historical data is used when creating a training set from streaming features linked to this Stream. The backfill data stored in this location will be copied into the ingestion table for offline querying and training. The schema for this source must match exactly that of the key and payload schemas specified for this Stream, except that it may omit any columns listed in excluded_columns.

The ID of the budget policy used to attribute the serverless compute cost of this stream's managed ingestion. If not specified, a default budget policy may be applied.

Column paths used to identify duplicate rows during ingestion; only one row per distinct combination of these values is kept. Use dot notation for nested fields (e.g. value.user_id). Empty list means every column is compared.

Destination for the -managed Delta table that holds an offline copy of the streaming data for querying and training. This table contains both 1) forward-filled data from the Stream and 2) backfilled data from the BackfillSource (if provided). This table is created and managed by and is deleted when the Stream is deleted.

The ID of the Databricks Job that performs the forward-fill ingestion.

The ID of the SDP pipeline that continuously copies new events from the streaming source into the ingestion Delta table.

Custom tags to associate with this stream's managed ingestion. They are applied to the ingestion pipeline and its forward-fill and backfill jobs, and forwarded to the underlying compute as cluster tags, so ingestion cost can be attributed in the billing system tables. These tags apply only to the managed ingestion compute; they are not applied to the Stream entity itself, and are distinct from any Unity Catalog tags on the Stream. A maximum of 25 tags is supported; keys and values are subject to the same limitations as cluster tags.