OptionalautoOptionalclusteringList of column names to use for clustering the destination table. When specified, the destination Delta table will be clustered by these columns. This can improve query performance when filtering on these columns. Note: clustering_columns in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
OptionalenableWhether to enable auto clustering on the destination table. When enabled, Delta will automatically optimize the data layout based on the clustering columns for improved query performance. Note: enable_auto_clustering in table specific configuration will override the pipeline definition. Note: we can only provide enable_auto_clustering or clustering_columns, added as separate fields as we cannot have repeated field in oneof.
OptionalexcludeA list of column names to be excluded for the ingestion.
When not specified, include_columns fully controls what columns to be ingested.
When specified, all other columns including future ones will be automatically included for ingestion.
This field in mutually exclusive with include_columns.
OptionalincludeA list of column names to be included for the ingestion.
When not specified, all columns except ones in exclude_columns will be included. Future
columns will be automatically included.
When specified, all other future columns will be automatically excluded from ingestion.
This field in mutually exclusive with exclude_columns.
OptionalprimaryThe primary key of the table used to apply changes.
OptionalqueryOptionalrow(Optional, Immutable) The row filter condition to be applied to the table. It must not contain the WHERE keyword, only the actual filter condition. It must be in DBSQL format.
OptionalsalesforceIf true, formula fields defined in the table are included in the ingestion. This setting is only valid for the Salesforce connector
OptionalscdOptionalsequenceThe column names specifying the logical order of events in the source data. Spark Declarative Pipelines uses this sequencing to handle change events that arrive out of order.
OptionaltableTable properties to set on the destination table. These are key-value pairs that configure various Delta table behaviors or any user defined properties. Example: {"delta.feature.variantType": "supported", "delta.enableTypeWidening": "true"} Note: table_properties in table specific configuration will override the table_properties of the pipeline definition.
Optionalworkday(Optional) Additional custom parameters for Workday Report
(Optional, Mutable) Policy for auto full refresh, if enabled pipeline will automatically try to fix issues by doing a full refresh on the table in the retry run. auto_full_refresh_policy in table configuration will override the above level auto_full_refresh_policy. For example, { "auto_full_refresh_policy": { "enabled": true, "min_interval_hours": 23, } } If unspecified, auto full refresh is disabled.