Interface SdpSource

Schema of the resulting dataframe after transformations, in Spark StructType JSON format (from df.schema.json()). Required if transformation_sql is specified. Example: {"type":"struct","fields":[{"name":"col_a","type":"integer","nullable":true,"metadata":{}},{"name":"col_c","type":"integer","nullable":true,"metadata":{}}]}

Single WHERE clause to filter the pipeline's output table before applying transformations. Will be row-wise evaluated, so should only include conditionals and projections.

The full three-part names of the tables the pipeline reads from. Required to materialize this feature with a TableTrigger, which starts a run when these tables update. Each table must exist when the feature is created. Feature Store doesn't confirm that the pipeline reads from these tables. At most 25 tables.

The full three-part (catalog, schema, table) name of the pipeline's output table to read from. The table doesn't need to exist until the pipeline first runs. Required.

The ID of the Spark Declarative Pipeline that Feature Store runs to produce this source, before each materialization or backfill. The pipeline must already exist when the feature is created, and you need CAN_RUN, CAN_MANAGE, or owner permission on it. It doesn't need to have run yet. Deleting the feature doesn't delete the pipeline. If the pipeline is deleted, this field reads back empty. Required.

State tracking pipeline execution and output table validation. PENDING until the pipeline has run at least once and created the output table; then RESOLVED on success, or ERROR on failure.

A single SQL SELECT expression applied after filter_condition. Should contain all the columns needed (eg. "SELECT , col_a + col_b AS col_c FROM x.y.z WHERE col_a > 0" would have transformation_sql ", col_a + col_b AS col_c") If transformation_sql is not provided, all columns of the pipeline's output table are present in the DataSource dataframe.