Databricks SDK for JavaScript
    Preparing search index...

    Interface IngestionPipelineDefinition

    interface IngestionPipelineDefinition {
        connectorType?: ConnectorType;
        dataStagingOptions?: DataStagingOptions;
        fullRefreshWindow?: OperationTimeWindow;
        netsuiteJarPath?: string;
        objects?: IngestionPipelineDefinition_IngestionConfig[];
        source?:
            | { $case: "connectionName"; connectionName: string }
            | { $case: "ingestionGatewayId"; ingestionGatewayId: string }
            | {
                $case: "ingestFromUcForeignCatalog";
                ingestFromUcForeignCatalog: boolean;
            };
        sourceConfigurations?: SourceConfig[];
        sourceType?: IngestionSourceType;
        tableConfiguration?: IngestionPipelineDefinition_TableSpecificConfig;
    }
    Index

    Properties

    connectorType? dataStagingOptions? fullRefreshWindow? netsuiteJarPath? objects? source? sourceConfigurations? sourceType? tableConfiguration?

    Properties

    connectorType?: ConnectorType

    (Optional) Connector Type for sources. Ex: CDC, Query Based.

    dataStagingOptions?: DataStagingOptions

    (Optional) Location of staged data storage. This is required for migration from Cdc Managed Ingestion Pipeline with Gateway pipeline to Combined Cdc Managed Ingestion Pipeline. If not specified, the volume for staged data will be created in catalog and schema/target specified in the top level pipeline definition.

    fullRefreshWindow?: OperationTimeWindow

    (Optional) A window that specifies a set of time ranges for snapshot queries in CDC.

    netsuiteJarPath?: string

    Netsuite only configuration. When the field is set for a netsuite connector, the jar stored in the field will be validated and added to the classpath of pipeline's cluster.

    objects?: IngestionPipelineDefinition_IngestionConfig[]

    Required. Settings specifying tables to replicate and the destination for the replicated tables.

    source?:
        | { $case: "connectionName"; connectionName: string }
        | { $case: "ingestionGatewayId"; ingestionGatewayId: string }
        | {
            $case: "ingestFromUcForeignCatalog";
            ingestFromUcForeignCatalog: boolean;
        }

    (Required, Mutable) Identifies the data source for the Lakeflow Connect Ingestion pipeline. Exactly one option must be specified.

    Type Declaration

    • { $case: "connectionName"; connectionName: string }
      • $case: "connectionName"
      • connectionName: string

        The Unity Catalog connection that this ingestion pipeline uses to communicate with the source. This is used with both connectors for applications like Salesforce, Workday, and so on, and also database connectors like Oracle, (connector_type = QUERY_BASED OR connector_type = CDC). If connection name corresponds to database connectors like Oracle, and connector_type is not provided then connector_type defaults to QUERY_BASED. If connector_type is passed as CDC we use Combined Cdc Managed Ingestion pipeline. Under certain conditions, this can be replaced with ingestion_gateway_id to change the connector to Cdc Managed Ingestion Pipeline with Gateway pipeline.

    • { $case: "ingestionGatewayId"; ingestionGatewayId: string }
      • $case: "ingestionGatewayId"
      • ingestionGatewayId: string

        Identifier for the gateway that is used by this ingestion pipeline to communicate with the source database. This is used with CDC connectors to databases like SQL Server using a gateway pipeline (connector_type = CDC). Under certain conditions, this can be replaced with connection_name to change the connector to Combined Cdc Managed Ingestion Pipeline.

    • { $case: "ingestFromUcForeignCatalog"; ingestFromUcForeignCatalog: boolean }
      • $case: "ingestFromUcForeignCatalog"
      • ingestFromUcForeignCatalog: boolean

        Immutable. If set to true, the pipeline will ingest tables from the UC foreign catalogs directly without the need to specify a UC connection or ingestion gateway. The source_catalog fields in objects of IngestionConfig are interpreted as the UC foreign catalogs to ingest from.

    sourceConfigurations?: SourceConfig[]

    Top-level source configurations

    sourceType?: IngestionSourceType

    The type of the foreign source. The source type will be inferred from the source connection or ingestion gateway. This field is output only and will be ignored if provided.

    tableConfiguration?: IngestionPipelineDefinition_TableSpecificConfig

    Configuration settings to control the ingestion of tables. These settings are applied to all tables in the pipeline.

    Settings

    Member Visibility

    On This Page

    Properties