Databricks SDK for JavaScript
    Preparing search index...

    Interface AutoLoader

    Ingests files from cloud storage with Auto Loader. Mutually exclusive with unity_catalog_connection and existing_table.

    interface AutoLoader {
        credentials?:
            | { $case: "serviceCredentials"; serviceCredentials: string }
            | { $case: "awsSecretScope"; awsSecretScope: string };
        effectiveIncludeExistingFiles?: boolean;
        effectiveRawTable?: TableName;
        envelope?: Envelope;
        format?: AutoLoader_Format;
        formatOptions?: Record<string, string>;
        includeExistingFiles?: boolean;
        location?: string;
        overridePartitionColumns?: string[];
        queueUrl?: string;
        rawTable?: TableName;
    }
    Index

    Properties

    credentials? effectiveIncludeExistingFiles? effectiveRawTable? envelope? format? formatOptions? includeExistingFiles? location? overridePartitionColumns? queueUrl? rawTable?

    Properties

    credentials?:
        | { $case: "serviceCredentials"; serviceCredentials: string }
        | { $case: "awsSecretScope"; awsSecretScope: string }

    AWS credentials for accessing the source files in Amazon S3, the Amazon SQS queue set in queue_url, or both. Set at most one.

    Type Declaration

    • { $case: "serviceCredentials"; serviceCredentials: string }
      • $case: "serviceCredentials"
      • serviceCredentials: string

        Name of the Unity Catalog service credential to use.

    • { $case: "awsSecretScope"; awsSecretScope: string }
      • $case: "awsSecretScope"
      • awsSecretScope: string

        Name of the workspace secret scope that holds the AWS access key ID and secret access key, under the keys awsAccessKeyId and awsSecretAccessKey.

    effectiveIncludeExistingFiles?: boolean

    Whether ingestion started with the files already present at location. Populated on get, create, and update responses.

    effectiveRawTable?: TableName

    The table that the raw data is written to: raw_table in the datasource's catalog and schema, under their current Unity Catalog names. Populated on get, create, and update responses.

    envelope?: Envelope

    How to unwrap a transport envelope around the source records before parsing. Supported only for the JSON and JSONL formats. If unset, each source record is treated as one event.

    format?: AutoLoader_Format

    File format of the source files. Required.

    formatOptions?: Record<string, string>

    Format-specific parsing options, passed to Auto Loader. Supported for the CSV, JSON, JSONL, and XML formats, each of which accepts a fixed set of options. The XML format requires the rowTag option.

    includeExistingFiles?: boolean

    Whether to ingest the files already present at location when ingestion first starts. Set true to ingest the existing files and then every new file. Set false, or leave unset, to ingest only files that arrive after the datasource starts. Defaults to false. Cannot be changed after creation. Reads return the value as supplied; the setting in effect is reported in effective_include_existing_files.

    location?: string

    URI of the source files: a path in a Unity Catalog volume or external location, or a cloud storage URL such as an Amazon S3 path. Required, including when new files are discovered through queue_url. Cannot be changed after creation.

    overridePartitionColumns?: string[]

    Hive-style partition columns to derive from the source file paths. If unset, no partition columns are derived.

    queueUrl?: string

    URL of a customer-managed Amazon SQS queue that receives file-arrival notifications. Set it to discover new files through the queue. If unset, new files are discovered through the file events of the source's Unity Catalog external location when it has them configured, and by directory listing otherwise. Cannot be changed after creation.

    rawTable?: TableName

    Table that stores the raw data before parsers schematize it. Every Auto Loader datasource writes its raw rows to this table before any parser reads them. The table must be in the datasource's catalog and schema; an omitted catalog_name or schema_name defaults to them. Cannot be changed after creation. Reads return the value as supplied; the table the data is written to is reported in effective_raw_table.

    Settings

    Member Visibility

    On This Page

    Properties