Interface PipelinesInitScriptInfo

Config for an individual init script

interface PipelinesInitScriptInfo {
    storageInfo?:
        | { $case: "dbfs"; dbfs: PipelinesDbfsStorageInfo }
        | { $case: "s3"; s3: PipelinesS3StorageInfo };
}

Index

Properties

Properties

`Optional`storageInfo

storageInfo?:
| { $case: "dbfs"; dbfs: PipelinesDbfsStorageInfo }
| { $case: "s3"; s3: PipelinesS3StorageInfo }

Type Declaration

{ $case: "dbfs"; dbfs: PipelinesDbfsStorageInfo }
- $case: "dbfs"
- dbfs: PipelinesDbfsStorageInfo
  destination needs to be provided. e.g. { "dbfs" : { "destination" : "dbfs:/init-scripts/my_script.sh" } }
{ $case: "s3"; s3: PipelinesS3StorageInfo }
- $case: "s3"
- s3: PipelinesS3StorageInfo
  destination and either region or endpoint should also be provided. e.g. { "s3": { "destination" : "s3://init-scripts/my_script.sh", "region" : "us-west-2" } } Cluster iam role is used to access s3, please make sure the cluster iam role in instance_profile_arn has permission to write data to the s3 destination.