Class PipelinesClient

Index

Constructors

Methods

applyEnvironment clone create delete edit events eventsIter get getUpdate list listIter listUpdates start stop

Constructors

constructor

new PipelinesClient(options: ClientOptions): PipelinesClient
Parameters
- options: ClientOptions
Returns PipelinesClient
- Defined in client.ts:91

Methods

applyEnvironment

applyEnvironment(
req: ApplyEnvironmentRequest,
options?: CallOptions,
): Promise<ApplyEnvironmentResponse>
Applies the current pipeline environment onto the pipeline compute. The environment applied can be used by subsequent dev-mode updates.
Parameters
- req: ApplyEnvironmentRequest
- Optionaloptions: CallOptions
Returns Promise<ApplyEnvironmentResponse>
- Defined in client.ts:111

clone

clone(
req: ClonePipelineRequest,
options?: CallOptions,
): Promise<ClonePipelineResponse>
Creates a new pipeline using Unity Catalog from a pipeline using Hive Metastore. This method returns the ID of the newly created clone. Additionally, this method starts an update for the newly created pipeline.
Parameters
- req: ClonePipelineRequest
- Optionaloptions: CallOptions
Returns Promise<ClonePipelineResponse>
- Defined in client.ts:145

create

create(
req: CreatePipelineRequest,
options?: CallOptions,
): Promise<CreatePipelineResponse>
Creates a new data processing pipeline based on the requested configuration. If successful, this method returns the ID of the new pipeline.
Parameters
- req: CreatePipelineRequest
- Optionaloptions: CallOptions
Returns Promise<CreatePipelineResponse>
- Defined in client.ts:178

delete

delete(
req: DeletePipelineRequest,
options?: CallOptions,
): Promise<DeletePipelineResponse>
Deletes a pipeline. If the pipeline publishes to Unity Catalog, pipeline deletion will cascade to all pipeline tables. Please reach out to support for assistance to undo this action.
Parameters
- req: DeletePipelineRequest
- Optionaloptions: CallOptions
Returns Promise<DeletePipelineResponse>
- Defined in client.ts:211

edit

edit(
req: EditPipelineRequest,
options?: CallOptions,
): Promise<EditPipelineResponse>
Updates a pipeline with the supplied configuration.
Parameters
- req: EditPipelineRequest
- Optionaloptions: CallOptions
Returns Promise<EditPipelineResponse>
- Defined in client.ts:249

events

events(
req: ListPipelineEventsRequest,
options?: CallOptions,
): Promise<ListPipelineEventsResponse>
Retrieves events for a pipeline.
Parameters
- req: ListPipelineEventsRequest
- Optionaloptions: CallOptions
Returns Promise<ListPipelineEventsResponse>
- Defined in client.ts:279

eventsIter

eventsIter(
req: ListPipelineEventsRequest,
options?: CallOptions,
): AsyncGenerator<PipelineEvent>
Parameters
- req: ListPipelineEventsRequest
- Optionaloptions: CallOptions
Returns AsyncGenerator<PipelineEvent>
- Defined in client.ts:322

get

get(
req: GetPipelineRequest,
options?: CallOptions,
): Promise<GetPipelineResponse>
Get a pipeline.
Parameters
- req: GetPipelineRequest
- Optionaloptions: CallOptions
Returns Promise<GetPipelineResponse>
- Defined in client.ts:340

getUpdate

getUpdate(
req: GetUpdateRequest,
options?: CallOptions,
): Promise<GetUpdateResponse>
Gets an update from an active pipeline.
Parameters
- req: GetUpdateRequest
- Optionaloptions: CallOptions
Returns Promise<GetUpdateResponse>
- Defined in client.ts:369

list

list(
req: ListPipelinesRequest,
options?: CallOptions,
): Promise<ListPipelinesResponse>
Lists pipelines defined in the Spark Declarative Pipelines system.
Parameters
- req: ListPipelinesRequest
- Optionaloptions: CallOptions
Returns Promise<ListPipelinesResponse>
- Defined in client.ts:398

listIter

listIter(
req: ListPipelinesRequest,
options?: CallOptions,
): AsyncGenerator<PipelineStateInfo>
Parameters
- req: ListPipelinesRequest
- Optionaloptions: CallOptions
Returns AsyncGenerator<PipelineStateInfo>
- Defined in client.ts:441

listUpdates

listUpdates(
req: ListUpdatesRequest,
options?: CallOptions,
): Promise<ListUpdatesResponse>
List updates for an active pipeline.
Parameters
- req: ListUpdatesRequest
- Optionaloptions: CallOptions
Returns Promise<ListUpdatesResponse>
- Defined in client.ts:459

start

start(
req: StartUpdateRequest,
options?: CallOptions,
): Promise<StartUpdateResponse>
Starts a new update for the pipeline. If there is already an active update for the pipeline, the request will fail and the active update will remain running.
Parameters
- req: StartUpdateRequest
- Optionaloptions: CallOptions
Returns Promise<StartUpdateResponse>
- Defined in client.ts:500

stop

stop(req: StopPipelineRequest, options?: CallOptions): Promise<StopWaiter>
Stops the pipeline by canceling the active update. If there is no active update for the pipeline, this request is a no-op.
Parameters
- req: StopPipelineRequest
- Optionaloptions: CallOptions
Returns Promise<StopWaiter>
- Defined in client.ts:560