OptionalawsOptionalazureAttributes related to instance pools running on Azure. If not specified at pool creation, a set of default values will be used.
OptionalcustomAdditional tags for pool resources. default_tags. Notes:
OptionaldiskDefines the specification of the disks that will be attached to all spark containers.
OptionalenableAutoscaling Local Storage: when enabled, this instances in this pool will dynamically acquire additional disk space when its Spark workers are running low on disk space. In AWS, this feature requires specific AWS permissions to function correctly - refer to the User Guide for more details.
OptionalgcpAttributes related to instance pools running on Google Cloud Platform. If not specified at pool creation, a set of default values will be used.
OptionalidleAutomatically terminates the extra instances in the pool cache after they are inactive for this time in minutes if min_idle_instances requirement is already met. If not set, the extra pool instances will be automatically terminated after a default timeout. If specified, the threshold must be between 0 and 10000 minutes. Users can also set this value to 0 to instantly remove idle instances from the cache if min cache size could still hold.
OptionalinstancePool name requested by the user. Pool name must be unique. Length must be between 1 and 100 characters.
OptionalmaxMaximum number of outstanding instances to keep in the pool, including both instances used by clusters and idle instances. Clusters that require further instance provisioning will fail during upsize requests.
OptionalminMinimum number of idle instances to keep in the instance pool
OptionalnodeFlexible node type configuration for the pool.
OptionalnodeThis field encodes, through a single value, the resources available to each of the Spark nodes in this cluster. For example, the Spark nodes can be provisioned and optimized for memory or compute intensive workloads. A list of available node types can be retrieved by using the :method:clusters/listNodeTypes API call.
OptionalpreloadedCustom Docker Image BYOC
OptionalpreloadedA list containing at most one preloaded Spark image version for the pool. Pool-backed clusters started with the preloaded Spark version will start faster. A list of available Spark versions can be retrieved by using the :method:clusters/sparkVersions API call.
OptionalremoteIf set, what the configurable throughput (in Mb/s) for the remote disk is. Currently only supported for GCP HYPERDISK_BALANCED types.
OptionaltotalIf set, what the total initial volume size (in GB) of the remote disks should be. Currently only supported for GCP HYPERDISK_BALANCED types.
Attributes related to instance pools running on Amazon Web Services. If not specified at pool creation, a set of default values will be used.