Data Quality Monitors

Package: databricks.bundles.data_quality_monitors

Classes

class AggregationGranularity

The granularity for aggregating data into time windows based on their timestamp.

AGGREGATION_GRANULARITY_5_MINUTES = 'AGGREGATION_GRANULARITY_5_MINUTES'
AGGREGATION_GRANULARITY_30_MINUTES = 'AGGREGATION_GRANULARITY_30_MINUTES'
AGGREGATION_GRANULARITY_1_HOUR = 'AGGREGATION_GRANULARITY_1_HOUR'
AGGREGATION_GRANULARITY_1_DAY = 'AGGREGATION_GRANULARITY_1_DAY'
AGGREGATION_GRANULARITY_1_WEEK = 'AGGREGATION_GRANULARITY_1_WEEK'
AGGREGATION_GRANULARITY_2_WEEKS = 'AGGREGATION_GRANULARITY_2_WEEKS'
AGGREGATION_GRANULARITY_3_WEEKS = 'AGGREGATION_GRANULARITY_3_WEEKS'
AGGREGATION_GRANULARITY_4_WEEKS = 'AGGREGATION_GRANULARITY_4_WEEKS'
AGGREGATION_GRANULARITY_1_MONTH = 'AGGREGATION_GRANULARITY_1_MONTH'
AGGREGATION_GRANULARITY_1_YEAR = 'AGGREGATION_GRANULARITY_1_YEAR'
class AnomalyDetectionConfig

Anomaly Detection Configurations.

notification_settings: NotificationSettings | None = None

Who to alert when anomaly detection finds unhealthy tables in this schema. Applicable to schema monitors only. This is the same schema alert you manage from Manage alerts in the Data Quality Monitoring UI, so a change made in either place shows in both.

Set the recipients in on_unhealthy_table; on_failure is not supported here. Only destination_ids and destination_user_ids are supported.

On update, an update mask of *, anomaly_detection_config, or anomaly_detection_config.notification_settings replaces the alert’s recipients with the ones in the request; omitting the field, or leaving both recipient lists empty, removes the alert. Other masks leave the alert unchanged.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class CronSchedule

The data quality monitoring workflow cron schedule.

quartz_cron_expression: str

The expression that determines when to run the monitor. See examples.

timezone_id: str

A Java timezone id. The schedule for a job will be resolved with respect to this timezone. See Java TimeZone for details. The timezone id (e.g., America/Los_Angeles) in which to evaluate the quartz expression.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class DataProfilingConfig

Data Profiling Configurations.

output_schema_id: str

ID of the schema where output tables are created.

assets_dir: str | None = None

Field for specifying the absolute path to a custom directory to store data-monitoring assets. Normally prepopulated to a default user location via UI and Python APIs.

baseline_table_name: str | None = None

Baseline table name. Baseline data is used to compute drift from the data in the monitored table_name. The baseline table and the monitored table shall have the same schema.

custom_metrics: list[DataProfilingCustomMetric]

Custom metrics.

inference_log: InferenceLogConfig | None = None

Analysis Configuration for monitoring inference log tables.

notification_settings: NotificationSettings | None = None

Field for specifying notification settings.

schedule: CronSchedule | None = None

The cron schedule.

skip_builtin_dashboard: bool | None = None

Whether to skip creating a default dashboard summarizing data quality metrics.

slicing_exprs: list[str]

List of column expressions to slice data with for targeted analysis. The data is grouped by each expression independently, resulting in a separate slice for each predicate and its complements. For example slicing_exprs=[“col_1”, “col_2 > 10”] will generate the following slices: two slices for col_2 > 10 (True and False), and one slice per unique value in col1. For high-cardinality columns, only the top 100 unique values by frequency will generate slices.

snapshot: SnapshotConfig | None = None

Analysis Configuration for monitoring snapshot tables.

time_series: TimeSeriesConfig | None = None

Analysis Configuration for monitoring time series tables.

warehouse_id: str | None = None

Optional argument to specify the warehouse for dashboard creation. If not specified, the first running warehouse will be used.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class DataProfilingCustomMetric

Custom metric definition.

definition: str

Jinja template for a SQL expression that specifies how to compute the metric. See create metric definition.

name: str

Name of the metric in the output tables.

output_data_type: str

The output type of the custom metric.

type: DataProfilingCustomMetricType

The type of the custom metric.

input_columns: list[str]

A list of column names in the input table the metric should be computed for. Can use ":table" to indicate that the metric needs information from multiple columns.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class DataProfilingCustomMetricType

The custom metric type.

DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE = 'DATA_PROFILING_CUSTOM_METRIC_TYPE_AGGREGATE'
DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED = 'DATA_PROFILING_CUSTOM_METRIC_TYPE_DERIVED'
DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT = 'DATA_PROFILING_CUSTOM_METRIC_TYPE_DRIFT'
class DataQualityMonitor

Monitor for the data quality of unity catalog entities such as schema or table.

object_id: str

The UUID of the request object. It is schema_id for schema, and table_id for table.

Find the schema_id from either: 1. The schema_id of the Schemas resource. 2. In Catalog Explorer > select the schema > go to the Details tab > the Schema ID field.

Find the table_id from either: 1. The table_id of the Tables resource. 2. In Catalog Explorer > select the table > go to the Details tab > the Table ID field.

object_type: str

The type of the monitored object. Can be one of the following: schema or table.

anomaly_detection_config: AnomalyDetectionConfig | None = None

Anomaly Detection Configuration, applicable to schema object types.

data_profiling_config: DataProfilingConfig | None = None

Data Profiling Configuration, applicable to table object types. Exactly one Analysis Configuration must be present.

lifecycle: Lifecycle | None = None

Settings that control the deployment lifecycle of the resource, such as preventing it from being destroyed.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class DependsOn
resource: str

Key of the resource to depend on, as foo or jobs.foo.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class InferenceLogConfig

Inference log configuration.

model_id_column: str

Column for the model identifier.

prediction_column: str

Column for the prediction.

problem_type: InferenceProblemType

Problem type the model aims to solve.

timestamp_column: str

Column for the timestamp.

granularities: list[AggregationGranularity]

List of granularities to use when aggregating data into time windows based on their timestamp.

label_column: str | None = None

Column for the label.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class InferenceProblemType

Inference problem type the model aims to solve.

INFERENCE_PROBLEM_TYPE_CLASSIFICATION = 'INFERENCE_PROBLEM_TYPE_CLASSIFICATION'
INFERENCE_PROBLEM_TYPE_REGRESSION = 'INFERENCE_PROBLEM_TYPE_REGRESSION'
class Lifecycle
depends_on: list[DependsOn]

Resources that must be deployed before this resource and deleted after it. Unlike variable references, this only affects ordering.

prevent_destroy: bool | None = None

Lifecycle setting to prevent the resource from being destroyed.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class NotificationDestination

Destination of the data quality monitoring notification.

destination_ids: list[str]

The notification-destination IDs (Slack / Teams / webhook) to send the notification to. The destinations must already exist in the workspace; a workspace admin creates them. A maximum of 25 destination IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

destination_user_ids: list[int]

The IDs of the workspace users to notify by email. A maximum of 25 user IDs is supported. Only supported on schema monitors, via anomaly_detection_config; rejected on table monitors.

email_addresses: list[str]

The list of email addresses to send the notification to. A maximum of 5 email addresses is supported.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class NotificationSettings

Settings for sending notifications on the data quality monitoring.

on_failure: NotificationDestination | None = None

Destinations to send notifications on failure/timeout.

on_unhealthy_table: NotificationDestination | None = None

Recipients to notify when anomaly detection finds unhealthy tables in the schema. Each recipient gets one notification per important unhealthy table. Only supported on schema monitors, via anomaly_detection_config.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class SnapshotConfig

Snapshot analysis configuration.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict
class TimeSeriesConfig

Time series analysis configuration.

timestamp_column: str

Column for the timestamp.

granularities: list[AggregationGranularity]

List of granularities to use when aggregating data into time windows based on their timestamp.

classmethod from_dict(
value: dict,
) → Self
as_dict(
self,
) → dict