For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/de/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/de/llms-full.txt, and this page is available as Markdown at https://docs.ovhcloud.com/de/guides/public-cloud/data-platform/developers-python-sdk-data-quality.md.
Zum Inhalt springen

Manage quality rules and send reports with the Python SDK

Als Markdown ansehen

Use the Python SDK to create the quality rules of a table, run your own checks in a Custom action and send their results as a quality report

Objective

The Data quality action covers the common checks: nulls, ranges, uniqueness and custom SQL (freshness and volume are evaluated by the Lakehouse Manager itself). To evaluate rules in your own code instead, for example inside a Custom action that already reads the table, send the results with the Python SDK. The report goes to the same place as the Data quality action's reports, so the quality status of the table takes your checks into account.

This guide shows how to:

  • create and manage the quality rules of a table;
  • read the rules of a table in a Custom action;
  • send a quality report for one run;
  • read reports and the quality status of a table.

Requirements

  • A Lakehouse Manager table. Rules belong to the table itself, so you can define them before the table is built, and they are kept when it is rebuilt.
  • A Custom action in Python, with the forepaas SDK it installs by default.
  • Permissions on the Quality Rule resource to manage rules, and on the Quality Report resource to send and read reports. See Roles and conditions.

All the methods below are on the Database class of forepaas.dwh. A table is addressed by its name or its id. A method raises DwhRequestException when the Lakehouse Manager refuses the request.

from forepaas.dwh import Database

api = Database()

Manage the quality rules of a table

Create a rule

A rule has a type, the columns it checks, type-specific parameters, a severity (the Job impact of the interface) and thresholds. Thresholds apply to the rule's metric: for a not_null, range or custom_sql rule the metric is a pass rate in percent, so a rule with a warning threshold of 99 and a critical threshold of 95 is in warning below 99% of valid rows and critical below 95%. For a uniqueness rule, the metric is the share of unique rows: 2% of duplicates is 98.

rule = api.quality_rules_create("orders", {
    "display_name": "Amount is filled",
    "type": "not_null",
    "columns": [{"name": "amount"}],
    "severity": "mandatory",
    "thresholds": {"warning": 99, "critical": 95},
})
print(rule["_id"], rule["version"])

A new rule is active. Send "active": False to create it inactive: an inactive rule is listed but never evaluated, and it does not count in the quality status of the table.

The severity is optional (the default, Report only in the interface) or mandatory (Fail the job). Only a mandatory rule with a critical result fails a Data quality job that runs without Critical Results Allowed.

To check a rule against the table without saving it, call quality_rule_validate. It returns valid and, in rule, the rule as it would be stored. When the rule would be refused, it raises DwhRequestException with the reason, for example a column the table does not have:

result = api.quality_rule_validate("orders", {
    "display_name": "Amount is positive",
    "type": "range",
    "columns": [{"name": "amount"}],
    "params": {"min": 0},
    "thresholds": {"critical": 95},
})
print(result["valid"], result["rule"]["params"])

List, update and delete rules

MethodWhat it does
table_quality_rules_list(table)Lists the rules of a table, active and inactive
quality_rule_get(rule_id)Reads one rule
quality_rule_update(rule_id, rule)Replaces a rule
quality_rule_patch(rule_id, changes)Changes the active flag, severity, thresholds, display name or description
quality_rule_delete(rule_id)Deletes a rule

Every change increments the rule's version. To make sure you do not overwrite a change made since you read the rule, send the version you read: the request is refused with QUALITY_RULE_VERSION_CONFLICT if the rule changed in between.

rule = api.quality_rule_get(rule_id)
api.quality_rule_patch(rule_id, {"active": False, "version": rule["version"]})

Send a quality report from a Custom action

A report covers one run on one table. It carries an execution_id that identifies the run, and one entry per rule evaluated. Each entry gives the rule and the version that was evaluated, its type, the outcome and, when you measured them, the metric, the number of rows evaluated and the number of rows failing the rule.

The outcome is one of:

OutcomeMeaning
okThe metric is within the thresholds
warningThe metric crossed the warning level
criticalThe metric crossed the critical level
no_dataThere was nothing to measure, for example an empty table
not_evaluatedThe rule could not be evaluated. Give the reason in reason

The example below evaluates the active not_null rules of a table in Python and sends one report for the run:

import uuid

from forepaas.core.settings import CONFIG
from forepaas.dwh import Database, connect

TABLE = "orders"


def outcome(pass_rate, thresholds):
    if "critical" in thresholds and pass_rate < thresholds["critical"]:
        return "critical"
    if "warning" in thresholds and pass_rate < thresholds["warning"]:
        return "warning"
    return "ok"


def my_custom(event):
    api = Database()
    rules = [r for r in api.table_quality_rules_list(TABLE) if r["active"] and r["type"] == "not_null"]
    if not rules:
        return "no rule"

    connector = connect("dwh/sales/")
    entries = []
    for rule in rules:
        column = rule["columns"][0]["name"]
        counts = connector.query(f"SELECT count(*) AS n, count({column}) AS valid FROM {TABLE}").iloc[0]
        rows, valid = int(counts["n"]), int(counts["valid"])
        if rows == 0:
            entries.append({"rule_id": rule["_id"], "rule_version": rule["version"],
                            "rule_type": rule["type"], "outcome": "no_data"})
            continue
        pass_rate = 100 * valid / rows
        entries.append({
            "rule_id": rule["_id"],
            "rule_version": rule["version"],
            "rule_type": rule["type"],
            "outcome": outcome(pass_rate, rule["thresholds"]),
            "metric": round(pass_rate, 2),
            "rows": rows,
            "violations": rows - valid,
        })

    receipt = api.table_quality_report_send(TABLE, {
        "execution_id": CONFIG.get("execution_id") or str(uuid.uuid4()),
        "engine": "custom",
        "entries": entries,
    })
    print(receipt)
Info

Use the job's execution id as execution_id. A report sent twice with the same execution_id is stored once, so a retried job does not count its run twice. The receipt counts the entries stored, the duplicates and the entries kept only in the history.

If both your action and the Data quality action evaluate the same rule, the most recent run wins: evaluate each rule in one place.

You do not need to report freshness and volume rules: the platform evaluates them from the metadata of the table's snapshot when a report is received, and when the quality status is read.

Optional fields of a report:

FieldContent
snapshot_idThe Iceberg snapshot your checks read
engineA label for what evaluated the rules
evaluated_atWhen the rules were evaluated, in UTC. Defaults to the time the report is received
action_idThe action that ran the checks

Read reports and the quality status

MethodReturns
table_quality_report_get(table, execution_id=None)The latest run of the table, or the run with that execution_id
quality_rule_report(rule_id)The last runs of one rule, each with the rule as it was when evaluated
table_quality_status(table)The quality status of the table: one entry per rule, inactive rules included, and the outcome that rolls them up
status = api.table_quality_status("orders")
print(status["outcome"])
for rule in status["rules"]:
    print(rule["display_name"], rule["outcome"])

The Lakehouse Manager keeps the last 10 runs of each rule. The rule stored with each run lets you see when a threshold or a parameter changed between two runs.

Go further

Run the built-in checks with the Data quality action, and follow a first rule end to end in the data quality tutorial.

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.

War diese Seite hilfreich?