For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/en/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/en/llms-full.txt, and this page is available as Markdown at https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-actions-data-quality-pyspark.md.
Skip to content

Check large tables with the Data quality (Spark) action

View as Markdown

The Data quality (Spark) action evaluates the quality rules of a large Lakehouse Manager table on Spark and sends a report to the Lakehouse Manager

Objective

The Data quality (Spark) action evaluates the quality rules of one Lakehouse Manager table on Spark executors, and sends the result to the Lakehouse Manager as a report, one entry per rule. The report updates the quality status of the table. It runs the same rules, gives the same results and fails the job under the same conditions as the Data quality action, for tables too large for a worker.

Use it when the report of a Data quality run carries suggested_package: data-quality-pyspark, or when you already know the table is too large: the Data quality action suggests it above 1,000,000,000 cells (rows times columns) by default.

Like any other action, it can be run on demand, scheduled with a cron trigger, or placed in a workflow.

Requirements

  • A Lakehouse Manager table with at least one active quality rule. The rules are read from the Lakehouse Manager at each run.
  • The Spark 4.1.2 image, the only one that carries the rule engine (Python 3.13).

Configure the action

Parameters

ParameterRequiredDescription
Table to check (table)YesThe table to check, written dwh/<dataset>/<table>, for example dwh/events/clicks
Spark version (spark_version)YesThe Spark image the job runs on. Only 4.1.2 carries the rule engine
Critical Results Allowed (max_criticals)NoIn When the job fails. The job fails when more rules than this end critical. Not set: only rules set to Fail the job can fail it
Warning Results Allowed (max_warnings)NoIn When the job fails. The job fails when more rules than this end in warning. Not set: warnings never fail the job

The action has no engine choice: every rule runs on Spark, custom_sql rules included. Their expression is written in the Trino SQL dialect, as for the Data quality action, and translated to Spark SQL.

The table is read at its current snapshot (VERSION AS OF), once per run, so all rules see the same data even if a load runs at the same time.

Size the Spark resources to the table, see Resources.

Example payload

To create the action through the API, send the action body to the Data Processing Engine API. The polar-data connector module is required next to the action:

{
  "display_name": "Quality of events (Spark)",
  "repository": "<repository id>",
  "repository_version": "<repository version id>",
  "action": "data-quality-pyspark",
  "language": "pyspark",
  "modules": [
    {"type": "action", "id": "data-quality-pyspark", "version": "latest"},
    {"type": "connector", "id": "polar-data", "version": "latest"}
  ],
  "params": {"table": "dwh/events/clicks", "spark_version": "4.1.2"},
  "environment": {"fpu": {"size": 2, "instances": 1}}
}

When the job fails

The report is always sent first: a failed job still has its report in the Lakehouse Manager, with the reason for each result, and the quality status of the table is updated regardless of the job result. The job result and the table status are two different things: a Report only rule ending critical makes the table critical without failing the job.

The job fails when:

  • more rules end critical than Critical Results Allowed, or, when that field is not set, a rule set to Fail the job ends critical;
  • more rules end in warning than Warning Results Allowed, when that field is set;
  • no rule at all could be evaluated, for example when the table cannot be read.

The job passes when the table has no active rule, or when the table was never loaded: each rule then reports no_data with the reason "This table has no data yet: it was never loaded".

Results that never fail the job:

  • not_evaluated and no_data results, as long as at least one rule could be evaluated;
  • a rule on a column the table no longer has. It is reported in warning, with the reason "Column <name> is no longer in the table", and it is not counted in Warning Results Allowed. Update or delete the rule.

Go further

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.

Was this page helpful?