---
title: "Check large tables with the Data quality (Spark) action"
description: "The Data quality (Spark) action evaluates the quality rules of a large Lakehouse Manager table on Spark and sends a report to the Lakehouse Manager"
url: https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/dpe-actions-data-quality-pyspark
lang: en
lastUpdated: 2026-11-16
---
> For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/pl/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/pl/llms-full.txt.

# Check large tables with the Data quality (Spark) action

## Objective

The _Data quality (Spark)_ action evaluates the [quality rules](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/lakehouse-manager-data-quality.md) of one Lakehouse Manager table on Spark executors, and sends the result to the Lakehouse Manager as a report, one entry per rule. The report updates the quality status of the table. It runs the same rules, gives the same results and fails the job under the same conditions as the [Data quality](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/dpe-actions-data-quality.md) action, for tables too large for a worker.

Use it when the report of a Data quality run carries `suggested_package: data-quality-pyspark`, or when you already know the table is too large: the Data quality action suggests it above 1,000,000,000 cells (rows times columns) by default.

Like any other action, it can be run on demand, scheduled with a cron trigger, or placed in a [workflow](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/dpe-workflows.md).

## Requirements

- A Lakehouse Manager table with at least one active quality rule. The rules are read from the Lakehouse Manager at each run.
- The Spark 4.1.2 image, the only one that carries the rule engine (Python 3.13).

## Configure the action

### Parameters

| Parameter                                      | Required | Description                                                                                                                                                                                                                                             |
| ---------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Table to check** (`table`)                   | Yes      | The table to check, written `dwh/<dataset>/<table>`, for example `dwh/events/clicks`                                                                                                                                                                    |
| **Spark version** (`spark_version`)            | Yes      | The Spark image the job runs on. Only `4.1.2` carries the rule engine                                                                                                                                                                                   |
| **Critical Results Allowed** (`max_criticals`) | No       | In **When the job fails**. The job fails when more rules than this end critical. Not set: only rules set to [**Fail the job**](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/lakehouse-manager-data-quality.md#job-impact) can fail it |
| **Warning Results Allowed** (`max_warnings`)   | No       | In **When the job fails**. The job fails when more rules than this end in warning. Not set: warnings never fail the job                                                                                                                                 |

The action has no engine choice: every rule runs on Spark, `custom_sql` rules included. Their expression is written in the Trino SQL dialect, as for the Data quality action, and translated to Spark SQL.

The table is read at its current snapshot (`VERSION AS OF`), once per run, so all rules see the same data even if a load runs at the same time.

Size the Spark resources to the table, see [Resources](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/dpe-jobs-resources.md).

### Example payload

To create the action through the API, send the action body to the Data Processing Engine API. The `polar-data` connector module is required next to the action:

```json
{
  "display_name": "Quality of events (Spark)",
  "repository": "<repository id>",
  "repository_version": "<repository version id>",
  "action": "data-quality-pyspark",
  "language": "pyspark",
  "modules": [
    {"type": "action", "id": "data-quality-pyspark", "version": "latest"},
    {"type": "connector", "id": "polar-data", "version": "latest"}
  ],
  "params": {"table": "dwh/events/clicks", "spark_version": "4.1.2"},
  "environment": {"fpu": {"size": 2, "instances": 1}}
}
```

## When the job fails

The report is always sent first: a failed job still has its report in the Lakehouse Manager, with the reason for each result, and the [quality status of the table](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/lakehouse-manager-data-quality.md#table-quality-status) is updated regardless of the job result. The job result and the table status are two different things: a **Report only** rule ending critical makes the table critical without failing the job.

The job fails when:

- more rules end critical than **Critical Results Allowed**, or, when that field is not set, a rule set to **Fail the job** ends critical;
- more rules end in warning than **Warning Results Allowed**, when that field is set;
- no rule at all could be evaluated, for example when the table cannot be read.

The job passes when the table has no active rule, or when the table was never loaded: each rule then reports `no_data` with the reason "This table has no data yet: it was never loaded".

Results that never fail the job:

- `not_evaluated` and `no_data` results, as long as at least one rule could be evaluated;
- a rule on a column the table no longer has. It is reported in warning, with the reason "Column `<name>` is no longer in the table", and it is not counted in **Warning Results Allowed**. Update or delete the rule.

## Go further

- [Check the quality of your tables with data quality rules](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/lakehouse-manager-data-quality.md)
- [Check table quality with the Data quality action](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/dpe-actions-data-quality.md)
- [Configure a job execution alert](https://docs.ovhcloud.com/pl/guides/public-cloud/data-platform/control-center-alerting-job-execution.md) to be notified when a quality job fails

For training or technical assistance implementing our solutions, contact your sales representative or visit our [Professional Services](https://www.ovhcloud.com/pl/professional-services/) page to request a quote and have your project analysed by our experts.

If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help).

Join our [community of users](https://community.ovhcloud.com/).
