--- title: "Custom PySpark Action" description: "Using our Software Development Kit (SDK) to easily interact with the different components of the platform" url: https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-actions-custom-pyspark lang: en lastUpdated: 2026-09-14 --- > For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/en/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/en/llms-full.txt. # Custom PySpark Action ## Objective A _Custom PySpark action_ allows you to execute custom PySpark scripts in a scalable cloud cluster environment using [Apache Spark™](https://spark.apache.org/). Using our [Software Development Kit (SDK)](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/landing-page-developers-python-sdk.md) to easily interact with the different components of the platform, _Custom PySpark actions_ can be used to implement a variety of use-cases such as: - Execute a manipulation algorithm or ETL job on your data warehouse - Execute a simple data analysis or machine learning algorithm - Extract data from data sources not available on the Data Platform marketplace without having to create connectors for it - Extract real time data (like MQTT, Kafka, etc..) :::info Custom PySpark actions can be orchestrated within [workflows](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-workflows.md) and triggered immediately or on a scheduled basis. At the moment, a single workflow **cannot** contain both PySpark actions and normal actions. ::: ## Configure a Custom PySpark action In the Data Processing Engine of your Project, go in the Actions tab and click on the **New Action** button. Choose the action type _Custom PySpark_. ![Creation screen of a custom action](/images/public-cloud/data-platform/product/dpe/actions/custom-pyspark/picts/custom-pyspark.png) Drag and drop your _.py_ PySpark script onto the "Drag and drop" section.\ Alternatively, select the **Start with a boilerplate** option to get started directly on the Platform's Python interface with example code snippets. ![Creation screen of a custom action](/images/public-cloud/data-platform/product/dpe/actions/custom-pyspark/picts/custom-pyspark2.png) You will be able to edit your source file directly in the editing interface (or drop a new file if needed). Check out the PySpark documentation below: [PySpark Documentation Portal](http://spark.apache.org/docs/latest/api/python/) Note you can also use any function provided in the [Software Development Kit (SDK)](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/landing-page-developers-python-sdk.md) to easily interact with other components of the platform. To read more about all the available SDK functions check out the article below: [Discover all SDK methods](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/landing-page-developers-python-sdk.md) ## Use the helper panel ![DPE Custom PySpark action helper](/images/public-cloud/data-platform/product/dpe/actions/custom-pyspark/picts/custom-action-helper.png) The Custom PySpark action editor includes a helper panel, so you can find guidance without leaving your script: - **Scenarios**: ready-to-use action scripts organized by category, to copy and adapt to your use case. - **SDK guides**: documentation for the [SDK](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/landing-page-developers-python-sdk.md) methods you can call from your script. - **Data**: browse your project's data directly from the editor, to check names and structures while you write your code. - **FAQ**: answers to common action questions. :::info The helper content is the same live catalog that powers the [Data Platform Extension](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-notebooks-data-platform-extension.md) in notebooks, so it is always up to date. ::: ## Manage parallelization Scale your job by adding more parallel worker instances in the action's [preferences](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-jobs-preferences.md). Simply input the desired number of **instances**, which is the Spark executors number. The CPU and RAM size of each instance can be managed by changing the number of [DPU](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-jobs-resources.md) allocated to each. ![Creation screen of a custom action](/images/public-cloud/data-platform/product/dpe/actions/custom-pyspark/picts/custom-pyspark-pref.png) Specify how to split your job's workload on those different workers inside your PySpark script. :::warning Contrary to actions that use the Platform's proprietary data processing engine, PySpark actions don't have settings for [segmentation](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-jobs-preferences.md#segmentation) and [perimeter](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-jobs-preferences.md#perimeter) in the action's preferences. These must be handled inside your PySpark code. ::: ## Manage dependencies ### Installing Python packages You might need to install specific packages not included by default. You can add them in the "Python Requirements" field respecting the format used in a basic requirements file for "pip" (Python package manager) then press "ENTER" on your keyboard. This is what it should looks like once you pressed "ENTER": ![Creation screen of a custom action](/images/public-cloud/data-platform/product/dpe/actions/custom-pyspark/picts/action-requirements.png) ### Installing packages from a Git repository You can install Python packages directly from a GitHub or GitLab repository using the `git+` prefix in your requirements: ``` git+https://github.com/{OWNER}/{REPO}.git ``` To pin a specific version, add a tag or commit hash: ``` git+https://github.com/{OWNER}/{REPO}.git@ ``` #### Auto-install the latest release To always install the most recent published release without manually tracking version tags, use `@latest`: ``` git+https://github.com/{OWNER}/{REPO}.git@latest ``` The platform detects the `git+` prefix and `@latest` suffix, then automatically resolves and substitutes the latest release tag before installing. :::info After adding or modifying a Git dependency, click **Force Build** to reinstall. You no longer need to manually update the tag or commit hash each time a new version of your module is published, but note that the latest release is not picked up automatically at runtime, a manual **Force Build** is always required. ::: ### Default list of dependencies :::warning Data Platform blocks the minors of the versions allowing bug fixes to be installed. If you need a more recent version of a library you can override it manually by adding the same package with the new version in the "Requirements" field. ::: Here is the list of all the packages and their version (as you could find them in a requirements file for pip) shipped with the Data Processing Engine workers: [Discover all default Python packages](https://docs.ovhcloud.com/en/guides/public-cloud/data-platform/dpe-actions-custom-default-packages.md) ## Go further If you need training or technical assistance to implement our solutions, contact your sales representative or click on [this link](https://www.ovhcloud.com/en-gb/professional-services/) to get a quote and ask our Professional Services experts for a custom analysis of your project. Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated [Discord channel](https://discord.gg/ovhcloud). If you need support with your OVHcloud services, create a request in our [Help Centre](https://help.ovhcloud.com/csm?id=csm_get_help). Join our [community of users](https://community.ovhcloud.com/).