SDK reference: connect and bulk_insert

Objective

The two functions every script in the Custom Actions SDK starts from: connect() to reach a dataset, table, bucket or source, and bulk_insert() to write a dataframe back.

The connect function

The connect() function is, most of the time, the main entry point to interact with the Project.

It takes a connection string as parameter and returns a Connector object.
For example:

from forepaas.dwh import connect

connection_string = "dwh/default_dataset/"
connector_default = connect(connection_string)

Change the connection string to connect to different sources within the Data Platform environment.
As a consequence, the object, and available methods, returned by connect(...) will be different.

Our Connectors and Connection Strings article explains each option available in more detail and below you will find the method list for two common connectors:

• Lakehouse Manager Dataset Connector
• Data Platform Buckets Connector

The bulk_insert function

The bulk_insert() function is a very important part of the SDK. You can use it to insert data back into a table. As shown in the code below, it takes a Dataset Connector, a table name and a pandas DataFrame as parameters.

import pandas as pd
from forepaas.dwh import connect
from forepaas.dwh import bulk_insert

# make connection to the default dataset
cn = connect("dwh/default_dataset/")

# extract data from the table with no SQL required
df = cn.select("stations_rides")

# perform your custom transform in the dataframe
df.loc[df["station_name"] == 'Harlem-Lake', "rides"] = 0

# reinsert your dataframe in the destination table
stats = bulk_insert(cn, "stations_rides", df) 

bulk_insert(connector, table, data, source_default_schema={}, batch_size=None)

Input Parameters

Output

PySpark Support

Data Platform offers extensive support to handle data with PySpark. All is done through Spark-compatible methods in the Connector object.
Check out our PySpark article for further details.

Go further

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.