Data Platform Buckets connector

Objective

Connect to the Datastore

In order to interact with the Datastore you have to connect to it first, as shown in the code below:

from forepaas.dwh import connect

cn_datastore = connect('data_store')

After that you can use the cn_datastore.list() method to see the buckets available in your Datastore and then connect to the bucket of your choice to interact with it.

You can connect directly to a specific bucket in the Datastore as shown in the code below:

from forepaas.dwh import connect

bucket_name = "name"
cn_bucket = connect('data_store/' + bucket_name)

The datastore connector will return a Data Store Connector object and connecting directly to a bucket will return a Bucket Connector object.

See the next section of this article for additional details on the methods of each connector.

Datastore Connector methods

datastore.list(return_type = 'array')

Lists all buckets in the Datastore.

Input Parameters

Output

datastore.get_buckets()

Gets all buckets from the Datastore

Output

datastore.get_bucket(name)

Gets a bucket instance from its name.

Input Parameters

Output

datastore.create_bucket(name)

Adds a bucket in the Data Store.

Input Parameters

Output

datastore.remove_bucket(name)

Removes a bucket from the Data Store.

Input Parameters

Output

datastore.bucket_exists(name)

Finds out if a bucket exists or not.

Input Parameters

Output

Bucket Connector methods

bucket.list(bool metadata=True, bool recursive=True, **kwargs)

Lists files from Data Store's bucket.

Input Parameters

Output

Short Example

from forepaas.dwh.connect import connect
import logging

bucket_name = "name"
bucket = connect('data_store/' + bucket_name)
files = bucket.list()
logger.info(f"Bucket contents: {files}")

bucket.list_filename(return_type='array', contains='', recursive=True, **kwargs)

Lists filenames from Data Store's bucket.

Input Parameters

Output

Short Example

filenames = bucket.list_filename()
logger.info(f"Filenames: {filenames}")

bucket.get(file_name, **kwargs)

Gets raw file content from a Data Store bucket.

Input Parameters

Output

Short Example

file_data = bucket.get('uploads/file.csv')
logger.info(f"Stream: {file_data.stream(1024)}")

bucket.fget(object_name, file_path, **kwargs)

Gets an object from Datastore's bucket to local path.

Input Parameters

Output

Short Example

bucket.fget("uploads/file.csv", "/tmp/file.csv")
logger.info("File downloaded to /tmp/file.csv")

bucket.put(object_name, data, int length, **kwargs)

Puts an object to Data Store's bucket.

Input Parameters

Output

Short Example

import io
data = io.BytesIO(b"name,age\nJohn,30")
etag = bucket.put("people.csv", data, data.getbuffer().nbytes)
logger.info(f"Uploaded with ETag: {etag}")

bucket.fput(object_name, file_path, **kwargs)

Puts a file to Data Store's bucket.

Input Parameters

Output

Short Example

bucket.fput("people.csv", "/tmp/people.csv")
logger.info("Uploaded /tmp/people.csv")

bucket.put_request(url, path, data={}, method='GET', headers={}, **kwargs)

Gets an object from an HTTP request and upload it to Datastore's bucket.

Input Parameters

Short Example

bucket.put_request(
    url="https://example.com/file.csv",
    path="remote/file.csv"
)
logger.info("File fetched from URL and uploaded to bucket.")

bucket.delete(path, **kwargs)

Deletes multiple/single file in Data Store's bucket.

Input Parameters

Short Example

bucket.delete("people.csv")
logger.info("File deleted from bucket.")

bucket.fcopy_to(new_bucket, object_name, object_source, **kwargs)

Copies file from bucket to a new bucket in Data Store.

Input Parameters

Short Example

bucket.fcopy_to("archive", "people_backup.csv", "people.csv")
logger.info("File copied to archive bucket.")

bucket.exists(filename, **kwargs)

Checks whether a file exists in the bucket.

Input Parameters

Output

Short Example

if bucket.exists("people.csv"):
    logger.info("File exists.")

bucket.get_content(file_name, **kwargs)

Retrieves the full content of a file from the bucket.

Input Parameters

Output

Short Example

content = bucket.get_content("people.csv")
logger.info(f"File content: {content.decode()}")

bucket.get_path(path)

Returns the full qualified path (URL or reference) of an object in the bucket.

Input Parameters

Output

Short Example

full_path = bucket.get_path("people.csv")
logger.info(f"Full path: {full_path}")

bucket.remove_path(path)

Removes a specific path from the bucket.

Input Parameters

Output

Short Example

bucket.remove_path("people.csv")
logger.info("Removed specific path from bucket.")

Deprecated Methods

Additional methods

The Data Platform Datastore is built on Minio technology. Please refer to the Minio Technical Documentation for more information on the advanced settings of the SDK functions.

Go further

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.

Ask questions, give your feedback and interact directly with the team building the Data Platform on the dedicated Discord channel.

If you need support with your OVHcloud services, create a request in our Help Centre.

Join our community of users.