For AI agents: the complete documentation index is available at https://docs.ovhcloud.com/en/llms.txt, the full documentation bundle is available at https://docs.ovhcloud.com/en/llms-full.txt, and this page is available as Markdown at https://docs.ovhcloud.com/en/guides/public-cloud/ai-machine-learning/ai-s3-compliance.md.
Skip to content
Appearance
Console(the OVHcloud Console, opens in a new tab)

Data - Compliance between AI Tools and S3 compatible Object Storage

View as Markdown

Learn how to use S3 compatible Object Storage buckets with AI Tools

Objective

This guide is intended to help you use S3 compatible Object Storage buckets with the OVHcloud AI Solutions via the OVHcloud Console and the ovhai CLI.

Indeed, you will learn how to:

Requirements

To follow this guide, ensure you meet the following requirements:

  • A Public Cloud project.
  • A user account created on this Public Cloud project, with the AI and Object Storage roles assigned. For more information on how to create such a user, please consult the Manage AI users and roles documentation.
  • The ovhai CLI installed. Please refer to our guide on how to install ovhai CLI for assistance.

OVHcloud Console Access

  • Direct link:
  • Navigation path: Public Cloud > Select your project > Storage > Object Storage

Instructions

Create an S3 compatible bucket

To create your first S3 compatible Object Storage bucket, click Object Storage under Storage in the left-hand menu of your Public Cloud project. Your buckets are listed in the Containers tab.

In the Containers tab, click Create an object container.

1. Container type and name

In the Container type section, select S3-compatible API, then enter the name of your bucket in the Container name section. Note that the bucket name must be unique across all buckets in your Public Cloud account and must meet the following conditions:

  • Between 3 and 63 characters
  • Consist only of lower case letters, numbers, dots (.), and hyphens (-)
  • Must start and end with lower case alphanumeric characters (a to z and 0 to 9)

2. Deployment mode

Then, select the deployment mode you want to use (3-AZ, 1-AZ or Local Zone). Find out more by reading the deployment modes comparison.

3. Region

Select a region.

4. Versioning, user and encryption

Enable or disable object versioning.

For 1-AZ and 3-AZ deployment modes, you need to link a user from your Public Cloud project to your new bucket in the User section: select an existing user in the list, or click Create a user to create one.

Warning

Warning! If you create a new user, make sure to save its credentials (username, access key, secret key).

For Local Zone deployment mode, no additional steps are required to link a user to the bucket, as all users of your Public Cloud project will automatically have access to all containers in Local Zones.

Finally, choose whether or not to encrypt your data (Encryption section).

Check the summary of your bucket, then click Create to confirm S3 compatible bucket creation.

Your bucket now appears in the list of your Object Storage buckets.

By clicking on its name, you can see the objects it contains (empty for now) in the Objects tab. The Overview tab shows information such as its endpoint and location (region), which we will use later.

Edit your S3 compatible user roles

Once your S3 compatible bucket has been created, you need to check that the associated user has the necessary rights to interact with your data and the OVHcloud AI Solutions (AI Notebooks, Training and Deploy).

For Local Zone deployment, you did not need to associate a user with your bucket. However, it is important to check that you also have at least one user with the necessary rights.

To do that, click Project Management in the left-hand menu of your Public Cloud project, then open the Users tab.

Whether you have created a new user or use an existing one, check that this user has at least the following roles: AI Operator and ObjectStore operator.

For more information about editing user rights, you can check the dedicated documentation.

Retrieve user credentials

Before continuing, you need to ensure that you have the following information so that you can authenticate via the CLI and use your bucket:

  • Username
  • Password
  • S3 compatible access key
  • S3 compatible secret key

As we saw in the previous step, all your Public Cloud project users are displayed in the Users tab of Project Management. You will find there your existing usernames. The username we are interested in is the one to which you gave permissions in the previous step.

If you have lost the password associated with this username, you can regenerate it by clicking the ..., then on Generate a password, which will revoke the old one.

Then, you need to retrieve the access key and secret key associated to this user. To do that, return to the Object Storage page and open its Users tab. Here you will find the access key for each of your existing users, in the Access key column. You can also display the secret key by clicking the ... button on the user's row, then Show secret key.

Set up proper access permissions

It is also important to ensure that this user has the appropriate permissions to access all the files in the bucket. Otherwise, you may encounter permission issues during the AI Solution deployment process.

To give your user access to your bucket, go to the Containers tab of the Object Storage page and click the ... button on the row of your bucket. Then, click Add S3 user, select the same User you used during bucket creation and its Role, then confirm.

By following these steps, you can grant the necessary permissions to your user, ensuring a smooth deployment process for your notebook, job, or app.

Add a datastore via the ovhai CLI

Now that your bucket has been created, and the associated user has the necessary permissions, you can move on to the ovhai CLI. If you have not installed it yet, please refer to the ovhai CLI installation guide.

The first step is to authenticate with the user that you have granted access to your bucket. To do this, use the following command:

ovhai login

We recommend that you choose the Terminal authentication mode, and enter your username and associated password to log in.

If you encounter an authentication error, please recheck this step and make sure you have the AI right.

Once authenticated, you can list your existing datastores with the following command:

ovhai datastore list

You should obtain the following result:

ALIAS STORE_TYPE OWNER    ENDPOINT
BHS   swift      ovhcloud ~
DE    swift      ovhcloud ~
GRA   swift      ovhcloud ~
SBG   swift      ovhcloud ~
SGP   swift      ovhcloud ~
UK    swift      ovhcloud ~
WAW   swift      ovhcloud ~
Info

Here you can see that there are no S3 compatible data stores in the list yet. That is why you will add your own, to then access the buckets of a given region, and for a specific access key and secrey key combination!

To add this datastore, you will use the ovhai datastore add s3 command pattern. You can use the --help flag to know more about adding S3 compatible datastores:

ovhai datastore add s3 --help

This will give us the different arguments to fill in:

Add an S3 compatible data store

Usage: ovhai datastore add s3 [OPTIONS] <ALIAS> <ENDPOINT_URL> <REGION> <ACCESS_KEY> <SECRET_KEY>

Arguments:
  <ALIAS>         Alias for the data store
  <ENDPOINT_URL>  Data store connection URL
  <REGION>        Data store region
  <ACCESS_KEY>    Connection access key
  <SECRET_KEY>    Connection secret key

Options:
      --store-credentials-locally  Whether or not to store the data store credentials locally when creating or updating a data store
      --token <TOKEN>              Authentication using Token rather than OAuth
      --no-color                   Remove colors from output
  -h, --help                       Print help

Here is the command pattern to add an OVHcloud S3 compatible datastore:

ovhai datastore add s3 <ALIAS> <endpoint_url> <region> <my-access-key> <my-secret-key> --store-credentials-locally

You will need to replace:

  • <alias> by the name you want to give for your datastore
  • <endpoint_url> by the Endpoint value you have retrieved at the end of bucket creation step. This endpoint should start with: https://s3....
  • <region> by the region where you have created your bucket. This region is indicated as the Location of your bucket, on the last screenshot of the bucket creation step. Make sure you use lower case letters.
  • <my-access-key> by the access key of the associated user
  • my-secret-key by the secret key of the associated user
Info

Since you used --store-credentials-locally, this datastore is only registered in your local ovhai CLI configuration for the region context (ovhai config set <region>) you were using when you ran the command. It is not shared across regions. If you want to use this datastore from notebooks, jobs or apps running in another region, run the same ovhai datastore add command again after switching to that region's context.

Info

Note: This endpoint also allows you to choose between Standard and High Performance storage classes. The storage class you select will determine the storage class used during file upload to the bucket when using the ovhai CLI. If you plan to upload files using the OVHcloud Console (UI), you will have the option to choose between Standard and High Performance storage. However, when using the CLI, the default endpoint will use Standard storage. If you require High Performance storage, please ensure that the endpoint ends with .perf.cloud.ovh.net.

For more information on storage classes, please consult this documentation.

Here is an example to add a datastore named 1azgra that will handle 1-AZ buckets created in Gravelines (gra) region (since they share the same endpoint_url):

ovhai datastore add s3 1azgra https://s3.gra.io.cloud.ovh.net/ gra <my-access-key> <my-secret-key> --store-credentials-locally

You can now check the datastore list:

ovhai datastore list

You should see your new datastore in the list:

ALIAS STORE_TYPE OWNER    ENDPOINT
BHS   swift      ovhcloud ~
DE    swift      ovhcloud ~
GRA   swift      ovhcloud ~
1azgra s3         customer https://s3.gra.io.cloud.ovh.net/
SBG   swift      ovhcloud ~
SGP   swift      ovhcloud ~
UK    swift      ovhcloud ~
WAW   swift      ovhcloud ~

Now that you have a datastore dedicated for the specified region, you can list the buckets associated to this region and access and secret keys combination by running the following command (replace <alias> by your datastore name, 1azgra in previous example):

ovhai bucket list <alias> 
DATE                     NAME
2024-12-25T09:00:00.000Z s3-gra-1az

You can see that the bucket s3-gra-1az you created using the OVHcloud Console (UI) has been retrieved.

Create new buckets linked to this datastore using CLI (Optional)

This step is optional and for information only.

Now that your datastore is configured, you can simply create new buckets for the region and user associated with your datastore using the following command, rather than using the OVHcloud Console (UI):

ovhai bucket create <alias> <new_bucket-name>
Warning

Keep in mind that the bucket name must be between 3 and 63 characters, can consist only of lower case letters, numbers, dots (.), and hyphens (-) and must start and end with lower case alphanumeric characters (a to z and 0 to 9).

Then you can check that your S3 compatible bucket has been created correctly, and linked to your datastore (replace <alias> by your datastore name, 1azgra in previous example):

ovhai bucket list <alias>

You should see your bucket in the list:

DATE                     NAME
2024-12-25T09:00:00.000Z s3-gra-1az
2024-12-25T10:00:00.000Z new-bucket-name

Upload files to your bucket

Before connecting your bucket to the AI solutions, you are going to upload a few objects to it (images, codes, models, ...) so that you can then retrieve these files from the AI Solutions. You can do this using the OVHcloud Console (UI) or the ovhai CLI:

OVHcloud Console
ovhai CLI

In the Containers tab of the Object Storage page, click the name of your bucket: its Objects tab opens. Click Add an object.

In the panel that opens, you can enter a Prefix to add to the name of your objects and select a Storage class (Standard or High Performance; see Choosing the right storage class for your needs). Drag and drop your files or click Browse files, then click Upload.

For example, if you add the python file file.py with the prefix /scripts, the resulting file path is /scripts/file.py.

Once the upload is completed, you should see uploaded files in the object list.

Use S3 compatible buckets with AI Solutions

Now that your S3 compatible bucket has been created and populated with some objects, you can mount it on any OVHcloud AI Solution to access your data.

Info

You can also mount an S3 compatible bucket from the OVHcloud Console: add it as a datastore in the S3 volumes tab of the Configuration page of AI Notebooks, AI Training or AI Deploy, then attach it in the Volumes part of the Advanced configuration section when creating your notebook, job or app. The examples below use the ovhai CLI.

AI Notebooks
AI Training
AI Deploy

To launch an AI notebook with an S3 compatible bucket mounted as remote data, you can use the following command pattern:

ovhai notebook run <framework-id> <editor-id> \
   --volume <bucket-name>@<datastore-alias>:<mount_path>:<permission>

Just make sure to replace the framework and editor by those of your choice. Also, specify the right <bucket-name> and <datastore-alias, and designate the destination where you want the bucket data to be stored (<mount_path>) and its access permission (rw for read & write, or ro for read only).

For example, to launch a basic Jupyter notebook on 2 CPUs, with a Conda pre-packaged environment, and the bucket s3-gra-1az from the 1azgra datastore mounted as remote data on /workspace/my-codes with read and write (RW) permissions, use the following command:

ovhai notebook run conda jupyterlab \
   --volume s3-gra-1az@1azgra/:/workspace/my-codes:rw \
   --cpu 2

This command will create an AI notebook with the specified framework and editor, and will mount the specified bucket at the specified destination with the specified permissions. After logging in to your notebook interface (using the same credentials you have used to ovhai login), you should see the uploaded file in your notebook's workspace:

notebook content bucket

For more information on the capabilities of the ovhai notebook run command, you can run the following command:

ovhai notebook run --help

You can also consult the dedicated CLI - Launch an AI notebook documentation to know more about the possible options, frameworks, and editors available.

If you want to use the boto3 library to manage your S3 compatible bucket objects, here is a notebook that contains basic commands.

Feedback

Please send us your questions, feedback and suggestions to improve the service:

For training or technical assistance implementing our solutions, contact your sales representative or visit our Professional Services page to request a quote and have your project analysed by our experts.

1: S3 is a trademark of Amazon Technologies, Inc. OVHcloud's service is not sponsored by, endorsed by, or otherwise affiliated with Amazon Technologies, Inc.

Was this page helpful?