An OCI / Docker image registry. You can use a public registry (such as Docker Hub for example) or a private registry. Refer to the Creating a private registry documentation to create a private registry based on Harbor. To make your registry compatible with AI Solutions usage, follow the Use & manage your registries guide.
To train the model, we will use AI Training. This powerful tool will allow you to automate your pipelines and build fine-tuning phases easily.
AI Training allows you to train models directly from your own Docker images.
First, you need to create a Python script that is in charge of doing the training.
You can copy and paste the following code in a file named train-audio-classification.py:
import numpy as npimport pandas as pdimport datetime# preprocessingfrom sklearn.preprocessing import LabelEncoderfrom sklearn.preprocessing import StandardScalerfrom sklearn.model_selection import train_test_split# modelimport tensorflow as tf######################################################################################################################################################### The goal of this script is to train a pre-construct model to recognize marine mammal sound. ## See the Notebook "notebook-marine-sound-classification" in the ai-training-examples for ## more details : https://github.com/ovh/ai-training-examples/blob/main/notebooks/audio/audio-classification/notebook-marine-sound-classification.ipynb ## You must mount 2 volumes for the data and the model (the same used for the Notebook for example 😉) : ## - /workspace/saved_model where the model is stored ## - /workspace/data/ where store the data for the training ########################################################################################################################################################## 🗃 Load pre-transform datadf = pd.read_csv('/workspace/data/data.csv')# dataframe shapedf.shape# dataframe typesdf.dtypes# 🔢 Encode the labels (0 => 44) class_list = df.iloc[:,-1]encoder = LabelEncoder()y = encoder.fit_transform(class_list)print("y: ", y)# 🧹 Uniformize data thanks to the initial data input_parameters = df.iloc[:, 1:27]scaler = StandardScaler()X = scaler.fit_transform(np.array(input_parameters))print("X:", X)# ⚗️ Create training and validation setsX_train, X_val, y_train, y_val = train_test_split(X, y, test_size = 0.2)# 🧠 Define model architecturemodel = tf.keras.models.Sequential([ tf.keras.layers.Dense(512, activation='relu', input_shape=(X_train.shape[1],)), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(256, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(128, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(64, activation='relu'), tf.keras.layers.Dropout(0.2), tf.keras.layers.Dense(45, activation='softmax'),])print(model.summary())# 💪 Train the model with datamodel.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics = 'accuracy')# 📈 Add the TensorBoard callback (optional)print('Model tracking')log_dir = "/workspace/saved_model/runs/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)model.fit(X_train, y_train, validation_data = (X_val, y_val), epochs = 100, batch_size = 128, callbacks = [tensorboard_callback])# 💿 Save the model for future usagesmodel.save('/workspace/saved_model/my_model2')print('End of training')
Info
The tensorboard step is not mandatory. It's just a way to monitor your training.
Then, create a requirements.txt file to declare the Python dependencies:
tensorflownumpy==1.22.4pandasscikit-learnkeras
Then, create a Dockerfile compliant with AI Training.
You can copy and paste the following code in a file named Dockerfile:
FROM --platform=linux/x86_64 python:3.8WORKDIR /workspaceADD . /workspaceRUN pip install -r requirements.txt# Mandatory to run the jobs in rootless modeRUN chown -R 42420:42420 /workspaceCMD [ "python3" , "/workspace/train-audio-classification.py"]
Then, build the Docker image and push it in the registry:
Once your Docker image is created and pushed into the registry, you can directly use the ovhai command to create your model training.
You can launch the training specifying more or less GPU depending on the speed you want for your training.
