Object Storage Swift - Swift commands Memo (EN)
Objective
You can use the OpenStack API to generate various scripts to automate your actions on your Public Cloud instances.
The OpenStack swiftclient allows you to interact with and manage your containers and objects. For example, you can send files regularly to your containers, in order to back them up.
In this guide, you can find the main swift commands for managing your object containers using python-swiftclient.
Requirements
Instructions
Create a PCS container
swift stat <container> <object>
List the contents of a container
Upload an object smaller than 5GB
swift upload <container> <object>
Upload an object higher than 5GB in SLO mode
swift upload --use-slo --segment-size 1G <container> <object>
Upload an object higher than 5GB in DLO mode
swift upload --segment-size 1G <container> <object>
LargeObject upload abortion
$ swift upload --use-slo --segment-size 500M <container> <object>
^C Aborted
$ swift list
<container>
<container_segments>
$ swift list <container>
$
$ swift list <container_segments>
<object>/slo/1628738591.297565/6442450944/524288000/00000000
<object>/slo/1628738591.297565/6442450944/524288000/00000001
<object>/slo/1628738591.297565/6442450944/524288000/00000002
<object>/slo/1628738591.297565/6442450944/524288000/00000003
<object>/slo/1628738591.297565/6442450944/524288000/00000004
<object>/slo/1628738591.297565/6442450944/524288000/00000005
<object>/slo/1628738591.297565/6442450944/524288000/00000006
<object>/slo/1628738591.297565/6442450944/524288000/00000007
<object>/slo/1628738591.297565/6442450944/524288000/00000008
<object>/slo/1628738591.297565/6442450944/524288000/00000009
It is recommended that you delete the <container_segments> or at least the segments that correspond to the dropped object.
Download an object
swift download <container> <object>
Delete an empty container
Delete a non-empty container
Delete an object
swift delete <container> <object>
Delete prefix objects
swift delete --prefix <prefix> <container>
swift post -H "X-Container-Meta-Access-Control-Allow-Origin:http://example.com" <container>
swift post -m "my-custom-key:value" <container> <object>
swift post -H "X-Container-Meta-Access-Control-Allow-Origin:" <container>
swift post -m "my-custom-key:" <container> <object>
swift post -H "X-Remove-Object-My-Custom-Key:" <container> <object>
Set Read ACL on a container
swift post <container> -r "${OS_TENANT_ID}:*"
Set write ACL on a container
swift post <container> -w "${OS_TENANT_ID}:*"
Delete the read ACL on a container
swift post <container> -r ""
Delete write ACL on a container
swift post <container> -w ""
Move objects from one container to another
swift copy -d /<destination_container> <container> <object>
LargeObjects
Info
In this example, the LargeObject was uploaded in SLO mode.
See the Curl command memo documentation for the manifest upload.
On a LargeObject, the swift copy command returns a 413 error:
swift copy -d /<destination_container> <container> <largeobject>
created container <cdestination_container>
Object COPY failed: https://storage.gra.cloud.ovh.net/v1/AUTH_702xxxxxxxxxxxxxxxxxxxxxxxxxxdaf/<container>/<largeobject> 413 Request Entity Too Large [first 60 chars of response] b'<html><h1>Request Entity Too Large</h1><p>The body of your r'
So we need to start by moving the segments:
for obj in $(swift list <container_segments>);do swift copy -d /<destination_container_segments> <container_segments> $obj;done
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000000 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000000
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000001 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000001
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000002 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000002
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000003 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000003
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000004 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000004
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000005 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000005
Then retrieve the manifest, adapt it and re-upload it
$(swift auth)
curl -s -X GET "$OS_STORAGE_URL/<container>/<largeobject>?multipart-manifest=get" -H "X-Auth-Token:$OS_AUTH_TOKEN" | jq '.' > <largeobject>.json
sed -i 's/name/path/g' <largeobject>.json
sed -i 's/bytes/size_bytes/g' <largeobject>.json
sed -i '/hash/d' <largeobject>.json
sed -i '/last_modified/d' <largeobject>.json
sed -i '/content_type/d' <largeobject>.json
sed -i '/path/s/,$//g' <largeobject>.json
curl -i -X PUT -H "X-Auth-Token:$OS_AUTH_TOKEN" -T <largeobject>.json "$OS_STORAGE_URL/<destination_container>/<largeobject>?multipart-manifest=put`
Rename a container
You cannot rename a container. To do this, create a new container and re-upload the objects in it. However, Swift has the copy feature, which seems to provide better performance.
Time to upload 2Gb (1500 objects of 1M and 1 object of 500M):
time swift upload <container> ./*
real 69m26,159s
user 0m20,017s
sys 0m3,689s
swift list --lh -t <container>
2.0G
time for obj in $(swift list <container>);do swift copy -d /<other_container> <container> $obj; done
real 54m43,898s
user 12m38,060s
sys 1m34,394s
swift list
<other_container>
<container>
swift list --lh -t <other_container>
2.0G
We notice that for small objects, this does not change much. However, for objects of a slightly larger size (500M):
time swift upload <container> <object> --object-name <bis_object>
<bis_object>
real 15m51,525s
user 0m4,245s
sys 0m0,848s
time swift copy -d /<other_container> <container> <bis_object>
created container <other_container>
<container>/<bis_object> copied to /<other_container>/<bis_object>
real 0m11,924s
user 0m0,464s
sys 0m0,091s
LargeObject
Info
In this example, the LargeObject was uploaded in SLO mode.
Please refer to the Curl Command Memo documentation for the manifest upload.
time swift upload --use-slo --segment-size 1G <container> <largeobject>
<largeobject> segment 4
<largeobject> segment 2
<largeobject> segment 0
<largeobject> segment 1
<largeobject> segment 5
<largeobject> segment 3
<largeobject>
real 190m55,547s
user 0m57,906s
sys 0m14,246s
On a LargeObject, the swift copy command returns a 413 error:
swift copy -d /<destination_container> <container> <largeobject>
created container <cdestination_container>
Object COPY failed: https://storage.gra.cloud.ovh.net/v1/AUTH_702xxxxxxxxxxxxxxxxxxxxxxxxxxdaf/<container>/<largeobject> 413 Request Entity Too Large [first 60 chars of response] b'<html><h1>Request Entity Too Large</h1><p>The body of your r'
So we need to start by moving the segments:
for obj in $(swift list <container_segments>);do swift copy -d /<destination_container_segments> <container_segments> $obj;done
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000000 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000000
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000001 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000001
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000002 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000002
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000003 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000003
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000004 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000004
created container <destination_container_segments>
<container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/00000005 copied to /<destination_container_segments>/<largeobject>/slo/1629978906.614903/6442450944/1073741824/0000000005
Then retrieve the manifest, adapt it and re-upload it
$(swift auth)
curl -s -X GET "$OS_STORAGE_URL/<container>/<largeobject>?multipart-manifest=get" -H "X-Auth-Token:$OS_AUTH_TOKEN" | jq '.' ><largeobject>.json
sed -i 's/name/path/g' <largeobject>.json
sed -i 's/bytes/size_bytes/g' <largeobject>.json
sed -i '/hash/d' <largeobject>.json
sed -i '/last_modified/d' <largeobject>.json
sed -i '/content_type/d' <largeobject>.json
sed -i '/path/s/,$//g' <largeobject>.json
curl -i -X PUT -H "X-Auth-Token:$OS_AUTH_TOKEN" -T <largeobject>.json "$OS_STORAGE_URL/<destination_container>/<largeobject>?multipart-manifest=put"
Get the space used in a container
swift list --lh -t <container>
8.4G
There is no concept of a folder in a container, however we can use the prefixes:
swift list -p <prefix> <container>
<prefix>/1.jpg
<prefix>/10.jpg
<prefix>/11.jpg
<prefix>/12.jpg
<prefix>/13.jpg
<prefix>/14.jpg
<prefix>/15.jpg
<prefix>/16.jpg
<prefix>/17.jpg
<prefix>/18.jpg
<prefix>/2.jpg
<prefix>/3.jpg
<prefix>/4.jpg
<prefix>/5.jpg
<prefix>/6.jpg
<prefix>/7.jpg
<prefix>/8.jpg
<prefix>/9.jpg
swift list --lh -t -p <prefix> <container>
685K
Get user id
openstack user show --format json "${OS_USERNAME}" | jq -r '.id'
openstack token issue -f json | jq -r '.user_id'
Get cluster limits
Limit container size
To limit the size of a container, simply add the metadata 'X-Container-Meta-Quota-Bytes' associated with the limit to set up:
swift post -H "X-Container-Meta-Quota-Bytes:<limit-in-bytes>" <container>
If the limit is exceeded, the following error is returned:
swift upload <container> logo_ovh.png
Object PUT failed: https://storage.gra.cloud.ovh.net/v1/AUTH_<account>/<container>/logo_ovh.png 413 Request Entity Too Large b'Upload exceeds quota.' (txn: txee1f2b77c26a424ebbda8-0062a1aa7f)
Consider using the --segment-size option to chunk the object
Go further
If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for assisting you on your specific use case of your project.
Join our community of users.
