How to access object storage from CloudFerro Cloud using boto3
In this article, you will learn how to access object storage from CloudFerro Cloud using Python library boto3.
What We Are Going To Cover
Prerequisites
No. 1 Account
You need a CloudFerro Cloud hosting account with access to the Horizon interface: https://horizon.cloudferro.com.
No. 2 Generated EC2 credentials
You need to generate EC2 credentials. Learn more here: How to generate and manage EC2 credentials on CloudFerro Cloud
No. 3 A Linux or Windows computer or virtual machine
This article was written for Ubuntu 22.04 and Windows Server 2022. Other operating systems might work, but are out of scope of this article and might require you to adjust the commands supplied here.
You can use this article both on Linux or Windows virtual machines or on a local computer running one of those operating systems.
To build a new a virtual machine hosted on CloudFerro Cloud cloud, one of the following articles can help:
No. 4 Additional Python libraries
We are going to use two Python libraries to push files to S3 server on CloudFerro Cloud:
- requests
Commonly used for HTTP access.
- requests_aws4auth
Library needed to sign uploads to S3 server. Install them with
pip install requests
pip install requests-aws4auth
and import with Python commands
import boto3
import requests
from requests_aws4auth import AWS4Auth
Terminology: container and bucket
The terms container (used by the Horizon dashboard) and bucket (used by boto3) are used here interchangeably, to describe the same thing.
Preparing the environment
To perform the examples in this article, you will need to have a working installation of Python 3 and the library boto3. The exact method of installation will depend on your goals and habits of working with Python.
Ubuntu 22.04
Method 1: Using virtualenvwrapper and pip
If you are a seasoned Python user and are working on multiple projects, you are bound to use Python virtual environments. It is advisable to create a new virtual environment for boto3 and here is one way to do it: How to install Python virtualenv or virtualenvwrapper on CloudFerro Cloud.
As mentioned in that article, use command workon to enter virtual environment. For example, if your environment is called managing-files execute:
workon managing-files
Once the environment is activated, install boto3:
pip3 install boto3
Method 2: Using apt
If you, however, don’t need such an environment and want to make boto3 available globally under Ubuntu, use apt. The following command will both install Python3 and the boto3 library:
sudo apt install python3 python3-boto3
Windows Server 2022
If you are using Windows, follow this article to learn how to install boto3: How to Install Boto3 in Windows on CloudFerro Cloud
How to use the examples provided?
Each of the examples provided can serve as standalone code. You should be able to
enter the code in your text editor,
define appropriate content for the variables provided,
save it as a file and
run.
You can also use these examples as a starting point for your own code.
Each example has three parts:
- blue rectangle Import necessary libraries
It is a standard Python practice to import all libraries needed for the code later on. In this case, you first import library boto3 as well as any others if such is the case.
- red rectangle Define input parameters
A typical call in boto3 library will look like this:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
For each call, several variables must be present:
aws_access_key_id – from Prerequisite No. 2
aws_secret_access_key – also from Prerequisite No. 2
endpoint_url – s3.waw3-1.cloudferro.com
region_name – US
Important
Use the concrete value of endpoint_url for the cloud you are working with. The possible values are:
WAW4-1 https://s3.waw4-1.cloudferro.com
WAW3-1 https://s3.waw3-1.cloudferro.com
Other variables may be needed on a case by case basis:
name,
path
and possibly others. Be sure to enter the values for all those additional variables before running the examples.
- green rectangle boto3 call
Most of the preparatory code stays the same from file to file and this last part is where various actions take place. Sometimes it will be only one line of code, sometimes several, usually a loop.
Running Python code
Once you have provided all the values to the appropriate variables, you can save the file with a .py extension. One of the ways of running Python scripts is using the command line. To run such a script, first navigate to the folder in which the script is located. After that, execute the command below, but replace script.py with the name of the script you wish to run.
python3 script.py
python script.py
Make sure that the name and/or location of the file is passed to the shell correctly – watch out for any spaces and other special characters.
Important
In these examples, even in case of potentially destructive operations, you will not be asked for confirmation. If, for example, a script saves a file and a file already exists under the designated name and location, it will likely be overwritten. Please be sure that that is what you want before running the script, or enhance the code with checks whether the file already exists.
In all the examples that follow, we assume that the corresponding files and buckets already exist. Adding code for checks of that kind is out of scope of this article.
Creating a container
To create a new container, first select a name for your bucket and enter it in the name variable. On CloudFerro Cloud cloud, use only letters, numbers and hyphens.
boto3_create_bucket.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
s3.Bucket(name).create()
except Exception as issue:
print("The following error occurred:")
print(issue)
Successful execution of this code should produce no output.
To test whether the bucket was created, you can, among other things, list buckets as described in section Listing buckets below.
Troubleshooting creating a container
Bucket already exists
If you receive the following output:
The following error occurred:
An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: Unknown
it means that you cannot choose this name for a bucket because a bucket under this name already exists. Choose a different name and try again.
Invalid characters used
If you used wrong characters in the container name, you should receive an error similar to this:
Invalid bucket name "this container should not exist": Bucket name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN matching the regex "^arn:(aws).*:(s3|s3-object-lambda):[a-z\-0-9]*:[0-9]{12}:accesspoint[/:][a-zA-Z0-9\-.]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]{1,63}$"
To resolve, choose a different name - on CloudFerro Cloud cloud, use only letters, numbers and hyphens.
Listing buckets
This code allows you to list buckets.
boto3_list_buckets.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
try:
s3=boto3.client('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
print(s3.list_buckets()['Buckets'])
except Exception as issue:
print("The following error occurred:")
print(issue)
The output should be a list of dictionaries, each providing information regarding a particular bucket, starting with its name. If two buckets, my-files and my-other-files, already exist, the output might be similar to this:
[{'Name': 'boto3', 'CreationDate': datetime.datetime(2025, 4, 30, 8, 52, 30, 658000, tzinfo=tzutc())}, {'Name': 'mapsrv', 'CreationDate': datetime.datetime(2025, 1, 7, 12, 43, 50, 522000, tzinfo=tzutc())}, {'Name': 'mapsrv-rasters', 'CreationDate': datetime.datetime(2025, 1, 7, 12, 44, 23, 39000, tzinfo=tzutc())}, {'Name': 's2ffb', 'CreationDate': datetime.datetime(2025, 1, 7, 13, 1, 37, 806000, tzinfo=tzutc())}, {'Name': 's2ffb82', 'CreationDate': datetime.datetime(2025, 1, 7, 13, 1, 45, 124000, tzinfo=tzutc())}, {'Name': 's2landkreise', 'CreationDate': datetime.datetime(2025, 1, 7, 13, 2, 6, 887000, tzinfo=tzutc())}, {'Name': 's2lkffb', 'CreationDate': datetime.datetime(2025, 1, 7, 13, 2, 20, 140000, tzinfo=tzutc())}, {'Name': 'yy-1', 'CreationDate': datetime.datetime(2025, 2, 20, 15, 38, 40, 454000, tzinfo=tzutc())}]
To simplify the output, use a for loop with print statement to display only names of your buckets, one bucket per line:
boto3_list_one_by_one.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
try:
s3=boto3.client('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
for i in s3.list_buckets()['Buckets']:
print(i['Name'])
except Exception as issue:
print("The following error occurred:")
print(issue)
An example output for this code can look like this:
boto3
mapsrv
mapsrv-rasters
If you have no buckets, the output should be empty.
Checking when a bucket was created
Use this code to check the date on which a bucket was created. Enter the name of that bucket in variable name
boto3_check_date.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
print(s3.Bucket(name).creation_date)
except Exception as issue:
print("The following error occurred:")
print(issue)
The output should contain the date the bucket was created, in format of Python datetime object:
2024-01-23 14:21:03.070000+00:00
Listing files in a bucket
To list files you have in a bucket, provide bucket name in name variable.
boto3_list_files_in_bucket.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
bucket=s3.Bucket(name)
for obj in bucket.objects.filter():
print(obj)
except Exception as issue:
print("The following error occurred:")
print(issue)
Your output should contain the list of your files, like so:
s3.ObjectSummary(bucket_name='my-files', key='some-directory/')
s3.ObjectSummary(bucket_name='my-files', key='some-directory/another-file.txt')
s2.ObjectSummary(bucket_name='my-files', key='some-directory/first-file.txt')
s3.ObjectSummary(bucket_name='my-files', key='some-directory/some-other-directory/')
s3.ObjectSummary(bucket_name='my-files', key='some-directory/some-other-directory/some-other-file.txt')
s3.ObjectSummary(bucket_name='my-files', key='some-directory/some-other-directory/yet-another-file.txt')
s3.ObjectSummary(bucket_name='my-files', key='text-file.txt')
If there are no files in your bucket, the output should be empty.
Troubleshooting listing files in a bucket
No access to bucket
If your key pair does not have access to the chosen bucket, you should get the error like this:
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjects operation: Unknown
In this case, choose a different bucket, or a different key pair if you have one which can access it.
Bucket does not exist
If a bucket you chose does not exist, the error might be:
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the ListObjects operation: Unknown
If you need a bucket which uses that name, try to create it as explained in the section Creating a container above.
Listing files from particular path in a bucket
This example will list only objects from a certain path. There are two rules to follow for path variable:
End it with a slash
Do not start it with a slash.
As always, add the name of the bucket to name variable.
boto3_list_files_from_path
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
path = ''
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
bucket=s3.Bucket(name)
for obj in bucket.objects.filter(Prefix=path):
print(obj)
except Exception as issue:
print("The following error occurred:")
print(issue)
A standard output should be produced but if there are no files under chosen path, the output will be empty.
Uploading file to a bucket
To upload a file to the container, add the following content to variables:
Variable name |
What should it contain |
bucket_name |
The name of the bucket to which you want to upload your file. |
source_file |
The location of the file you wish to upload in your local file system. |
destination_file |
The path in your container under which you want to upload the file. Should only contain letters, digits, hyphens and slashes. |
Two caveats for variable destination:
Finish it with the name of the file you are uploading and
Do not start or finish it with a slash.
This is the code:
boto3_upload_files.py
import boto3
import requests
from requests_aws4auth import AWS4Auth
access_key = ''
secret_key = ''
bucket_name = 'boto3'
source_file = 'onefile.txt'
destination_file = 'onefile.txt'
import requests
from requests_aws4auth import AWS4Auth
def upload_file_to_object_storage(bucket_name, source_path, destination_path, access_key, secret_key):
"""
Upload a file to a CloudFerro S3 bucket.
Args:
bucket_name (str): Name of the destination S3 bucket.
source_path (str): Local path to the file to upload.
destination_path (str): Destination path in the bucket.
access_key (str): AWS access key.
secret_key (str): AWS secret key.
Returns:
response (requests.Response): The HTTP response object.
"""
region = 'RegionOne'
service = 's3'
endpoint = 'https://s3.waw3-1.cloudferro.com'
auth = AWS4Auth(access_key, secret_key, region, service)
with open(source_path, 'rb') as file_data:
file_content = file_data.read()
headers = {
'Content-Length': str(len(file_content)),
}
url = f'{endpoint}/{bucket_name}/{destination_path}'
response = requests.put(url, data=file_content, headers=headers, auth=auth)
return response
#demo
response = upload_file_to_object_storage(bucket_name, source_file, destination_file, access_key, secret_key)
if response.status_code == 200:
print(f"Uploaded {source_file} successfully to {bucket_name}/{destination_file}")
else:
print(f"Upload failed with status {response.status_code}: {response.text}")
This will upload file onefile.txt from local folder to bucket named boto3; the uploaded file will also be called onefile.txt. The function itself, upload_file_to_object_storage, is reusable and can be used in various contexts; here, the main program just prints one line report of what happened. You can also make the it
raise an exception on upload error,
retry automatically,
upload many files in a batch, or
log success/failure in a file
but that would be out of scope of this article.
Troubleshooting uploading file to a bucket
File you want to upload does not exist
If you specified a non-existent file in variable source, you should get error similar to this:
The following error occurred:
[Errno 2] No such file or directory: 'here/wrong-file.txt'
To resolve, specify the correct file and try again.
Downloading file from a bucket
To save file from a bucket to your local hard drive, fill in the values of the following variables and run the code below:
Variable name |
What should it contain |
name |
The name of the bucket from which you wish to download your file. |
source |
The path in the container from which you wish to download your file. |
destination |
The path in your local file system under which you wish to save your file. |
Do not start or finish the variable source with a slash.
boto3_download_file.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
source='onefile.txt'
destination='onefile2.txt'
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
bucket=s3.Bucket(name)
bucket.download_file(source, destination)
except Exception as issue:
print("The following error occurred:")
print(issue)
Successful execution of this code should produce no output.
Troubleshooting uploading file to a bucket
File does not exist in bucket
If a file you chose does not exist in the bucket, the following error should appear:
The following error occurred:
An error occurred (404) when calling the HeadObject operation: Not Found
To resolve, make sure that the correct file was specified in the first place.
Removing file from a bucket
To remove file from your bucket, supply the name of the bucket to the variable name and its full path to the variable path.
boto3_remove_file_from_bucket.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
path = 'onefile.txt'
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
s3.Object(name, path).delete()
except Exception as issue:
print("The following error occurred:")
print(issue)
Successful execution of this code should produce no output.
Removing a bucket
To remove a bucket, first remove all objects from it. Once it is empty, define variable name for the bucket you want to remove and execute the code below:
boto3_remove_bucket.py
import boto3
access_key = ''
secret_key = ''
name = 'boto3'
endpoint_url='https://s3.waw3-1.cloudferro.com'
region_name='US'
try:
s3=boto3.resource('s3',
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
endpoint_url=endpoint_url,
region_name=region_name)
s3.Bucket(name).delete()
except Exception as issue:
print("The following error occurred:")
print(issue)
Successful execution of this code should produce no output.
Troubleshooting removing a bucket
Bucket does not exist or is unavailable to your key pair
If the bucket does not exist or is unavailable for your key pair, you should get the following output:
The following error occurred:
An error occurred (NoSuchBucket) when calling the DeleteBucket operation: Unknown
Bucket is not empty
If the bucket is not empty, it cannot be deleted. The message will be:
The following error occurred:
An error occurred (BucketNotEmpty) when calling the DeleteBucket operation: Unknown
To resolve, remove all objects from the bucket and try again.
General troubleshooting
No connection to the endpoint
If you do not have connection to the endpoint (for example because you lost Internet connection), you should get output similar to this:
The following error occurred:
Could not connect to the endpoint URL: "https://s3.waw3-1.cloudferro.com/"
If that is the case, make sure that you are connected to the Internet. If you are sure that you are connected to the Internet and no downtime for CloudFerro Cloud object storage was announced, contact CloudFerro Cloud customer support: Helpdesk and Support
Wrong credentials
If the access and/or secret keys are wrong, a message like this will appear:
The following error occurred:
An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: Unknown
Refer to Prerequisite No. 2 for how to obtain valid credentials.
Bucket does not exist or is unavailable to your key pair
If the bucket you chose for this code does not exist or is unavailable for your key pair, you can get different outputs depending on the command you are executing. Below are two such examples:
None
The following error occurred:
An error occurred (NoSuchBucket) when calling the DeleteObject operation: Unknown
What To Do Next
boto3 can also be used to access the EODATA repository on virtual machines hosted on CloudFerro Cloud cloud. Learn more here: /eodata/How-to-access-EODATA-using-boto3-on-CloudFerro-Cloud
Another tool for accessing object storage in the cloud is s3cmd: How to access object storage from CloudFerro Cloud using s3cmd.