Uploading to s3 with python script

Boto3 is AWS's SDK for Python which can be easily integrated.

What is the need of uploading assets to s3 with python?

A developer might need to upload assets to s3 bucket to use them publicly without needing them to store in their project which also increases the bundler size of the application. So if there are a large number of assets which are required to be uploaded to s3 then it becomes a tedious task to upload them manually on the s3 console. The object_url of the uploaded asset is required to use it in the code. So uploading to s3 can be automated using python.

Boto3

Boto3 is AWS's SDK that is used in python. Boto3 provides a simplified approach to interact with various services of AWS like s3, EC2 and many more. Boto3 can help in performing many tasks and some of them are listed below:

Different resources that we use in AWS can be created, read, updated and deleted using Boto3.
Providing access to different resources on AWS.
Running and stopping EC2 instances on AWS.
Boto3 also allows to work with lambda functions in AWS.

So many such tasks that are required to be done manually can be automated using python script.

Steps for uploading assets on s3 with python script

Make sure you have python installed.

 Install the boto3 in python using the command: pip install boto3

Now to securely use the AWS credentials open the terminal.
```
 export AWS_ACCESS_KEY_ID=your_access_key_id
 export AWS_SECRET_ACCESS_KEY=your_secret_access_key
 export AWS_REGION=your-region
```
This will securely store the credentials in the current session of the terminal. Once the terminal is closed. Credentials are required to be set again. This also asserts security.
Create a python script and create a folder and put all the assets in that folder which are required to be uploaded on s3.

Using the os module in python, we can use the credentials that were stored in terminal:

 import os
 import boto3

 access_key_id = os.environ.get('AWS_ACCESS_KEY_ID')
 secret_access_key = os.environ.get('AWS_SECRET_ACCESS_KEY')
 region = os.environ.get('AWS_REGION')

Now create an object of boto3 client to connect with various AWS services:

 s3 = boto3.client('s3', aws_access_key_id=access_key_id,
                   aws_secret_access_key=secret_access_key)

Create a variable by assigning it the path of the folder in which all the assets are residing:
```
 local_folder_path = './assets'
```

for filename in os.listdir(local_folder_path):
    if (filename.startswith('.')): # to skip any hidden files
        continue
    file_path = os.path.join(local_folder_path, filename)
    if (os.path.isfile(file_path)):
        file_key = f'{s3_folder_name}/{filename}'
        try:
            s3.upload_file(file_path, bucket_name, file_key)
            object_url = f'https://{bucket_name}.s3.{region}.amazonaws.com/{file_key}'
            print(
                f'File uploaded to s3 successfully with object_url: {object_url}')
        except FileNotFoundError:
            print(f'This file is not found: {file_path}')
        except Exception as e:
            print(f'Error while uploading file on s3: {str(e)}')

Looping through the folder using the os module. We also need to skip the hidden files. The file_path looks something like ./assets/your_file.png.
We are creating a file key which will relate to where the asset will be stored. We will use the upload_file() method to upload to s3. The object_url that we see on s3 after uploading is the form that we are creating in the code.

In the terminal object_url of all the uploaded files will be printed.

Conclusion

Boto3 allows developers to automate different AWS tasks and one of them is uploading assets to s3 and it provides an intuitive way to do so. This script is executed with security and automates bulk upload on s3.

Anshik Gupta's Blog