AWS S3

Scheme

The scheme you can use to access your files in Amazon Simple Storage is

s3://

Configuration

The configuration of the Amazon Web Services Simple Cloud Storage can be done through a variety of ways. Most require you to have an Access Key and a Secret Key.

Best practice is to create a specific IAM user for Apache Hop so that if needed you can fine-tune the permissions (set it to read-only for example) or indeed delete the user if it’s no longer needed.

For a complete list see Working with credentials in the AWS documentation.

Below are 2 popular ways of configuring access.

Credentials file

You can create a file in your home folder in the .aws/credentials file which then contains content like this:

properties

[default]
aws_access_key_id = yourSecretKey
aws_secret_access_key = a-long/series-of-characters-for-your-access-key

Variables

You can set the following system environment variables:

AWS_ACCESS_KEY_ID : set it to your AWS access key
AWS_SECRET_ACCESS_KEY : set it to your secret access key

Running Hop in Docker? See Cloud storage credentials to pass these settings to the container.

S3-compatible services

Storage services that implement the S3 API can be used through this plugin as well. Create an S3 Connection metadata item and fill in the following options:

Endpoint URL : the endpoint of the service, for example https://s3.example-region.example.com. Leave this empty for Amazon S3, where the endpoint is derived from the region.
Path-style access : check this when the service expects the bucket in the path (endpoint/bucketName) instead of in the hostname (bucketName.endpoint). This option is only applied when an endpoint URL is set.

The name of the connection is registered as a VFS scheme of its own, so a connection named storage gives you paths like storage://bucketName/path/to/file.csv. Explore on the S3 connection editor opens the VFS File Explorer at name:///.

This works with Amazon S3 and with S3-compatible services such as Backblaze B2, Cloudflare R2 and MinIO. Check the documentation of your service for the endpoint to use and for whether path-style access is required.

Part size

You can set the default part size for new files on S3 by setting the following variable:

HOP_S3_VFS_PART_SIZE

This needs to be set as a global Hop configuration variable (in hop-config.json). You can use the Tools/Edit config variables menu in Hop GUI or you can use the hop-conf command line tool to do so.

Acceptable are 5MB as a minimum and 5GB as a maximum value.

If this variable is not set, 5MB will be taken as value and a message will be printed on the console while creating files on S3:

Part size null less than minimum of 5MB, set to minimum.

Read block size

Transforms that read a file in blocks, such as Text File Input fetching ahead, download it in blocks of the Read block size of the S3 connection, 8MB when it’s left empty. Every block is downloaded with one request for exactly its bytes, several at a time, so one file uses several connections. A request read to its end hands its connection back to the client, so the connections are reused.

Larger blocks make fewer requests per file and take more memory; smaller blocks start the first rows sooner.

Maximum connections

The Maximum connections of an S3 connection is how many HTTP connections to S3 can be open at the same time, 50 when it’s left empty. One connection to S3 is often limited to a few MB/s, so reading many blocks at the same time is how a large file is read fast, and every block in flight needs a connection of its own. Text File Input running in parallel keeps up to copies × read-ahead blocks requests in flight: 20 copies with 4 blocks each need 80 connections. With fewer connections allowed than that, the other requests wait for a free one.

Usage and testing

To test if the configuration works you can simply upload a small CSV file in an S3 bucket and then use File/Open in Hop GUI. Then you type in s3:// as a file location and hit enter (or click the refresh button). Browse to the CSV file you uploaded and open it. If all is configured correctly you should be able to see the content in the Hop GUI.