Google Cloud Storage VFS

Scheme

The scheme you can use to access your files in Google Cloud Storage is

gs://

You need to generate a key file for a service account to make it work. Go to the Google Cloud console to do this. Once you have a key file for your service account, with permissions to access your GCP storage, point to it with either a system environment variable called GOOGLE_APPLICATION_CREDENTIALS (standard Google way of doing this) or in the Options dialog in the 'Google Cloud' tab. You can also use hop-conf:

      -gck, --google-cloud-service-account-key-file=<serviceAccountKeyFile>
                            Configure the path to a Google Cloud service account JSON key file

Once done you will see a googleCloud entry in the central hop-config.json file:

json

{
  "googleCloud" : {
    "serviceAccountKeyFile" : "/path/to/your/google-key.json"
  }
}

Running Hop in Docker? See Cloud storage credentials to pass these settings to the container.

Without a key file, Hop falls back to Application Default Credentials, for example those of gcloud auth application-default login or of the machine it runs on in Google Cloud. If no credentials can be found, Hop does not report it when it starts, as most installations never use gs://. The first use of gs:// fails instead, with an error that says why, for example that the key file could not be read.

Options

The options below are the ones in the Plugins section of the configuration perspective. Each is also settable with hop-conf.

These options are stored under googleCloud in hop-config.json.

Option Description Default

Option	Description	Default
Account key file	The path to a Google Cloud service account JSON key file. Accepts variables.
Scan folders to find last modified data	Scans all files in a folder and uses the most recent modification data as the folder modification data. This may have a performance impact in buckets with lots of files.	`false`
Max number of attempts	Maximum number of attempts for a single request, including the first one. Accepts variables.	`6`
Initial retry delay (seconds)	Delay before the first retry, in seconds. Accepts variables.	`1`
Retry delay multiplier	Retry delay multiplier. Accepts variables.	`2.0`
Maximum retry delay (seconds)	Ceiling for the retry delay once the multiplier has been applied, in seconds. Accepts variables.	`32`
Total timeout (minutes)	Deadline for a request including all of its retries, in minutes. Accepts variables.	`50`
Initial RPC timeout (seconds)	Timeout for the first attempt, in seconds. Raised to the max RPC timeout if that is lower. Accepts variables.	`50`
RPC Timeout Multiplier	RPC Timeout Multiplier. Accepts variables.	`1.0`
Max RPC timeout (seconds)	Ceiling for a single attempt timeout, in seconds. Never lower than the initial RPC timeout. Accepts variables.	`50`
Connect timeout (seconds)	Socket connect timeout, in seconds. Accepts variables.	`20`
Read timeout (seconds)	Socket read timeout, in seconds. Accepts variables.	`20`
Retry non-idempotent operations	Also retry writes (create, delete, starting an upload). Google Cloud Storage only retries reads by default, so without this the retry settings above do not apply to writing files. Note that a retried delete can report "not found" when the first attempt already succeeded, and a retried create overwrites.	`false`
Cache TTL (seconds)	How long to cache folder listing results (in seconds, default 5). If not set, falls back to 10 seconds. Accepts variables.	`5`
Read block size	How many bytes one download request fetches when a file is read, like 8MB. Leave empty for 8MB. Transforms that fetch ahead, such as Text File Input, download blocks of this size, several at a time, each with one request. Accepts variables.

Account key file

The path to a Google Cloud service account JSON key file. Accepts variables.

Scan folders to find last modified data

Scans all files in a folder and uses the most recent modification data as the folder modification data. This may have a performance impact in buckets with lots of files.

false

Max number of attempts

Maximum number of attempts for a single request, including the first one. Accepts variables.

6

Initial retry delay (seconds)

Delay before the first retry, in seconds. Accepts variables.

1

Retry delay multiplier

Retry delay multiplier. Accepts variables.

2.0

Maximum retry delay (seconds)

Ceiling for the retry delay once the multiplier has been applied, in seconds. Accepts variables.

32

Total timeout (minutes)

Deadline for a request including all of its retries, in minutes. Accepts variables.

50

Initial RPC timeout (seconds)

Timeout for the first attempt, in seconds. Raised to the max RPC timeout if that is lower. Accepts variables.

50

RPC Timeout Multiplier

RPC Timeout Multiplier. Accepts variables.

1.0

Max RPC timeout (seconds)

Ceiling for a single attempt timeout, in seconds. Never lower than the initial RPC timeout. Accepts variables.

50

Connect timeout (seconds)

Socket connect timeout, in seconds. Accepts variables.

20

Read timeout (seconds)

Socket read timeout, in seconds. Accepts variables.

20

Retry non-idempotent operations

Also retry writes (create, delete, starting an upload). Google Cloud Storage only retries reads by default, so without this the retry settings above do not apply to writing files. Note that a retried delete can report "not found" when the first attempt already succeeded, and a retried create overwrites.

false

Cache TTL (seconds)

How long to cache folder listing results (in seconds, default 5). If not set, falls back to 10 seconds. Accepts variables.

5

Read block size

How many bytes one download request fetches when a file is read, like 8MB. Leave empty for 8MB. Transforms that fetch ahead, such as Text File Input, download blocks of this size, several at a time, each with one request. Accepts variables.

Read block size

Files are downloaded in pieces of the Read block size option, 8MB when it’s left empty, instead of the 2MB the Google Cloud client uses on its own. Transforms that read a file in blocks, such as Text File Input fetching ahead, download several blocks at a time, each with one request for exactly its bytes.

Retries, timeouts and stalled transfers

The Google Cloud Storage client retries timeouts and temporary errors, such as HTTP 503, by itself, following the options above. Hop logs every retry, so a slow or unreliable connection does not look like a hang:

Google Cloud Storage: No response in time while accessing gs://bucket/data.csv (Read timed out), trying again. The connection timeout is 20 seconds, the read timeout 20 seconds.

A download can also stall without failing. The client then reconnects and tries again within the same read, and never gives up by itself, whatever the number of attempts. Hop therefore watches every read and write of a gs:// file:

A read that makes no progress for three read timeouts, and at least 60 seconds, is reported. It is reported again after every further period without progress, and once more if it starts moving again.
A read that still makes no progress after the time the retry options allow for one request is ended. The transform reading the file then fails with an error that ends in giving up. That time is Max number of attempts × (Connect timeout + Read timeout), plus the retry delays between the attempts, and at most the Total timeout. With the default options that is 6 × (20 + 20) + 1 + 2 + 4 + 8 + 16 = 271 seconds. With an unlimited number of attempts (0), or a connect or read timeout of 0 (no timeout), only the total timeout applies. When the total timeout is shorter than the time after which a read is reported, the read is reported and ended when the total timeout is reached. To wait longer before giving up, raise the number of attempts or the timeouts.
An upload sends the file in parts of 16 MB, so a slow upload cannot be told apart from a stalled one. A write is reported when one part takes longer than it would at 100 KB per second, 164 seconds with the default options. A write is never ended.

These messages are written to the general Hop log with the prefix Google Cloud Storage:, for example in the console output of hop-run.

Usage and testing

To test if the configuration works type use File/Open in the GUI and type in gs:// as a file location. Then hit enter (or click the refresh button). Browse to a CSV, JSON or text file you uploaded and open it. If all is configured correctly you should be able to see the content in the Hop GUI.

Google Cloud Storage VFS

Scheme

Configuration

Options

Read block size

Retries, timeouts and stalled transfers

Usage and testing