Google Technology

Introduction

The Google technology stack is supported in Hop through a number of plugins. We briefly touch upon them below.

Pipeline Transforms

VFS

Apache VFS Support in Hop allows you to directly read from a multitude of file systems and protocols, including Google:

  • Google Drive: read and write data directly from and to Google Drive files and folders.

  • Google Cloud Storage: read and write data directly from and to files and folders in Google Cloud Storage buckets

Beam vs Google Cloud

When executing your pipeline using a Beam runner which is NOT DataFlow, make sure to pass the default Google cloud project ID by running:

gcloud config set project <project-id>

This affects Google Cloud specific APIs like BigQuery, Pub/Sub and others.