Google sheets

Spreadsheet data consumer designed to extend processing capabilites

Overview

With Joule’s Google Sheets connector, you can seamlessly pull row data from spreadsheets, transform it, and stream it into your real-time pipelines.

Client library: com.google.api-client:google-api-client:2.7.0

Examples & DSL attributes

This configuration sets up the consumer to read row data from a specified Google Spreadsheet, with the nasdaq tab defined as the source. Custom deserialization can also be applied to transform the incoming data as needed.

google sheets consumer:
  application name: Joule google sheets test
  spreadsheetId: 1PNhq2t7fJ8AqJBmlSvywdpI4maFaAaUC
  range: 'Nasdaq'

  event type: 'Nasdaq'
  polling frequency: 5
  header included: true

  oauth:
      credentials filepath: /home/joule/credentials/sheets/credentials.json

Setting up

1

Setup google project and enable spreadsheet API

Create a project and enable Google Spreadsheet API.

2

Configure consent and download credentials

To access the spreadsheet Joule needs the user credentials to be created and downloaded. This can be done from the Google Developers Console.

3

Prepare the spread with data

Load the spreadsheet with data you need to process. This data relates to the example above.

Copy the spreadsheet id from the URL e.g. https://docs.google.com/spreadsheets/d/1PNhq2t7fJ8AqJBmlSvywdpI4maFaAaUC-BUTyIc8utg

4

Define and deploy source configuration

google sheets consumer:
  application name: Joule google sheets test
  spreadsheetId: 1PNhq2t7fJ8AqJBmlSvywdpI4maFaAaUC
  range: 'Nasdaq'

  event type: 'Nasdaq'
  polling frequency: 5
  header included: true

  oauth:
      credentials filepath: /home/joule/credentials/sheets/credentials.json

5

Now you are ready to subscribe and use the data

Deploy a use case that subscribes to the spreadsheet.

Follow the official google documentation for more information on setting up projects on GCP

Attributes schema

Attribute	Description	Data Type	Required
application name	Application name	String
spreadsheetId	Spreadsheet id that is embedded in the url	String
range	Target sheet and start cell	String
event type	Specify the emitting StreamEvent type	String
polling frequency	Frequency on minutes the spreadsheet is called to download data	Integer Default: 1 minute
header included	Boolean to inform the default deserializer the first row is the header	Boolean Default: true
oauth	OAuth configuration	See OAuth section
deserialiser	For custom deserialization	See Deserializer documentation

OAuth Attribute

Attribute	Description	Data Type	Required
credentials filepath	Path to the google OAuth consent credentials file.	String