Google PubSub

Overview

Hightouch supports streaming events from Google Pub/Sub in realtime.

Simply configure your Google Pub/Sub source in the Hightouch UI, and Hightouch will automatically extract, transform, and load your data into Hightouch Events. Your data can then be synced to your warehouse or streamed to our catalog of realtime destinations.

Setup

To get started, create an event source, select Google Pub/Sub and follow the steps below.

Configure your source

Configure the following fields in Hightouch:

• Credentials: GCP credentials for connecting to Pub/Sub.
• Subscription: The full name of the Pub/Sub subscription to consume events from.
• Project ID: The Google Cloud Project ID.
• Regional API Endpoint: List of endpoints

Grant permissions

Hightouch believes in the principle of least privilege. We ask for no more permissions than necessary. For the Google Pub/Sub event source, Hightouch requires roles/Pub/Sub.subscriber and roles/Pub/Sub.viewer. Ensure you run the command in your Google Cloud Console to grant Hightouch's service account the correct permissions.

Message format

Hightouch Events supports PubSub messages encoded using Avro and Protobuf as long as a PubSub Schema is configured and associated with the topic. Otherwise, messages need to be encoded as JSON.

Event Structure

All events are ingested into Hightouch as track events. The Pub/Sub message id, data, and attributes are exposed as custom properties on the track event. Metadata such as subscription are also exposed in the context.Pub/Sub object.

For example, the following Pub/Sub message:

{
  "id": "123",
  "data": {
    "user_id": "user_123",
    "amount": 123.45
  },
  "attributes": {
    "a1": "abc"
  }
}

Will be ingested into Hightouch as:

{
  "type": "track",
  "event": "Google Pub/Sub Event",
  "properties": {
    "id": "123",
    "data": {
      "user_id": "user_123",
      "amount": 123.45
    },
    "attributes": {
      "a1": "abc"
    }
  },
  "context": {
    "Pub/Sub": {
      "subscription": "projects/my-project/subscriptions/my-subscription"
    }
  },
  "timestamp": "2024-07-29T14:52:37.648Z"
}

Field mappings

You can configure how Hightouch transforms messages from your Pub/Sub topic to Hightouch track events. Mappings can be configured for the following fields:

• event
• userId
• anonymousId
• messageId
• timestamp

You can also transform your events using Functions in order to standardize the schema or enrich its contents.

Schema Enforcement

Event Contracts can be attached to Google Pub/Sub sources just like any other Event Source.

Tips and troubleshooting

If you encounter an error or question not listed below and need assistance, don't hesitate to reach out. We're here to help.