Configure a data pipeline

PRIVATE BETA

This feature is in private beta. Private beta features are available in private preview to selected customers. Customers must opt-in and be accepted into the beta.

During the private beta, Workato may update its functionality or change its availability without prior notice.

Prerequisites

Before you create a data pipeline, ensure you have the following:

• Source application: The beta release supports Salesforce as a source.
• Destination data warehouse: The beta release supports Snowflake as a destination.

Refer to the Connect to sources and destinations guide for more information about data pipeline sources and destinations.

Configure your data pipeline source application

Complete the following steps to configure a data pipeline using Snowflake as the source application:

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline set up

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Select Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select Salesforce from Your Connected Source Apps. This is the only supported source in the beta release.

7

Choose the Salesforce connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to Salesforce guide for more information.

Choose a Salesforce connection

8

Select the Objects you plan to use in your pipeline. The pipeline loads a list of available Salesforce objects, which allows you to:

• Search for specific objects.
• Expand objects to view related fields.
• Select multiple objects to sync.

Select objects to sync

9

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand object

You can expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

10

Select Auto-sync new fields to detect and apply schema changes automatically. Alternatively, select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Unsynchronized schema changes, also known as schema drift, can cause issues if not managed. Refer to the Schema drift section for more information.

11

Complete the following steps in the Frequency field to specify how often the pipeline syncs data to the destination:

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

12

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

13

Click Save to save your progress before you configure your data pipeline destination.

Configure your data pipeline destination

SNOWFLAKE DESTINATION SETUP

Before you start the pipeline, ensure the schema in Snowflake is newly created and empty. This prevents errors during the initial sync and ensures the pipeline can create destination tables without conflicts.

Complete the following steps to configure Snowflake as the destination:

1

Click the Load data to target table in destination app action. This action defines how the pipeline replicates data in the destination.

Configure the Load data to target table in destination app action

2

Select Snowflake in the Your Connected Destination App field. This is the only supported destination in the beta release.

3

Choose the Snowflake connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to a Snowflake destination guide for more information.

Choose a Snowflake connection

The Load data to target table in destination app action automatically replicates the object schema from the source to Snowflake. Explicit field mapping isn't required.

Workato pipelines automatically create destination tables based on the source schema. The pipeline also creates a stage and temporary tables to support data replication and update operations.

4

Select Save to save the pipeline.

Start your data pipeline

Select Start pipeline to start the data pipeline. After you start the pipeline, it syncs selected objects and loads historical data.

Start your data pipeline

You can also choose to Move, Edit, or Apply tags to the pipeline.

Refer to the Monitor data pipeline recipes guide to learn how to monitor pipeline activity and troubleshoot issues.