Configure a data pipeline

Prerequisites

Before you create a data pipeline, ensure you have the following:

• Source application: Salesforce, NetSuite2, Jira, Coupa, or Marketo
• Destination data warehouse: Snowflake or Databricks

Refer to the Connect to sources and destinations guide for more information about data pipeline sources and destinations.

Configure your data pipeline source application

Complete the following steps to configure a data pipeline based on the source application you plan to use:

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline setup

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Click Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select Salesforce from Your Connected Source Apps.

7

Choose the Salesforce connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to Salesforce guide for more information.

Choose a Salesforce connection

8

Select the Objects you plan to use in your pipeline. The pipeline loads a list of available Salesforce objects, which allows you to:

• Search for specific objects.
• Expand objects to view related fields.
• Select multiple objects to sync.

Select objects to sync

9

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand object

You can expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

10

Choose how to handle schema changes:

• Select Auto-sync new fields to detect and apply schema changes automatically.
• Select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Unsynchronized schema changes, also known as schema drift, can cause issues if not managed. Refer to the Schema drift section for more information.

11

Complete the following steps in the Frequency field to specify how often the pipeline syncs data to the destination:

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select Hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

12

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

13

Click Save to save your progress before you configure your data pipeline destination.

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline setup

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Click Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select NetSuite2 from Your Connected Source Apps.

7

Choose the NetSuite2 connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to NetSuite2 guide for more information.

Choose a NetSuite2 connection

8

Select the Objects you plan to use in your pipeline. The pipeline loads a list of available Salesforce objects, which allows you to:

• Search for specific objects.
• Expand objects to view related fields.
• Select multiple objects to sync.

Select objects to sync

9

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand object

You can expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

10

Choose how to handle schema changes:

• Select Auto-sync new fields to detect and apply schema changes automatically.
• Select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Unsynchronized schema changes, also known as schema drift, can cause issues if not managed. Refer to the Schema drift section for more information.

11

Complete the following steps in the Frequency field to specify how often the pipeline syncs data to the destination:

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select Hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

12

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

13

Click Save to save your progress before you configure your data pipeline destination.

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline setup

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Select Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select Marketo from Your Connected Source Apps.

7

Choose the Marketo connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to Marketo guide for more information.

Choose a Marketo connection

8

Select the Objects you plan to use in your pipeline. The pipeline loads a list of available Salesforce objects, which allows you to:

• Search for specific objects.
• Expand objects to view related fields.
• Select multiple objects to sync.

Select objects to sync

9

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand object

You can expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

10

Choose how to handle schema changes:

• Select Auto-sync new fields to detect and apply schema changes automatically.
• Select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Unsynchronized schema changes, also known as schema drift, can cause issues if not managed. Refer to the Schema drift section for more information.

11

Complete the following steps in the Frequency field to specify how often the pipeline syncs data to the destination:

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

12

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

13

Click Save to save your progress before you configure your data pipeline destination.

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline setup

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Select Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select Jira from Your Connected Source Apps.

7

Choose the Jira connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to Jira guide for more information.

Choose a Jira connection

8

Select the Objects you plan to use in your pipeline. The pipeline loads a list of available Salesforce objects, which allows you to:

• Search for specific objects.
• Expand objects to view related fields.
• Select multiple objects to sync.

Select objects to sync

9

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand object

You can expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

10

Choose how to handle schema changes:

• Select Auto-sync new fields to detect and apply schema changes automatically.
• Select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Unsynchronized schema changes, also known as schema drift, can cause issues if not managed. Refer to the Schema drift section for more information.

11

Complete the following steps in the Frequency field to specify how often the pipeline syncs data to the destination:

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select Hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

12

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

13

Click Save to save your progress before you configure your data pipeline destination.

1

Select Create > Data pipeline.

2

Provide a Name for the data pipeline.

Data pipeline setup

3

Use the Location drop-down menu to select the project where you plan to store the data pipeline.

4

Select Start building.

5

Click the Extract new/updated records from source app trigger. This trigger defines how the pipeline retrieves data from the source application.

Configure the Extract new/updated records from source app trigger

6

Select Coupa from Your Connected Source Apps.

7

Choose the Coupa connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to Coupa guide for more information.

Choose a Coupa connection

8

Select the Objects you plan to use in your pipeline. The pipeline loads a list of available Salesforce objects, which allows you to:

• Search for specific objects.
• Expand objects to view related fields.
• Select multiple objects to sync.

Select objects to sync

9

Review and customize the schema for each selected object. When you select an object, the pipeline automatically fetches its schema to ensure the destination matches the source.

Expand object

You can expand any object to view its fields. Keep all fields selected to extract all available data, or deselect specific fields to exclude them from data extraction and schema replication.

10

Choose how to handle schema changes:

• Select Auto-sync new fields to detect and apply schema changes automatically.
• Select Block new fields to manage schema changes manually. This option may cause the destination to fall out of sync if the source schema updates.

Unsynchronized schema changes, also known as schema drift, can cause issues if not managed. Refer to the Schema drift section for more information.

11

Complete the following steps in the Frequency field to specify how often the pipeline syncs data to the destination:

1

Select the Time unit, such as Minutes, Hours, or Days.

2

Specify the sync interval in the Trigger every field.

For example, if you select Hours as the Time unit and enter 12 in the Trigger every field, the pipeline syncs every 12 hours. The minimum interval you can set is 30 minutes.

Configure sync frequency

12

Select a start date for the historical data sync in the When first started, this pipeline should pick up records from field. This defines the earliest date from which the pipeline extracts records. If you leave this field blank, the pipeline picks up all available records from the source.

You can't change this value after you run the pipeline. Refer to the When first started, this recipe should pick up events from section for more information.

13

Click Save to save your progress before you configure your data pipeline destination.

Configure your data pipeline destination

Complete the following steps based on the destination you plan to use:

Before you start the pipeline, ensure the schema in Snowflake is newly created and empty. This prevents errors during the initial sync and ensures the pipeline can create destination tables without conflicts.

1

Click the Load data to target table in destination app action. This action defines how the pipeline replicates data in the destination.

Configure the Load data to target table in destination app action

2

Select Snowflake in the Your Connected Destination App field.

3

Choose the Snowflake connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to a Snowflake destination guide for more information.

Choose a Snowflake connection

The Load data to target table in destination app action automatically replicates the object schema from the source to Snowflake. Explicit field mapping isn't required.

Workato pipelines automatically create destination tables based on the source schema. The pipeline also creates a stage and temporary tables to support data replication and update operations.

4

Select Save to save the pipeline.

Before you start the pipeline, ensure the schema in Databricks is newly created and empty. This prevents errors during the initial sync and allows the pipeline to create destination tables without conflicts.

1

Click the Load data to target table in destination app action. This action defines how the pipeline replicates data in the destination.

Configure the Load data to target table in destination app action

2

Select Databricks in the Your Connected Destination App field.

3

Choose the Databricks connection you plan to use for this pipeline. Alternatively, click + New connection to create a new connection. Refer to the Connect to a Databricks destination guide for more information.

Choose a Databricks connection

The Load data to target table in destination app action automatically replicates the object schema from the source to Databricks. Explicit field mapping isn't required.

Workato pipelines automatically create destination tables based on the source schema. The pipeline also creates a stage and temporary tables to support data replication and update operations.

4

Select Save to save the pipeline.

Start your data pipeline

Select Start pipeline to start the data pipeline. After you start the pipeline, it syncs selected objects and loads historical data.

Start your data pipeline

You can also choose to Move, Edit, or Apply tags to the pipeline.

Refer to the Monitor data pipeline recipes guide to learn how to monitor pipeline activity and troubleshoot issues.