Sync types

Data pipeline recipes support two sync types:

• Full sync: Loads historical data when the pipeline starts.
• Incremental sync (CDC): Captures new, modified, or deleted records on a schedule.

Each sync type ensures the destination remains accurate, up to date, and aligned with the source application.

Full sync

The full sync runs once when the pipeline starts. It loads historical data from the selected source objects into the destination. By default, the pipeline fetches all records when the When first started, this pipeline should pick up records from field is blank. You can also set a specific start date to limit the sync window.

The pipeline replicates the source schema, creates new tables in the destination, and processes each object as a separate run in parallel. It also creates a temporary stage in Snowflake to upload data before loading it into destination tables.

The full sync assumes the destination is empty. If the destination contains existing tables created by Workato, the pipeline overwrites them.

The pipeline labels each run as a full sync during the initial run because it processes all records. This sync runs automatically when the pipeline starts and doesn't require any manual setup or selection.

How a full sync works

The full sync extracts all data starting from the When first started, this pipeline should pick up records from timestamp onwards. If you leave this field blank, the pipeline retrieves all available records from the source.

The pipeline creates one run per object and loads data into newly created tables at the destination.

The Runs tab displays each object's status as In Progress until the pipeline finishes loading all records. Workato doesn't provide an estimated completion time because sync duration depends on data volume. The pipeline transitions to incremental syncs after it completes the initial load.

INITIAL SYNC COMPLETION

The pipeline doesn't run incremental syncs until the full sync completes.

Incremental sync (CDC)

After the full sync completes, the pipeline runs incremental syncs at scheduled intervals to track and apply changes from the source system.

How an incremental sync works

The pipeline extracts new, updated, or deleted records from the source at scheduled intervals. It only processes changes since the last successful sync. Before it adds records to the destination, the pipeline detects schema changes, such as new fields or updated field sizes, and updates the destination tables. The pipeline processes each object as a separate run and syncs all selected objects in parallel.

The pipeline doesn't reprocess historical data for newly added fields. It starts capturing values only from the time it detects the field. It doesn't start a new incremental sync until the previous one finishes. If a sync runs longer than expected, the pipeline skips the next scheduled run.

The Runs tab tracks each execution's progress and duration. Refer to Object runs for more information on real-time sync monitoring.

INCREMENTAL SYNC EXECUTION

The pipeline skips the next scheduled sync if the current one is still in progress.