# File streaming

File streaming is a technique used to process a large file by dividing it into smaller pieces, transferring one piece at a time, and then putting the pieces back in order.

For example, when Workato uses file streaming to transfer a file between systems, each piece is downloaded from the source and uploaded to the destination in sequence.

File streaming is useful when processing a large file is impractical or impossible due to size or memory limits at the source or destination app. With file streaming, you can stay within each app's constraints because Workato processes smaller chunks one at a time. This makes it possible for Workato to transfer files of any size between apps that support streaming. Workato also supports file streaming for large volume storage and data transformations.

The following diagram illustrates how file streaming works:

How file streaming works How file streaming works

# Use cases

You can use Workato's file streaming for use cases such as the following:

  • Send data in bulk to applications
  • Receive data from applications using bulk triggers or bulk actions
  • Store large files in Workato FileStorage or other file systems such as SFTP and S3
  • Send or receive data from SQL Transformations

For example, with file streaming, you can:

  • Extract a file with 1 million rows of leads data from an on-prem file server and load it into a Google BigQuery table
  • Accumulate 100,000 rows from a non-streaming source such as HubSpot to Workato FileStorage using batch actions, and then stream the rows from FileStorage to Salesforce bulk APIs all at once
  • Extract new or updated contact records from Marketo using bulk triggers and aggregate the records with other data sources in SQL Transformations

# Comparison with batch processing

Batch processing is when multiple jobs each process multiple records. The number of records that can be processed in a single job is determined by the batch size. Workato creates as many jobs as needed to process all relevant records, which makes it possible to process very large files. However, due to batch size and memory limits, this approach can result in a high volume of API calls, and it does not scale well as the volume of data increases.

In contrast, file streaming uses a single job to process all relevant records in a single call to a streaming API, chunk by chunk. This results in fewer API calls and faster processing, even when the file is very large.

Batch triggers and actions should typically be paired with other batch actions. Likewise, streaming-compatible data producers (such as file downloads, bulk triggers, and bulk export actions) must be paired with streaming-compatible data consumers (such as file uploads and bulk create, bulk upsert, and bulk import actions).

We recommend using triggers and actions that support file streaming whenever possible to help:

  • Simplify recipe design
  • Reduce task count
  • Reduce the number of API calls made between sources and destinations

# How to use file streaming

If either the source (such as a file download, bulk trigger, or bulk action) or the destination (such as a file upload or bulk action) supports file streaming, Workato automatically initiates file streaming when you pass a File contents or CSV contents datapill to the File contents input field. No additional configuration is required in the recipe.

Passing a File contents datapill to a File contents input field Passing a File contents datapill to a File contents input field

Note that the streaming transfer begins when the recipe reaches the consumer action step in the recipe. File streaming actions are usually long or deferred actions. As a result, streaming actions do not time out, no matter how long the transfer takes.

# Triggers and actions that support file streaming

The following table lists the triggers and actions that support file streaming:

Connector name Triggers and actions that support file streaming
Adobe Experience Manager
  • Upload asset by streaming action
Amazon S3
  • Download file contents action
  • Upload file (streaming) action
  • New file trigger
  • New/updated file or folder trigger
Anaplan
  • Download file action
  • Download dump file action
  • Upload file action
Azure Blob Storage
  • Download blob contents action
  • Upload blob action
BIM360
  • Download document in a project action
  • Upload document to a project action
Box
  • Download file action
  • Upload file using file contents action
Bynder
  • Upload asset action
  • Download asset action
Dropbox
  • Upload file using file contents action
Egnyte
  • Download file from selected folder action
  • Upload file action
File tools by Workato
  • Get file from URL action
  • Compress files action
Freshdesk
  • Get ticket attachments action
FTP/FTPS
  • Download large file action
  • Upload file action
Google Cloud Storage
  • Download object action
  • Upload object action
Google Drive
  • Download file action
  • Upload file action
  • New file or folder in folder hierarchy trigger
  • New file or folder trigger
Jira
  • Export new issues in Jira trigger
  • Export new/updated issues in Jira trigger
Marketo
  • Export New leads in Marketo trigger
  • Export New/updated leads in Marketo trigger
Microsoft Dynamics 365
  • Export New records trigger
  • Export New/updated records trigger
Microsoft Sharepoint
  • Upload file in library action
NetSuite
  • Export new records trigger
  • Export New/updated records trigger
OneDrive
  • Download File action
  • Upload large file via session action
On-prem
  • Download file action
  • Upload file action
  • Upload file (Legacy) action
  • New file in folder trigger
Outlook
  • Send email action
Percolate
  • Download asset action
PlanGrid
  • Download object action
Quickbase
  • Create and update records in bulk from CSV file action
Salesforce
  • Upload file action
ServiceNow
  • Upload attachment action
  • Get attachment contents action
  • Export New records trigger
  • Export New/updated records trigger
SFTP
  • Download file action
  • Upload file action
SMB
  • Download file action
  • Upload file action
Workato FileStorage
  • Get file contents action
  • New File trigger
Workfront
  • Download document action
Zoom
  • Download cloud recording action

BUILD CUSTOM CONNECTORS

To learn how to build custom connectors that use file streaming, refer to the Connector SDK file streaming guide.

SEE FILE STREAMING IN ACTION

View a sample recipe (opens new window) that uses file streaming to transfer files from an on-prem file system to Amazon S3.


Last updated: 10/14/2025, 5:22:27 PM