File Streaming

File streaming is the concept of reading and writing a file in smaller parts (chunks) in sequence. This allows us to transfer large files between apps or file systems without worrying about hitting the size limit of the sending and receiving systems. A common example is transferring records from a shared file system (SFTP) to a file hosting platform for analysis (Amazon S3).

How it works

When transferring a file from a source app to a destination app, Workato splits the file into smaller chunks and downloads them. These chunks are then uploaded to the destination app in separate requests. This allows us to stay within the size limit of both apps, regardless of the total size of the original file.

The following diagram illustrates how this works:

As a result, Workato is able to transfer files of any size between apps that support streaming.

How to use it

File streaming is supported only if both the source download and destination upload actions support this feature. It is automatically initiated when you map a File contents datapill (from the source app) to a File contents input field (of the destination app).

If either the source datapill or the destination input field does not support streaming, then standard data mapping is performed, i.e. the entire file is downloaded and then uploaded without any chunking. In this case, the file transfer is limited by the size limit imposed by the respective apps. The smaller of the 2 will be the resultant limit.

In either case, no additional configuration is required in the recipe.

Connector	Triggers/Actions that support file streaming
	Download	Upload
Amazon S3	New file trigger Download file action	Upload file action
Anaplan	Download file action	Upload file action
BigQuery		Load data into BigQuery
Box	Download file action
Bynder	Download asset action	Upload asset action
Microsoft Sharepoint		Upload file to folder action
On-prem files	New file in folder trigger Download file action	Upload file action
Quickbase		Create and update records in bulk from CSV file action
Salesforce		Insert object in bulk from CSV file action Update object in bulk from CSV file action Upsert object in bulk from CSV file action
SFTP		Upload file action
Zoom	Download cloud recording

Table of all current triggers/actions that support file streaming.

This sample recipe (opens new window) transfers files from an On-prem file system to Amazon S3 with file streaming.