File Streaming

File streaming is the concept of reading and writing a file in smaller parts (chunks) in sequence. This allows us to transfer large files between apps or file systems without worrying about hitting the size limit of the sending and receiving systems. A common example is transferring records from a shared file system (SFTP) to a file hosting platform for analysis (Amazon S3).

How it works

When transferring a file from a source app to a destination app, Workato splits the file into smaller chunks and downloads them. These chunks are then uploaded to the destination app in separate requests. This allows us to stay within the size limit of both apps, regardless of the total size of the original file.

The following diagram illustrates how this works:

File streaming

As a result, Workato is able to transfer files of any size between apps that support streaming.

How to use it

File streaming is supported only if both the source download and destination upload actions support this feature. It is automatically initiated when you map a File contents datapill (from the source app) to a File contents input field (of the destination app).

If either the source datapill or the destination input field does not support streaming, then standard data mapping is performed, i.e. the entire file is downloaded and then uploaded without any chunking. In this case, the file transfer is limited by the size limit imposed by the respective apps. The smaller of the 2 will be the resultant limit.

In either case, no additional configuration is required in the recipe.

Connector Triggers/Actions that support file streaming
Download Upload
Amazon S3
  • New file trigger
  • Download file action
  • Upload file action
  • Anaplan
  • Download file action
  • Upload file action
  • Box
  • Download file action
  • Microsoft Sharepoint
  • Upload file to folder action
  • On-prem files
  • New file in folder trigger
  • Download file action
  • Upload file action
  • Quickbase
  • Create and update records in bulk from CSV file action
  • Salesforce
  • Insert object in bulk from CSV file action
  • Update object in bulk from CSV file action
  • Upsert object in bulk from CSV file action
  • SFTP
  • Upload file action
  • Table of all current triggers/actions that support file streaming.

    This sample recipe transfers files from an On-prem file system to Amazon S3 with file streaming.

    results matching ""

      No results matching ""