On-prem file trigger - New CSV file in folder
1. How it works
This trigger monitors an on-premises folder. Based on a schedule you define, this trigger will check if there is a new CSV file inside that folder. It will read that CSV file and output the file metadata and CSV rows.
Use this trigger when you have an on-premises folder to which you frequently add CSV files. Examples of these files could be weekly sales reports or daily data dumps from other systems. Using this trigger, you can build a recipe to read those files and import the records into other system (e.g. Salesforce, databases).
This trigger supports batching, which allows you to process huge amount of data in shorter time.
Every time this trigger processes a file, it will rename the file and append the text
processingat the end of file name. This is to prevent the trigger from picking up the file again in the next run, and make sure it only pickup new file in folder.
Trigger renames files when processing them
2. Input fields
For this trigger to work, you need to configure 4 sections:
- Schedule settings
- Folder settings
- CSV settings
- Batch size
2.1 Schedule settings
Configure this to tell the trigger how often it should check for new CSV file in on-premises folder. There are 2 ways to schedule: interval and date/time.
Using interval, you are telling the trigger to check after a certain amount of time.
Using date/time, you are telling the trigger to check at a specific time every day.
Optionally, you can choose which days of the week the trigger should run. Use
Add/remove optional fields at the bottom of the form to add this.
2.2 Folder settings
Configure this to specify which on-premises folder the trigger should monitor for new CSV file.
Optionally, you can define a naming pattern so the trigger only picks up files with certain names. Enable this input field using
Add/remove optional fields at the bottom of the form.
2.3 CSV settings
Configure this so the trigger will understand the structure of your CSV file. It is recommended that you upload a sample CSV file for fast and accurate setup.
2.4 Batch size
Lastly, set a batch size. Batch size determines how many CSV rows trigger will process at the same time. Learn more about batching here.
This is useful when you want to import the data from your CSV files into other apps like Salesforce or databases. Larger batch size will speed up this data import.
However, apps will often have an API limit of how much data they can receive at once. Therefore, try to experiment with different batch sizes or check the apps' API limits to see what works best for you.
The following is how this trigger's output looks like:
Expand the pill, you will see all columns in your CSV file, which can be used for mapping.
|File path (original)||File path before trigger processed this file.
e.g. C:/Program Files/sales_report.csv
|File path (processed)||When trigger processes a file, it will rename the file and mark as
e.g. C:/Program Files/sales_report.csv.1526632883663.processing
|File name (processed)||When trigger processes a file, it will rename the file and mark as
|File size (bytes)||File size in bytes.|
|Created time||The time the file was created.|
|Last modified time||The time the file was last modified.|
|Batch size||The batch size that you have set in the input.|
|Batch number||Trigger divides the CSV rows into multiple batches, then assign each batch to a job for processing.
Learn more about batching here.