Configure Algolia as a data pipeline source ​

Set up Algolia as a data pipeline source to extract index records, index configuration, search and click analytics, and API logs into your destination.

Use this guide to prepare an Algolia API key, set up a connection, configure your pipeline, add objects, review sync behavior, and understand known limitations.

Features supported ​

The following features are supported when you use Algolia as a pipeline source:

• Cloud connectivity: Connect to Algolia over HTTPS. Workato reads index data from https://<APPLICATION_ID>-dsn.algolia.net. Analytics data comes from https://analytics.us.algolia.com or https://analytics.de.algolia.com, depending on the analytics region you select. On-prem agents aren't required.
• Full sync and incremental sync: Supports full sync and incremental sync modes, configured per object. Analytics objects and Logs default to incremental sync, and index objects, such as Records and Rules, support full sync only. Refer to Sync modes for more information.
• Object-level selection: Select application-level objects and per-index objects, including Records, Index Settings, Rules, Synonyms, and 14 analytics objects, to sync as separate tables in your destination. Refer to Supported objects for the full list.
• Records schema discovery: Detect the attributes in each index from a sample of its records, with a fallback column for attributes outside the sample. Refer to Records schema discovery for more information.
• Delete tracking: Reflect records deleted in Algolia for full-sync objects on the next sync. Refer to Delete tracking for more information.
• Schema drift detection and handling: Detect and apply schema changes automatically with Auto-sync new fields, or keep the schema fixed with Block new fields.
• Field-level data protection: Hash or replicate sensitive fields as is before the data reaches your destination.
• Configurable sync frequency: Schedule syncs on a time-based interval or with a cron expression. The minimum supported interval is 15 minutes.

Prerequisites ​

Connecting Algolia as a data pipeline source requires:

• An Algolia application and its application ID
• An Algolia API key with the listIndexes, browse, settings, logs, and analytics ACLs. You can use the Admin API key or a restricted API key. You can't use a Search-only API key.
• Click analytics and conversion events sent to Algolia through the Insights API, if you plan to sync click and conversion analytics objects. Refer to Click and conversion analytics objects for more information.

Recommended permissions ​

Grant the following ACLs to your API key to sync the supported Algolia objects:

Every connection requires listIndexes, because Workato uses it to verify the connection and discover the indexes in your application.

Refer to Algolia's API keys guide to create a restricted API key and assign ACLs.

Supported connection types ​

Algolia data pipelines support API key authentication. Provide your Algolia application ID and an API key with the ACLs listed in Recommended permissions.

Connect to Algolia ​

Complete the following steps to connect Algolia as a data pipeline source:

Configure the pipeline ​

Complete the following steps to configure Algolia as your data pipeline source:

[minute] [hour] [day of month] [month] [day of week]

0 8 * * *

Supported objects ​

Algolia data pipelines sync index content, index configuration, analytics, and API logs from your Algolia application. The following tables list the supported objects, grouped by category. Each object syncs as a separate table in your destination. Index-level objects appear once for each index in your application.

Application objects ​

These objects sync data for your entire Algolia application:

Indices syncs one row per index in your application. Logs syncs recent API request logs. Refer to Logs retention for the limits that apply to Logs.

Index objects ​

These objects sync the content and configuration of a single index:

Index Settings syncs one row per index. Records uses a schema discovered from your index. Refer to Records schema discovery for more information.

Search analytics objects ​

Search analytics objects sync daily metrics for a single index, with one row per day or one row per query, record, country, or filter attribute per day. These objects require the analytics ACL, and fail with a permissions error if your API key or Algolia plan doesn't allow Analytics API access.

Click and conversion analytics objects ​

Each object syncs click and conversion metrics by day for one index. Search No Click and Click Position sync one row per query or click position per day, and the rest sync one row per day. They require the analytics ACL and click events sent to Algolia through the Insights API. Conversion Rate also requires conversion events.

Sync modes ​

Algolia data pipelines support full sync and incremental sync. The sync mode is configured per object when you add it to your pipeline.

Full sync ​

A full sync reads all available records within the selected object's extraction window on every run and replaces the record set in your destination table. Indices, Records, Index Settings, Rules, and Synonyms don't have a timestamp cursor, so they support full sync only.

Analytics objects and Logs also support full sync. A full sync marks rows that fall outside the extracted date range as deleted, even if Algolia still retains them. Analytics extraction starts on the preceding day and log extraction reaches back at most 7 days if you don't set a start date. Workato recommends incremental sync for these objects if you need to retain older rows.

Incremental sync ​

Analytics objects and Logs read a bounded date range on each run. Your destination deduplicates re-read rows by primary key:

• Analytics objects: The date column is the cursor. Metrics for the current day are partial and are updated on later runs.
• Logs: The timestamp column is the cursor. Each run considers at most the preceding 7 days of logs. It resumes from the previous timestamp with a one-second overlap when no start date is set. A configured start date can narrow the window.

Delete tracking ​

Delete tracking depends on the object:

• Full sync only objects: Yes. Workato marks records you delete in Algolia as deleted in your destination on the next full sync.
• Analytics objects and Logs: No in incremental sync. Rows remain in your destination after they leave the extraction window. A full sync marks rows absent from its current extraction window as deleted. Refer to Full sync for more information.

Schema and data type handling ​

The following considerations apply to schema and data types when you sync data from Algolia:

Records schema discovery ​

Algolia index records don't have a fixed schema. Workato samples up to 1,000 records from the index when you add Records and creates a column for each top-level attribute it finds:

• objectID: Syncs as the primary key.
• Scalar attributes: Boolean, integer, decimal, and string attributes sync as matching column types. Attributes with mixed types sync as strings, except mixed integer and decimal values, which sync as decimals.
• raw_attributes: Contains every attribute that doesn't have its own column, such as an attribute missing from the sample, an attribute added to the index later, or an attribute you deselect during schema review.

Workato adds attributes it finds in later samples as new columns when you select Auto-sync new fields.

Nested fields ​

Nested objects and arrays, such as nested Records attributes, the conditions and consequence fields on Rules, and the attribute lists on Index Settings, sync as JSON objects. Workato stores them in your destination's native JSON type, such as VARIANT in Snowflake, and serializes them to strings only when the destination doesn't have a native JSON type.

Synthetic columns ​

Workato adds the following columns to destination tables for specific objects:

Sensitive data handling ​

Algolia objects can contain personally identifiable information (PII). The following objects commonly contain sensitive fields:

Search queries contain the text your users enter, which can include personal information.

Deselecting a Records attribute doesn't keep its value out of your destination, because Workato stores deselected attributes in the raw_attributes column. Deselect or hash raw_attributes to exclude those values.

Use the Hash option in field-level data protection during pipeline configuration to protect PII before it reaches your destination. Refer to the Configure the pipeline steps for more information.

Limitations ​

The following limitations apply when you use Algolia as a data pipeline source:

Logs retention ​

Algolia keeps API logs for 7 days and logs up to 1,000 API requests per server. Each run syncs at most the 1,000 most recent log entries from the preceding 7 days. Schedule frequent syncs for applications with high request volume to reduce the number of log entries that aren't captured.

Query Suggestions indexes aren't listed ​

Workato excludes indexes with query_suggestions in their name from object discovery, so their index-level and analytics objects aren't available. These indexes still appear as rows in the Indices table.

Deleted or renamed indexes ​

Objects for an index that you delete or rename in Algolia fail with an error that Algolia couldn't find the index. Remove these objects from your pipeline, and add the objects for the renamed index if needed.

Minimum sync frequency ​

The minimum supported sync interval is 15 minutes.