AI Gateway ​

AI Gateway is a single point of control between your applications and LLM providers. Applications connect to one OpenAI-compatible endpoint, and the gateway handles provider authentication, routing, and policy enforcement so your applications don't have to.

AI Gateway

You can use AI Gateway to perform the following:

• Issue scoped access keys to teams: Generate access keys for each team, project, or application instead of sharing provider credentials. Revoke or rotate a key without touching downstream applications, and keep provider keys out of environment variables, configuration files, and CI pipelines.
• Apply usage limits: Define rate and token limits in a policy, then assign the policy to the access keys a team or application uses to cap how much they can consume.
• Restrict access by IP address: Allow or block requests from specific IP addresses or ranges in a policy.
• Swap providers without rewriting applications: Route Workato recipes, genies, copilots, and external applications through a single OpenAI-compatible endpoint. Switch models without code changes in the calling applications.

Key components ​

AI Gateway includes the following components:

• Providers: Connections to LLM services, such as Anthropic, OpenAI Compatible, Azure OpenAI, AWS Bedrock, and Google Gemini, along with the credentials AI Gateway uses to authenticate. Providers are stored in a project and can be reused in genies.
• LLM endpoints: The URLs your applications call. Each LLM endpoint contains its own routes, access keys, and policies.
• Routes: Pairings of one provider and one model, with their own timeout and retry limits. Applications reference a route by passing its name as the model field's value in prompt calls.
• Access keys: Credentials that applications present to AI Gateway. Each access key grants access to one or more routes and attributes usage to a team or application.
• Policies: Reusable rate limits, token limits, and IP access restrictions that you assign to access keys.

Refer to Configure AI Gateway to add providers, create LLM endpoints and routes, issue access keys, and define policies.

How AI Gateway works ​

Applications send requests to an LLM endpoint and authenticate with an access key. The access key determines which routes the application can use, and each route determines the provider and model that serves the request. When a policy is assigned to the access key, AI Gateway enforces the policy's rate and token limits, as well as IP access restrictions, before the request reaches the provider.

Supported applications ​

Any client that's compatible with the OpenAI SDK can route through AI Gateway without code changes because the gateway exposes an OpenAI-compatible API. This includes Workato recipes, genies, copilots, and external applications. AI Gateway supports Server-Sent Events (SSE) end to end to allow streaming responses to render incrementally rather than waiting for the full payload.