AI Gateway ​

AI Gateway is a single point of control between your applications and LLM providers. Applications connect through one OpenAI-compatible API, and the gateway handles provider authentication, routing, and policy enforcement so your applications don't have to.

The following diagram shows the path a request takes through AI Gateway:

You can use AI Gateway to perform the following:

• Issue scoped access keys to teams: Generate access keys for each team, project, or application instead of sharing provider credentials. Revoke or rotate a key without touching downstream applications, and keep provider keys out of environment variables, configuration files, and CI pipelines.
• Swap providers without rewriting applications: Route Workato recipes, genies, copilots, and external applications through a single OpenAI-compatible API. Switch models without code changes in the calling applications.
• Apply usage limits: Define rate limits and token limits in a policy, then assign the policy to the access keys a team or application uses to cap how much they can consume.
• Restrict access by IP address: Allow or block requests from specific IP addresses or ranges in a policy.

Key components ​

AI Gateway includes the following components:

• Providers: Connections to LLM services, such as Anthropic, OpenAI Compatible, Azure OpenAI, and AWS Bedrock, along with the credentials AI Gateway uses to authenticate. Providers are stored in a project and can be reused in genies.
• Routes: The targets that serve requests. A route resolves to a provider and model, either directly or through rules that match on tags, keys, parameters, or request complexity. Applications reference a route by its route ID.
• Access keys: Credentials that applications present to AI Gateway. Each access key grants access to one or more routes and attributes usage to a team or application.
• Policies: Reusable rate limits, token limits, and IP access restrictions that you assign to routes and access keys.
• Dashboard: Workspace-level visibility into request volume, token usage, and success and failure rates.

Refer to Configure AI Gateway to add providers, create routes, issue access keys, and define policies, the Route builder to define the rules a rule-based or intelligent route uses, or the AI Gateway dashboard to monitor traffic and token usage.

How AI Gateway works ​

Applications send a request to AI Gateway and authenticate with an access key. The access key determines which routes the application can use, and the route it names determines the provider and model that serves the request. When a policy applies, AI Gateway enforces its rate limits, token limits, and IP access restrictions before the request reaches the provider.

Supported applications ​

Any client that's compatible with the OpenAI SDK can route through AI Gateway without code changes because the gateway exposes an OpenAI-compatible API. This includes Workato recipes, genies, copilots, and external applications. AI Gateway supports Server-Sent Events (SSE) end to end to allow streaming responses to render incrementally rather than waiting for the full payload.