Configure AI Gateway
Set up AI Gateway by connecting your LLM providers, creating an LLM endpoint for your applications to call, and issuing access keys that control who can use it.
Configuration order
AI Gateway configuration builds on four objects. Create them in the following order:
- Provider: A connection to an LLM service, such as Anthropic or Azure OpenAI, along with the credentials AI Gateway uses to authenticate.
- LLM endpoint: The URL your applications call. Each LLM endpoint contains its own routes, access keys, and policies.
- Route: A pairing of one provider and one model, with its own timeout limit.
- Access key: The credential an application presents to AI Gateway. Each access key grants access to one or more routes.
Policies are optional. Create a policy when you need to cap the request rate or token usage of the access keys assigned to it, or restrict access by IP address.
The first time you open AI Gateway, the gateway guides you through this sequence. You can also complete each step individually using the procedures in this guide.
Add a provider
A provider stores the credentials AI Gateway uses to authenticate with an LLM service. Providers are stored in a project, and you can reuse the same provider connection in genies.
AI Gateway supports the following LLM providers: Anthropic, OpenAI Compatible, Azure OpenAI, AWS Bedrock, and Google Gemini. The fields you configure depend on the provider you select.
Complete the following steps to add a provider:
Go to AI Gateway > Providers.
Click Add provider.
Add provider
Enter a descriptive name in the Connection name field.
Create provider
Use the Location drop-down menu to select the project or folder where you plan to store this provider.
Use the Connection type drop-down menu to select Cloud or an on-prem group. Select an on-prem group when the provider's endpoint isn't reachable from the public internet.
ON-PREM GROUPS
If you select an on-prem group in Connection type, the API URL or Endpoint URL must be reachable from the on-prem agent's network. Private DNS names and IP addresses are allowed.
Use the LLM Provider drop-down menu to select the provider this connection uses. The remaining fields depend on the provider you select:
Click Create to complete the setup.
PROVIDER UNREACHABLE
AI Gateway returns Error 503 and the message Could not reach provider. The service may be temporarily down or overloaded. Please try again shortly. when it can't reach the provider. Your entries are preserved. Click Create again to retry.
The provider displays in the Providers list with a Connected status and the date it was added.
Edit a provider
Complete the following steps to edit a provider:
Go to AI Gateway > Providers.
Click the actions menu for the provider you plan to edit, then click Edit. This opens the Edit provider page.
Update the fields for the provider.
Click Save.
Delete a provider
Complete the following steps to delete a provider:
Go to AI Gateway > Providers.
Click the actions menu for the provider you plan to delete, then click Delete.
Review the impact summary. The Delete provider dialog lists the number of access keys affected and confirms that deleting the provider removes it from all routes that use it, that keys invoking those routes may return errors, and that access is revoked for all users with keys.
Select I understand that this action cannot be undone.
Click Delete.
Create an LLM endpoint
An LLM endpoint is the URL your applications send requests to. Each LLM endpoint contains its own routes, access keys, and policies, which lets you separate configuration by team, application, or environment.
PREREQUISITES
You must add at least one provider before you create an LLM endpoint.
Complete the following steps to create an LLM endpoint:
Go to AI Gateway > Endpoints.
Click + Create LLM endpoint.
Create LLM endpoint
Enter a name in the LLM endpoint name field.
Create LLM endpoint
Use the Location drop-down menu to select the project or folder where you plan to store this LLM endpoint.
Click Create. The Add route modal displays.
Enter a name in the Route name field.
Add route
AI Gateway generates the Route ID from the route name.
Use the Provider drop-down menu to select the provider to use for this route. Manage providers in the Providers tab.
Use the Model drop-down menu to select which model from the selected provider to use for this route.
Optional. Expand Limits and configure the following field:
Enter the maximum time allowed for a single attempt before it fails in the Attempt timeout (in seconds) field. This value must be less than or equal to the request timeout.
Click Create.
The route displays in the Routes list with its route ID, provider, models, and creation date.
LLM endpoint details
The details panel on an LLM endpoint page displays the following:
- LLM endpoint name: The name of the LLM endpoint.
- LLM endpoint URL: The base URL your applications call. Click Copy URL to copy it.
- Location: The project or folder that stores the LLM endpoint.
- Created: The date and time the LLM endpoint was created.
Create a route
A route pairs one provider with one model and defines the timeout behavior for requests that use it.
PREREQUISITES
You must add at least one provider before you create a route.
Complete the following steps to create a route:
Go to your LLM endpoint and click the Routes tab.
Click + Add route.
Add route
Enter a name in the Route name field.
Add route
AI Gateway generates the Route ID from the route name.
Use the Provider drop-down menu to select the provider to use for this route. Manage providers in the Providers tab.
Use the Model drop-down menu to select which model from the selected provider to use for this route.
Optional. Expand Limits and configure the following field:
Enter the maximum time allowed for a single attempt before it fails in the Attempt timeout (in seconds) field. This value must be less than or equal to the request timeout.
Click Create.
The route displays in the Routes list with its route ID, provider, models, and creation date.
Edit a route
Complete the following steps to edit a route:
Go to your LLM endpoint and click the Routes tab.
Click the actions menu for the route you plan to edit, then click Edit.
Edit route
Update the route fields, then click Save.
Delete a route
KEYS STOP FUNCTIONING
Any access key that uses a deleted route no longer functions and must be recreated.
Complete the following steps to delete a route:
Go to your LLM endpoint and click the Routes tab.
Click the actions menu for the route you plan to delete, then click Delete.
Delete route
Click Delete to confirm.
Create an access key
An access key is the credential an application presents to AI Gateway. Each key grants access to one or more routes, which determine the provider and model the key can use.
PREREQUISITES
You must create at least one route before you create an access key.
Complete the following steps to create an access key:
Go to your LLM endpoint and click the Keys tab.
Click Add key.
Add key
Enter a descriptive name in the Name field.
Create key
Use the Route(s) drop-down menu to select the routes this key can use, then click OK. Click Select all to select every route in the LLM endpoint. Manage routes in the Routes tab.
Optional. Use the Policy drop-down menu to select a policy for this key. Click Create policy to create a policy without leaving the dialog, or manage policies in the Policies tab. Refer to Create a policy for the field descriptions.
SHARED USAGE
All users who share an access key contribute to the same rate and token limits. Issue separate access keys for teams or applications that need independent limits.
Click Create.
Copy the generated key and store it in a safe place. Click Copy, then click Close.
THE KEY IS DISPLAYED ONCE
You can't view the generated key again after you close the dialog. Refresh the key to generate a new value if you lose it.
The access key displays in the Keys list with its routes, policy, creation date, and a masked key value. You can filter the list by route and by policy.
Edit an access key
Complete the following steps to edit an access key:
Go to your LLM endpoint and click the Keys tab.
Click the actions menu for the key you plan to edit, then click Edit.
Edit key
Update the Name, Route(s), or Policy fields, then click Save.
Refresh an access key
Refresh an access key to rotate its value. Refreshing generates a new token and revokes access for all current users of the key.
CLIENTS MUST BE UPDATED
Applications that use the previous token lose access when you refresh a key. You must update each client with the refreshed token to regain access.
Complete the following steps to refresh an access key:
Go to your LLM endpoint and click the Keys tab.
Click the actions menu for the key you plan to refresh, then click Refresh.
Refresh key
Click Refresh token.
Copy the refreshed key and store it in a safe place. Click Copy, then click Close. You can't view the refreshed key again after you close the dialog.
Delete an access key
ACCESS IS REVOKED IMMEDIATELY
Deleting an access key revokes access for all current users of that key. You can't undo this action.
Complete the following steps to delete an access key:
Go to your LLM endpoint and click the Keys tab.
Click the actions menu for the key you plan to delete, then click Delete.
Click Delete to confirm.
Delete key
Create a policy
A policy defines the rate limits, token limits, and IP access restrictions that apply to the access keys assigned to it. Policies are scoped to an LLM endpoint and you can reuse them across the access keys in that LLM endpoint.
Complete the following steps to create a policy:
Go to your LLM endpoint and click the Policies tab.
Click Add policy.
Add policy
Enter a name in the Name field.
Create new policy
Optional. Expand Rate limit to specify the throttle limit per key, then configure the following fields:
Use the Time interval drop-down menu to select the interval the limit applies to.
Enter the number of requests allowed per key in the selected time interval in the Number of requests field.
Optional. Expand Token limit to define each key's token usage quota, then configure the following fields:
Use the Time interval drop-down menu to select the interval the limit applies to.
Enter the number of tokens allowed per key in the selected time interval in the Number of tokens field.
Optional. Expand IP access control to restrict access by IP address, then configure the following fields:
Optional. Enter one or more IP addresses in the Allowed IPs field to allow requests only from those addresses. Separate multiple IP addresses with commas, or define a netmask, for example 106.226.96.0/20.
Optional. Enter one or more IP addresses in the Blocked IPs field to block requests from those addresses. Separate multiple IP addresses with commas, or define a netmask, for example 106.226.96.0/20.
Click Create.
Validation messages
AI Gateway returns the following validation messages:
| Field | Message | Condition |
|---|---|---|
| Connection name, Name | Name cannot exceed 100 characters. | The name is longer than 100 characters. |
| API URL | Please enter a valid URL. | The value isn't a valid URL. |
| Attempt timeout (in seconds) | Value must be between 1 and 360 seconds. | The value is outside the supported range. |
Last updated: