Prompt templates

Use model-level transformations to dynamically customize LLM request parameters based on request context such as headers, user identity, or other runtime information. Agentgateway uses CEL (Common Expression Language) expressions to evaluate and set LLM request fields at runtime.

About LLM transformations

Model-level transformations allow you to dynamically compute LLM request fields using CEL expressions that can reference incoming request headers, existing request fields, and other context. This is useful for enforcing per-user policies, customizing model behavior based on caller identity, and applying conditional request modifications without changing client code.

To learn more about CEL, see the following resources:

• CEL expression reference
• cel.dev tutorial

Before you begin

Conditionally set max tokens based on user identity

Use a CEL expression in the model-level transformation field to dynamically set max_tokens based on the caller’s identity from a request header. This example gives admin users a higher token limit than regular users.

cat <<'EOF' > config.yaml
# yaml-language-server: $schema=https://agentgateway.dev/schema/config

llm:
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      max_tokens: "request.headers['x-user-id'] == 'admin' ? 100 : 10"
EOF

Send a request as an admin user and verify the response uses the higher token limit.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-user-id: admin" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Tell me a story"}]
  }' | jq .

Send a request as a regular user and verify the response is capped at the lower token limit.

curl -s http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "x-user-id: alice" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Tell me a story"}]
  }' | jq .

In the responses, the admin user receives up to 100 completion tokens while the regular user is capped at 10.

Available CEL variables

You can use these variables in your CEL transformation expressions.

For a complete list of available variables and functions, see the CEL reference documentation.

Common transformation patterns

Cap token usage

Enforce a maximum token limit regardless of what the client requests.

llm:
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      max_tokens: "min(llmRequest.max_tokens, 1024)"

Set temperature based on headers

Allow callers to control creativity through a header while enforcing bounds.

llm:
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      temperature: "request.headers['x-creativity'] == 'high' ? 0.9 : 0.1"

Combine multiple transformations

Apply several field-level transformations in a single configuration.

llm:
  models:
  - name: "*"
    provider: openAI
    params:
      apiKey: "$OPENAI_API_KEY"
    transformation:
      max_tokens: "request.headers['x-user-tier'] == 'premium' ? 4096 : 256"
      temperature: "request.headers['x-user-tier'] == 'premium' ? 0.8 : 0.3"

Next steps

• Learn about CEL expressions for advanced expression logic.
• Explore transformations for more LLM request transformation examples.
• Set up authentication to use JWT claims in transformations.