Documentation

OpenAI Codex

OPENAI_BASE_URL={your-endpoint} codex

Claude Code

ANTHROPIC_BASE_URL={your-endpoint} claude

Any OpenAI-compatible tool

Set the base URL environment variable to your gateway endpoint. Works with Cursor, Continue, Aider, and any tool that reads OPENAI_BASE_URL.

curl (OpenAI)

curl {your-endpoint}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {your-api-key}" \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

curl (Anthropic)

curl {your-endpoint}/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {your-api-key}" \
  -d '{
    "model": "claude-sonnet-4-20250514",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello"}]
  }'

OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="{your-endpoint}",
    api_key="{your-api-key}",
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

Anthropic SDK

import anthropic

client = anthropic.Anthropic(
    base_url="{your-endpoint}",
    api_key="{your-api-key}",
)

message = client.messages.create(
    model="claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)

Failover Router

Sends every request to the primary upstream. If the primary returns an error (5xx) or times out, the gateway automatically retries with the next upstream in priority order until one succeeds. If all upstreams fail, the gateway returns 503.

Load Balancer (Weighted Router)

Distributes requests across upstreams using weighted random selection. Each upstream has a weight value (1–100); the probability of selection equals the upstream's weight divided by the total weight of all upstreams.

Smart Router

An LLM evaluates each incoming request and selects the optimal upstream based on quality, speed, and cost scores you define per model. The guardrail service analyzes the request content and picks the best match considering the configured scores.

Combined (Multi-Layer Routing)

Chain routers together by selecting an existing router as an upstream of another router. For example, a smart router can pick between two failover routers, each with its own set of providers. This creates a multi-layer routing topology where each tier applies its own strategy independently.

Request Phase (Anonymization)

The guardrail service returns an anonymization map — a list of detected text and their replacement tokens. The gateway replaces every occurrence in the request body before forwarding.

# Original request body
"Contact me at john@acme.com, my key is sk-abc123"

# After anonymization (forwarded to upstream)
"Contact me at pii_8f3a2b, my key is cred_e7c4d1"

# Anonymization map (stored internally)
john@acme.com  →  pii_8f3a2b
sk-abc123      →  cred_e7c4d1

Response Phase (Rehydration)

When the upstream responds, the gateway reverses the anonymization map — replacing tokens back with the original text before returning the response to the caller.

# Upstream response (contains tokens)
"Sure, I'll contact pii_8f3a2b regarding..."

# After rehydration (returned to caller)
"Sure, I'll contact john@acme.com regarding..."

Default Policy

Auto-created per region. Blocks prompt injection, malicious content, and sensitive data. Anonymizes personal information and credentials without blocking.

Category               Severity   Blocking   Anonymization
─────────────────────  ─────────  ─────────  ─────────────
Personal Information   low        off        on
Credentials            low        off        on
Prompt Injection       high       on         off
Malicious Content      critical   on         off
Sensitive Data         medium     on         off

Custom Policies

Create custom policies to adjust the behavior per category. For example, a "Strict" policy could block all categories, while a "Permissive" policy could anonymize everything without blocking.

# Example: Strict — block everything
Personal Information   high       on         off
Credentials            critical   on         off
Prompt Injection       critical   on         off
Malicious Content      critical   on         off
Sensitive Data         high       on         off

# Example: Permissive — anonymize, never block
Personal Information   low        off        on
Credentials            low        off        on
Prompt Injection       medium     off        on
Malicious Content      medium     off        on
Sensitive Data         low        off        on

Private (SQS)

The region polls an AWS SQS FIFO queue for commands. No inbound network access required — the region only makes outbound connections. Ideal for regions behind firewalls or NAT.

Public (HTTP)

The console sends commands directly to the region's endpoint via HTTPS. The region agent registers itself as an API gateway service with key-auth and rate limiting. Requires the region to be reachable from the internet.

Heartbeat

Both modes send periodic heartbeats (default: every hour) with stack health, service versions, and uptime. The console uses heartbeats to display region status and trigger automatic credential rotation.

Local Ollama

Run Ollama on the same machine as the region stack. The guardrail service reaches it via host.docker.internal.

# Install and start Ollama
ollama serve

# Pull a model
ollama pull gpt-oss:120b

# When creating the region, configure:
#   Provider: Ollama
#   Endpoint: http://host.docker.internal:11434
#   Model:    gpt-oss:120b

OpenAI-compatible endpoint

Use any provider that exposes an OpenAI-compatible chat completions API. Requires an API token.

# When creating the region, configure:
#   Provider: OpenAI-compatible
#   Endpoint: https://api.openai.com/v1
#   Model:    gpt-4o-mini
#   Token:    sk-...