LiteLLM is an open-source AI gateway that provides a unified, OpenAI-compatible API for over 100 large language model (LLM) providers, removing the need to integrate with each provider's SDK and authentication scheme. In production, teams use LiteLLM to issue virtual keys with budget limits, track token usage and spend across providers, and configure centralized access control and routing. This guide walks through deploying LiteLLM on a Linux server using Docker Compose with PostgreSQL for persistent storage, Prometheus for metrics collection, and Traefik as the reverse proxy for HTTPS access. By the end, you'll have a fully functional, OpenAI-compatible AI gateway accessible over a custom domain, with virtual key management, spend tracking, and provider routing configured.
Before you begin, you need a Linux-based server with at least 4 CPU cores and 8 GB of RAM as a non-root user with sudo privileges, Docker and Docker Compose installed, a DNS A record (such as litellm.example.com) pointing to your server's IP address, and an API key from at least one supported LLM provider. Set Up the Directory Structure, Configuration, and Environment Variables
LiteLLM requires a configuration file to define model providers and routing rules, a Prometheus configuration file for metrics scraping, and environment variables for secrets and database credentials. Create the project directory with subdirectories for persistent data:
letsencrypt stores SSL/TLS certificates, postgres persists PostgreSQL database files, and prometheus persists Prometheus metrics data. Navigate to the project directory: Set matching ownership on the Prometheus host directory (Prometheus runs as UID 65534 inside the container): Generate a master key and salt key for LiteLLM. Run this command twice to produce two separate values:
Save both values for the LITELLMMASTERKEY and LITELLMSALTKEY fields in the next step. Create the .env file to store credentials and secrets: Add the following values:
Replace the placeholders with your own values: litellm.example.com is the domain pointing to your server's IP address; admin@example.com is the email address for Let's Encrypt notifications; sk-YOURMASTERKEY is the admin key used to authenticate with the LiteLLM API (replace YOURMASTERKEY with the first generated value and keep the sk- prefix); sk-YOURSALTKEY encrypts provider credentials stored in PostgreSQL and cannot be changed after the first model is added; YOURLLMPROVIDERAPIKEY is the API key for the provider configured in config.yaml; STRONGDATABASEPASSWORD is the PostgreSQL password used by both the database container and the connection string; admin/YOURUIPASSWORD are the LiteLLM dashboard credentials. Create the LiteLLM configuration file: Add the following contents:
Replace my-model with a name of your choice to identify this model within LiteLLM, and provider/model with the LiteLLM provider prefix followed by the model identifier (for example, anthropic/claude-haiku-4-5). The os.environ/ prefix tells LiteLLM to read the value from an environment variable at runtime rather than hardcoding it. The litellmsettings block enables Prometheus metrics via the callbacks field, and requireauthformetricsendpoint: true restricts the /metrics endpoint to authenticated requests, so only Prometheus (sending the master key as a Bearer token) can scrape metrics successfully. The generalsettings block defines the master key LiteLLM uses to authenticate admin API requests and virtual key management operations. Create the Prometheus configuration file: Add the following contents:
Replace sk-YOURMASTERKEY with the value of LITELLMMASTERKEY from your .env file. Prometheus uses this token to authenticate its scrape requests against the protected /metrics endpoint. Deploy with Docker Compose
The deployment stack runs LiteLLM behind Traefik, which handles TLS termination and automatic certificate provisioning through Let's Encrypt. PostgreSQL provides persistent storage for virtual keys, spend data, and usage logs. Prometheus collects gateway metrics by scraping the LiteLLM /metrics endpoint every 15 seconds over the internal Docker network. Add your user account to the Docker group: Apply the new group membership: Create the Docker Compose manifest file: Add the following contents:
This configuration deploys four services behind a single HTTPS endpoint: traefik acts as the reverse proxy and TLS terminator, redirecting HTTP to HTTPS via Let's Encrypt; litellm runs the proxy image pinned to a specific release tag, mounts config.yaml, and waits for PostgreSQL to become healthy before starting; db runs PostgreSQL 16 as the persistent backend for virtual keys, spend data, and usage logs; prometheus scrapes LiteLLM metrics every 15 seconds with a 15-day retention window. The named volumes for db and prometheus ensure their data survives container removal or recreation.
LiteLLM Docker images are signed with Cosign. Verifying the image signature before deployment confirms the image has not been tampered with since it was published by the LiteLLM team. Download and install the Cosign binary: Move the binary into your system path: Make the binary executable: Verify the installation: Verify the LiteLLM image signature using the pinned public key. This checks the same release tag deployed in the Docker Compose file. LiteLLM publishes signatures for the ghcr.io registry specifically, so this confirms the v1.89.1 release itself, not the docker.litellm.ai mirror byte-for-byte.
A successful verification outputs a JSON payload confirming the image was signed with the LiteLLM public key.
Verify that all four containers show an Up status, with litellm and db marked (healthy) and traefik listing ports 80 and 443 under PORTS. Access and Configure LiteLLM
LiteLLM exposes a web-based admin dashboard at the /ui path of your configured domain, with visibility into model configuration, virtual keys, usage, and spend tracking. Open your web browser and navigate to https://litellm.example.com/ui, replacing litellm.example.com with your configured domain. Log in using the credentials from your .env file — the value of UIUSERNAME in the Username field and UIPASSWORD in the Password field, then click Login. Verify that the dashboard loads and displays the main navigation panels. Click Models + Endpoints in the left sidebar to verify the configured model providers. The model defined in config.yaml appears in the list with its alias and underlying provider model. The left sidebar also provides access to Virtual Keys for managing scoped access credentials, Usage for per-model and per-key request tracking, and Logs for spend and cost breakdowns by model, key, and team. Test the Gateway and Create a Virtual Key
LiteLLM exposes an OpenAI-compatible API, so any application built for the OpenAI SDK works with LiteLLM by pointing baseurl at the gateway. Virtual keys provide scoped, credential-isolated access without exposing the master key. Install the Python virtual environment package: Create a Python virtual environment: Activate the virtual environment: Install the OpenAI Python SDK: Export your master key as an environment variable, replacing sk-YOURMASTERKEY with the LITELLMMASTERKEY value from your .env file: Export your gateway domain as an environment variable, replacing litellm.example.com with your configured domain: Create the test script: Add the following contents, replacing my-model with the model name you set in config.yaml: Run the script:
The script returns a response from the configured LLM provider, confirming the gateway is routing requests correctly. Create a virtual key using the LiteLLM API, replacing my-model with the model name you set in config.yaml:
The command prints the generated virtual key, similar to sk-M9M0aUDLi7AuhXtOAw6uw. Each key can have its own model access list, budget limit, and rate limit. Export the virtual key returned in the previous step, replacing sk-your-virtual-key with the full key string printed: Send a request using the virtual key, replacing my-model with the model name you set in config.yaml:
The gateway authenticates the virtual key, routes the request to the configured provider, and returns the model response. Return to the dashboard at https://litellm.example.com/ui and click Logs in the left sidebar. Each entry shows the model alias, key alias, token count, and estimated cost for requests sent through both the master key and the virtual key.