---
title: "LiteLLM with EU-hosted models · Proxy and SDK setup"
description: "Add inference.eu to the LiteLLM proxy or Python SDK with the openai/ prefix and route your team's requests to open-weight models processed in the EU."
canonical: "https://inference.eu/integrations/litellm/"
language: "en"
---

Gateways · LiteLLM

# Use LiteLLM with open-source models hosted in the EU

LiteLLM is an open-source gateway and Python SDK that puts many model providers behind one OpenAI-style API, with virtual keys, budgets and logging. Add inference.eu to its model list and every application behind the gateway can use open-weight models processed in European data centres.

[Contact sales](https://inference.eu/contact/)[Setup](#setup)

1.  01API format: OpenAI
2.  02EU data centres
3.  03Covered by one DPA

## Set it up in a few minutes

1.  1

    Create an API key in your inference.eu account and set `INFERENCE_EU_API_KEY` in the environment of the LiteLLM proxy.

2.  2

    Add one `model_list` entry per model, with `model` set to `openai/` plus the inference.eu model id.

3.  3

    Set `api_base` to the inference.eu base URL and `api_key` to `os.environ/INFERENCE_EU_API_KEY`.

4.  4

    Start the proxy with `litellm --config config.yaml`.

5.  5

    Call the proxy with the public `model_name` and a LiteLLM key to confirm the route works.

config.yaml

```
model_list:
  - model_name: qwen3-coder
    litellm_params:
      model: openai/qwen3-coder-480b-a35b-instruct
      api_base: https://api.inference.eu/v1
      api_key: os.environ/INFERENCE_EU_API_KEY
  - model_name: glm-5.3
    litellm_params:
      model: openai/glm-5.3
      api_base: https://api.inference.eu/v1
      api_key: os.environ/INFERENCE_EU_API_KEY
  - model_name: gpt-oss-120b
    litellm_params:
      model: openai/gpt-oss-120b
      api_base: https://api.inference.eu/v1
      api_key: os.environ/INFERENCE_EU_API_KEY

# Start with: litellm --config config.yaml
```

## What LiteLLM is

LiteLLM comes in two parts. The Python SDK calls over a hundred providers through one `completion()` function. The proxy server, often called the LiteLLM AI gateway, exposes an OpenAI-compatible API to your teams and adds virtual keys, spend tracking, rate limits, fallbacks, guardrails and logging in front of the providers you configure.

Companies often run the proxy as a central gateway so that coding agents, chat interfaces and internal apps share one endpoint. Adding inference.eu as a provider in that gateway makes open-weight EU models available to all of them without changing each client.

## How the configuration works

In the proxy's `config.yaml`, each `model_list` entry maps a public `model_name` to a provider route. The `openai/` prefix in `litellm_params.model` tells LiteLLM to call an OpenAI-compatible Chat Completions endpoint; the rest is the inference.eu model id. `api_base` is the inference.eu base URL ending in `/v1`, and `os.environ/INFERENCE_EU_API_KEY` reads the key from the environment.

In the Python SDK, the same route is `litellm.completion(model="openai/gpt-oss-120b", api_base=..., api_key=...)`. Clients of the proxy only see `model_name`, so you can change the underlying inference.eu model without touching their configuration.

## Which models to choose

Expose the models your teams need under clear names. `qwen3-coder-480b-a35b-instruct` suits coding agents and IDE integrations routed through the gateway. `glm-5.3` and `kimi-k3` are strong general models for agents and long documents, and `gpt-oss-120b` is a fast default for chat, summaries and high-volume internal tools.

LiteLLM can group several entries under one `model_name` for load balancing, or define fallbacks between names. Keep fallbacks within inference.eu models, for example from `kimi-k3` to `glm-5.3`, so that a retry never sends data to a provider outside the EU.

## Troubleshooting

A 404 Not Found from the route usually means `api_base` is missing the `/v1` suffix or has extra path segments; LiteLLM adds `/chat/completions` itself. A 401 means the proxy process has no `INFERENCE_EU_API_KEY`, which often happens when it runs in Docker without the variable passed in.

"LLM Provider NOT provided" means the `openai/` prefix is missing from `litellm_params.model`. If tool calls or streaming usage look wrong, update LiteLLM, since compatibility fixes ship often. Check the model id after the prefix matches inference.eu exactly, and test the route with the proxy's `/health` endpoint.

## Keeping data in the EU

Requests routed to inference.eu are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros.

The gateway itself sees every prompt, so where it runs matters. Host the proxy in the EU and review its logging callbacks: integrations that send traces or prompts to observability services store that data under their own terms. Remove providers outside the EU from the model list or restrict them with team permissions if your policy requires all traffic to stay in Europe.

## LiteLLM

Category

Gateways

Maker

BerriAI

API format

OpenAI

Processing

EU data centres

Website

[www.litellm.ai](https://www.litellm.ai/)

Documentation

[docs.litellm.ai](https://docs.litellm.ai/docs/providers/openai_compatible)

## LiteLLM: common questions

The questions buyers, developers and legal teams ask us most. Something missing? Ask us directly.

Add a \`model\_list\` entry with a \`model\_name\`, and under \`litellm\_params\` set \`model\` to \`openai/\` plus the provider's model id, \`api\_base\` to its base URL and \`api\_key\` to the key or \`os.environ/VARIABLE\`. Restart the proxy.

The prefix tells LiteLLM which provider adapter to use. \`openai/\` selects the OpenAI-compatible Chat Completions route. LiteLLM strips it before sending, so inference.eu receives the plain model id, such as \`glm-5.3\`.

Most often \`api\_base\` is missing \`/v1\` or includes \`/chat/completions\`. Set it to the inference.eu base URL ending in \`/v1\`; LiteLLM adds the endpoint path. Then check that the model id after \`openai/\` is correct.

Yes. The LiteLLM proxy accepts Anthropic-format requests on its \`/v1/messages\` route and forwards them to the configured model. You can also point Claude Code straight at inference.eu, which accepts the Anthropic format natively, and skip the gateway.

A common set is Qwen3 Coder 480B for coding tools, GLM-5.3 or Kimi K3 for agents and long documents, and gpt-oss-120b for fast chat and internal tools. Give each a clear \`model\_name\` that your teams will recognise.

Yes. Define fallbacks in the router settings, for example from \`kimi-k3\` to \`glm-5.3\`. Keeping fallbacks within inference.eu means a retry never sends data to a provider outside the EU, even during an outage.

Write \`api\_key: os.environ/INFERENCE\_EU\_API\_KEY\` and set the variable in the proxy's environment, for example in your Docker or Kubernetes secret. LiteLLM resolves the value at start-up and never writes it back to the file.

Yes. Call \`litellm.completion(model="openai/<model id>", api\_base=..., api\_key=..., messages=\[...\])\`. This is useful in scripts and notebooks, while the proxy suits teams that need shared keys, budgets, rate limits and central logging for many applications.

LiteLLM tracks token usage per key, team and model. Its cost figures depend on prices you configure in the model info; billing itself happens in euros in your inference.eu account, which is the authoritative record.

The proxy sees every prompt and response it forwards. Host it in the EU, and check logging callbacks and observability integrations, since they may store prompts elsewhere. Model processing at inference.eu takes place in EU data centres.

[Ask a question](https://inference.eu/contact/)

## Related

[All integrations](https://inference.eu/integrations/)

-   [OpenAI SDK (Python and JavaScript)Point the official OpenAI Python or JavaScript SDK at inference.eu with base\_url and an API key, and call open-weight models processed in the EU.SDKs](https://inference.eu/integrations/openai-sdk/)
-   [Open WebUIConnect Open WebUI to inference.eu as an OpenAI-compatible connection and give your team a self-hosted chat on open-weight models processed in the EU.Chat apps](https://inference.eu/integrations/open-webui/)
-   [LibreChatAdd inference.eu as a custom endpoint in librechat.yaml and give your organisation a self-hosted chat with agents on open-weight models processed in the EU.Chat apps](https://inference.eu/integrations/librechat/)
-   [Claude CodePoint Claude Code at inference.eu with ANTHROPIC\_BASE\_URL and run it on open-weight models processed in EU data centres. No Anthropic account needed.Coding agents](https://inference.eu/integrations/claude-code/)

## Move your AI workloads to EU infrastructure

Tell us which models you use and roughly how much traffic you have. We reply within one working day with pricing and next steps.

-   Reply within one working day
-   Pricing for your volume
-   DPA ready to sign