Use LiteLLM with open-source models hosted in the EU

LiteLLM comes in two parts. The Python SDK calls over a hundred providers through one completion() function. The proxy server, often called the LiteLLM AI gateway, exposes an OpenAI-compatible API to your teams and adds virtual keys, spend tracking, rate limits, fallbacks, guardrails and logging in front of the providers you configure.

Companies often run the proxy as a central gateway so that coding agents, chat interfaces and internal apps share one endpoint. Adding inference.eu as a provider in that gateway makes open-weight EU models available to all of them without changing each client.

In the proxy's config.yaml, each model_list entry maps a public model_name to a provider route. The openai/ prefix in litellm_params.model tells LiteLLM to call an OpenAI-compatible Chat Completions endpoint; the rest is the inference.eu model id. api_base is the inference.eu base URL ending in /v1, and os.environ/INFERENCE_EU_API_KEY reads the key from the environment.

In the Python SDK, the same route is litellm.completion(model="openai/gpt-oss-120b", api_base=..., api_key=...). Clients of the proxy only see model_name, so you can change the underlying inference.eu model without touching their configuration.

Expose the models your teams need under clear names. qwen3-coder-480b-a35b-instruct suits coding agents and IDE integrations routed through the gateway. glm-5.3 and kimi-k3 are strong general models for agents and long documents, and gpt-oss-120b is a fast default for chat, summaries and high-volume internal tools.

LiteLLM can group several entries under one model_name for load balancing, or define fallbacks between names. Keep fallbacks within inference.eu models, for example from kimi-k3 to glm-5.3, so that a retry never sends data to a provider outside the EU.

A 404 Not Found from the route usually means api_base is missing the /v1 suffix or has extra path segments; LiteLLM adds /chat/completions itself. A 401 means the proxy process has no INFERENCE_EU_API_KEY, which often happens when it runs in Docker without the variable passed in.

"LLM Provider NOT provided" means the openai/ prefix is missing from litellm_params.model. If tool calls or streaming usage look wrong, update LiteLLM, since compatibility fixes ship often. Check the model id after the prefix matches inference.eu exactly, and test the route with the proxy's /health endpoint.

Requests routed to inference.eu are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros.

The gateway itself sees every prompt, so where it runs matters. Host the proxy in the EU and review its logging callbacks: integrations that send traces or prompts to observability services store that data under their own terms. Remove providers outside the EU from the model list or restrict them with team permissions if your policy requires all traffic to stay in Europe.