---
title: "LlamaIndex with EU-hosted models · OpenAILike setup"
description: "Connect LlamaIndex to inference.eu with the OpenAILike class and run RAG pipelines and agents on open-weight models processed in EU data centres."
canonical: "https://inference.eu/integrations/llamaindex/"
language: "en"
---

Frameworks · LlamaIndex

# Use LlamaIndex with open-source models hosted in the EU

LlamaIndex is a framework for building retrieval-augmented generation pipelines and agents over your own documents. Its \`OpenAILike\` class connects to any OpenAI-compatible API, so query engines, chat engines and agents can run on open-weight models at inference.eu, processed in European data centres.

[Contact sales](https://inference.eu/contact/)[Setup](#setup)

1.  01API format: OpenAI
2.  02EU data centres
3.  03Covered by one DPA

## Set it up in a few minutes

1.  1

    Create an API key in your inference.eu account and store it as `INFERENCE_EU_API_KEY`.

2.  2

    Install the integration with `pip install llama-index-llms-openai-like`.

3.  3

    Create `OpenAILike` with `api_base` set to the inference.eu base URL, your key and an inference.eu `model` id.

4.  4

    Set `is_chat_model=True`, `is_function_calling_model=True` and a `context_window` that matches the model.

5.  5

    Assign the model to `Settings.llm` or pass it to your query engine or agent, then run a test query.

python

```
import os
from llama_index.core import Settings
from llama_index.llms.openai_like import OpenAILike

llm = OpenAILike(
    model="glm-5.3",
    api_base="https://api.inference.eu/v1",
    api_key=os.environ["INFERENCE_EU_API_KEY"],
    is_chat_model=True,
    is_function_calling_model=True,
    context_window=131072,
)

Settings.llm = llm
print(llm.complete("Summarise GDPR Article 28 in two sentences."))
```

## What LlamaIndex is

LlamaIndex is an open-source data framework for language model applications. It loads and parses documents, splits them into nodes, builds indexes and retrieves the right context for each question. On top of that it offers query engines, chat engines, agents and event-driven workflows.

LlamaIndex separates the language model from the rest of the pipeline. Any component can use an LLM object, and `OpenAILike` is the class designed for third-party APIs that implement the OpenAI format. Set it once in `Settings.llm` and every index and engine in your application uses inference.eu.

## How the configuration works

`OpenAILike` takes `api_base` (the inference.eu base URL ending in `/v1`), `api_key` and `model`. Unlike the plain `OpenAI` class, it does not assume model capabilities, so three settings matter. `is_chat_model=True` sends requests to the chat endpoint; the default is the legacy completion endpoint.

`is_function_calling_model=True` lets agents use native tool calls; without it they fall back to text-based reasoning. `context_window` defaults to 3,900 tokens, far below what current models handle, so set it to the model's real window. LlamaIndex uses that value to decide how many retrieved chunks fit into each prompt.

## Which models to choose

For question answering over documents, `gpt-oss-120b` gives fast, grounded answers and keeps query latency low. For long documents or questions that need synthesis across many retrieved chunks, `glm-5.3` and `kimi-k3` handle large contexts and careful reasoning well.

For `FunctionAgent` and workflows that call several tools, choose a model with dependable tool calling, such as `glm-5.3`, `kimi-k3` or `qwen3-coder-480b-a35b-instruct`. You can also use different models in one application, for example a fast model for query rewriting and a stronger one for the final answer.

## Troubleshooting

A 401 error means the key is missing or wrong. A 404 usually means `api_base` has an extra path, or `is_chat_model` is still `False`, so LlamaIndex calls the legacy completion endpoint. Set it to `True`.

If an agent never calls tools and answers in plain text, set `is_function_calling_model=True`. If answers ignore most retrieved context or you see context length errors, check `context_window`: too low and LlamaIndex trims context, too high and requests exceed the model's limit. Do not use the plain `OpenAI` class with a custom model id; it rejects names it does not recognise.

## Keeping data in the EU

With `OpenAILike` pointed at inference.eu, the questions and retrieved document chunks LlamaIndex sends to the model are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros.

A full RAG pipeline also includes embeddings, a vector store and possibly a document parser or observability tool. Each of these processes your documents too. Choose EU-hosted or self-hosted options for them, so that your documents, embeddings and model requests all stay in Europe.

## LlamaIndex

Category

Frameworks

Maker

LlamaIndex

API format

OpenAI

Processing

EU data centres

Website

[www.llamaindex.ai](https://www.llamaindex.ai/)

Documentation

[developers.llamaindex.ai](https://developers.llamaindex.ai/python/framework-api-reference/llms/openai_like/)

## LlamaIndex: common questions

The questions buyers, developers and legal teams ask us most. Something missing? Ask us directly.

Install \`llama-index-llms-openai-like\` and create \`OpenAILike\` with \`api\_base\`, \`api\_key\` and \`model\`. Set \`is\_chat\_model=True\` and a realistic \`context\_window\`, then assign it to \`Settings.llm\` so every index, engine and agent in your application uses it.

The \`OpenAI\` class validates model names against OpenAI's own list and infers capabilities from them. \`OpenAILike\` accepts any model id and lets you declare capabilities yourself, which is what third-party APIs such as inference.eu need.

gpt-oss-120b gives fast, grounded answers for most document question answering. GLM-5.3 and Kimi K3 suit long documents and synthesis across many chunks, and also work well for agents that call tools in several steps.

Set the context window of the model you use, as listed for it at inference.eu. The default of 3,900 tokens is far too small and makes LlamaIndex discard retrieved context. Setting a value above the real limit causes context length errors.

\`OpenAILike\` sets \`is\_function\_calling\_model\` to \`False\` by default, so agents fall back to text-based reasoning. Set it to \`True\` and choose a model with dependable tool calling, such as GLM-5.3, Kimi K3 or Qwen3 Coder 480B.

Most often \`is\_chat\_model\` is left at \`False\`, so LlamaIndex calls the legacy \`/completions\` endpoint. Set it to \`True\`. Otherwise check that \`api\_base\` ends in \`/v1\` and that the model id exists at inference.eu.

Yes. Use \`stream\_complete\`, \`stream\_chat\` or a query engine created with \`streaming=True\`. Tokens arrive as they are generated, which keeps chat interfaces responsive for long answers over large documents and slow retrieval steps.

This page covers the language model. For embeddings, choose a model whose hosting meets your requirements, ideally in the EU or self-hosted, and set it in \`Settings.embed\_model\` so documents and queries stay in Europe along with your model requests.

LlamaIndex sends data only to the services you configure. Embedding providers, hosted vector stores, cloud document parsers and observability tools each receive data under their own terms. Review each one to keep your whole pipeline in Europe.

Yes. Workflows and \`FunctionAgent\` use the same LLM object, so once \`Settings.llm\` points to \`OpenAILike\` with inference.eu, every step runs on your chosen model. You can still pass a different model to specific steps if needed.

[Ask a question](https://inference.eu/contact/)

## Related

[All integrations](https://inference.eu/integrations/)

-   [LangChainConnect LangChain's ChatOpenAI to inference.eu with base\_url and run chains, agents and LangGraph workflows on open-weight models processed in the EU.Frameworks](https://inference.eu/integrations/langchain/)
-   [HaystackPoint Haystack's OpenAIChatGenerator at inference.eu with api\_base\_url and run RAG pipelines and agents on open-weight models processed in the EU.Frameworks](https://inference.eu/integrations/haystack/)
-   [Pydantic AIConnect Pydantic AI agents to inference.eu with OpenAICompatibleProvider and get typed, validated output from open-weight models processed in the EU.Frameworks](https://inference.eu/integrations/pydantic-ai/)

## Move your AI workloads to EU infrastructure

Tell us which models you use and roughly how much traffic you have. We reply within one working day with pricing and next steps.

-   Reply within one working day
-   Pricing for your volume
-   DPA ready to sign