Use LlamaIndex with open-source models hosted in the EU

LlamaIndex is an open-source data framework for language model applications. It loads and parses documents, splits them into nodes, builds indexes and retrieves the right context for each question. On top of that it offers query engines, chat engines, agents and event-driven workflows.

LlamaIndex separates the language model from the rest of the pipeline. Any component can use an LLM object, and OpenAILike is the class designed for third-party APIs that implement the OpenAI format. Set it once in Settings.llm and every index and engine in your application uses inference.eu.

OpenAILike takes api_base (the inference.eu base URL ending in /v1), api_key and model. Unlike the plain OpenAI class, it does not assume model capabilities, so three settings matter. is_chat_model=True sends requests to the chat endpoint; the default is the legacy completion endpoint.

is_function_calling_model=True lets agents use native tool calls; without it they fall back to text-based reasoning. context_window defaults to 3,900 tokens, far below what current models handle, so set it to the model's real window. LlamaIndex uses that value to decide how many retrieved chunks fit into each prompt.

For question answering over documents, gpt-oss-120b gives fast, grounded answers and keeps query latency low. For long documents or questions that need synthesis across many retrieved chunks, glm-5.3 and kimi-k3 handle large contexts and careful reasoning well.

For FunctionAgent and workflows that call several tools, choose a model with dependable tool calling, such as glm-5.3, kimi-k3 or qwen3-coder-480b-a35b-instruct. You can also use different models in one application, for example a fast model for query rewriting and a stronger one for the final answer.

A 401 error means the key is missing or wrong. A 404 usually means api_base has an extra path, or is_chat_model is still False, so LlamaIndex calls the legacy completion endpoint. Set it to True.

If an agent never calls tools and answers in plain text, set is_function_calling_model=True. If answers ignore most retrieved context or you see context length errors, check context_window: too low and LlamaIndex trims context, too high and requests exceed the model's limit. Do not use the plain OpenAI class with a custom model id; it rejects names it does not recognise.

With OpenAILike pointed at inference.eu, the questions and retrieved document chunks LlamaIndex sends to the model are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros.

A full RAG pipeline also includes embeddings, a vector store and possibly a document parser or observability tool. Each of these processes your documents too. Choose EU-hosted or self-hosted options for them, so that your documents, embeddings and model requests all stay in Europe.