Use AnythingLLM with open-source models hosted in the EU

AnythingLLM from Mintplex Labs is an open-source application for retrieval-augmented chat and agents. It runs as a desktop app on Windows, macOS and Linux, or as a multi-user Docker deployment. You organise documents into workspaces, and AnythingLLM splits, embeds and stores them so that answers can cite your own files. Agents add web browsing, file handling and custom skills.

By default, embeddings and the vector database run locally. Only the language model needs a provider, and the Generic OpenAI option connects to any OpenAI-compatible API, which is how you use inference.eu.

In Settings > AI Providers > LLM, select Generic OpenAI and fill in the Base URL (the inference.eu base URL ending in /v1), the API key, the chat model name exactly as inference.eu lists it, the model's context window and the maximum tokens per reply. Each workspace can also override the provider and model.

For Docker deployments, the same settings can be set in the server .env with LLM_PROVIDER='generic-openai', GENERIC_OPEN_AI_BASE_PATH, GENERIC_OPEN_AI_API_KEY, GENERIC_OPEN_AI_MODEL_PREF and GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT. The Generic OpenAI option is very flexible, so check every field carefully.

Document chat benefits from a model that stays close to the retrieved text and handles long context. glm-5.3 and kimi-k3 are strong choices for answering questions across many documents, comparing contracts or summarising reports.

For quick questions and lighter workspaces, gpt-oss-120b responds fast. AnythingLLM agents call tools for web browsing, saving files and custom skills, so use a model with dependable tool calling, such as kimi-k3 or glm-5.3, in workspaces where you use @agent. Set the context window to the model's real limit so AnythingLLM can include enough retrieved snippets.

A 401 error means the API key is wrong or was pasted with extra spaces. A 404 usually means the base URL is missing /v1 or includes /chat/completions. "Model not found" means the chat model name differs from the inference.eu id; type it exactly.

If answers ignore your documents, the context window is set too low or too few snippets are retrieved; raise the token limit and check the workspace's similarity threshold. If agents do not call tools, switch the workspace agent to a model with stronger tool calling. After changing provider settings, start a new chat so old context does not interfere.

With inference.eu as the LLM provider, the questions and retrieved document snippets AnythingLLM sends to the model are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros.

AnythingLLM keeps documents, embeddings and the vector database on your machine or server by default. It collects anonymous usage telemetry unless you set DISABLE_TELEMETRY to true or turn it off under Privacy. Agent skills such as web search use their own providers, so review those before enabling them.