--- title: "AnythingLLM with EU-hosted models · Generic OpenAI setup" description: "Connect AnythingLLM to inference.eu with the Generic OpenAI provider and chat with your documents using open-weight models processed in the EU." canonical: "https://inference.eu/integrations/anythingllm/" language: "en" --- Chat apps · AnythingLLM # Use AnythingLLM with open-source models hosted in the EU AnythingLLM is a desktop and self-hosted application for chatting with your documents and running agents. Its Generic OpenAI provider connects to any OpenAI-compatible API, so you can keep your documents and vector store local while the model runs on open-weight models at inference.eu, in European data centres. [Contact sales](https://inference.eu/contact/)[Setup](#setup) 1. 01API format: OpenAI 2. 02EU data centres 3. 03Covered by one DPA ## Set it up in a few minutes 1. 1 Create an API key in your inference.eu account. 2. 2 In AnythingLLM, open Settings > AI Providers > LLM and choose Generic OpenAI. 3. 3 Enter the inference.eu base URL, your API key and a chat model name such as `glm-5.3`. 4. 4 Set the model context window and maximum tokens to match the model, then save. 5. 5 Open a workspace, upload a document and ask a question about it to confirm the connection. .env ``` LLM_PROVIDER='generic-openai' GENERIC_OPEN_AI_BASE_PATH='https://api.inference.eu/v1' GENERIC_OPEN_AI_API_KEY='' GENERIC_OPEN_AI_MODEL_PREF='glm-5.3' GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT=131072 DISABLE_TELEMETRY='true' # Desktop app: Settings > AI Providers > LLM > Generic OpenAI, # then fill in Base URL, API Key, Chat Model Name and context window. ``` ## What AnythingLLM is AnythingLLM from Mintplex Labs is an open-source application for retrieval-augmented chat and agents. It runs as a desktop app on Windows, macOS and Linux, or as a multi-user Docker deployment. You organise documents into workspaces, and AnythingLLM splits, embeds and stores them so that answers can cite your own files. Agents add web browsing, file handling and custom skills. By default, embeddings and the vector database run locally. Only the language model needs a provider, and the Generic OpenAI option connects to any OpenAI-compatible API, which is how you use inference.eu. ## How the configuration works In Settings > AI Providers > LLM, select Generic OpenAI and fill in the Base URL (the inference.eu base URL ending in `/v1`), the API key, the chat model name exactly as inference.eu lists it, the model's context window and the maximum tokens per reply. Each workspace can also override the provider and model. For Docker deployments, the same settings can be set in the server `.env` with `LLM_PROVIDER='generic-openai'`, `GENERIC_OPEN_AI_BASE_PATH`, `GENERIC_OPEN_AI_API_KEY`, `GENERIC_OPEN_AI_MODEL_PREF` and `GENERIC_OPEN_AI_MODEL_TOKEN_LIMIT`. The Generic OpenAI option is very flexible, so check every field carefully. ## Which models to choose Document chat benefits from a model that stays close to the retrieved text and handles long context. `glm-5.3` and `kimi-k3` are strong choices for answering questions across many documents, comparing contracts or summarising reports. For quick questions and lighter workspaces, `gpt-oss-120b` responds fast. AnythingLLM agents call tools for web browsing, saving files and custom skills, so use a model with dependable tool calling, such as `kimi-k3` or `glm-5.3`, in workspaces where you use `@agent`. Set the context window to the model's real limit so AnythingLLM can include enough retrieved snippets. ## Troubleshooting A 401 error means the API key is wrong or was pasted with extra spaces. A 404 usually means the base URL is missing `/v1` or includes `/chat/completions`. "Model not found" means the chat model name differs from the inference.eu id; type it exactly. If answers ignore your documents, the context window is set too low or too few snippets are retrieved; raise the token limit and check the workspace's similarity threshold. If agents do not call tools, switch the workspace agent to a model with stronger tool calling. After changing provider settings, start a new chat so old context does not interfere. ## Keeping data in the EU With inference.eu as the LLM provider, the questions and retrieved document snippets AnythingLLM sends to the model are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros. AnythingLLM keeps documents, embeddings and the vector database on your machine or server by default. It collects anonymous usage telemetry unless you set `DISABLE_TELEMETRY` to true or turn it off under Privacy. Agent skills such as web search use their own providers, so review those before enabling them. ## AnythingLLM Category Chat apps Maker Mintplex Labs API format OpenAI Processing EU data centres Website [anythingllm.com](https://anythingllm.com/) Documentation [docs.anythingllm.com](https://docs.anythingllm.com/setup/llm-configuration/cloud/openai-generic) ## AnythingLLM: common questions The questions buyers, developers and legal teams ask us most. Something missing? Ask us directly. Open Settings > AI Providers > LLM and choose Generic OpenAI. Enter the base URL, API key, chat model name, context window and maximum tokens, then save. Every workspace that uses the system default will then send requests to that endpoint. GLM-5.3 and Kimi K3 handle long context and stay close to the retrieved text, which suits contracts, reports and manuals. gpt-oss-120b is a quick choice for lighter workspaces and short questions. Enter the context window of the model you selected, as listed at inference.eu. AnythingLLM uses it to decide how much chat history and how many document snippets fit into each request. A value that is too low makes answers ignore your documents. Documents are embedded and stored locally by default. Only the question and the relevant snippets retrieved for it are sent to the model at inference.eu, where they are processed in EU data centres and never used for training. Yes. In Docker deployments, set \`LLM\_PROVIDER='generic-openai'\` with \`GENERIC\_OPEN\_AI\_BASE\_PATH\`, \`GENERIC\_OPEN\_AI\_API\_KEY\`, \`GENERIC\_OPEN\_AI\_MODEL\_PREF\` and \`GENERIC\_OPEN\_AI\_MODEL\_TOKEN\_LIMIT\` in the server \`.env\`, then restart the container. Values set later in the settings screen are saved by the app and apply from then on. Yes. Agents use the workspace's model or a dedicated agent model. Choose one with dependable tool calling, such as Kimi K3 or GLM-5.3, so that skills like web browsing, file saving and custom tools are called reliably. The chat model name must match the inference.eu model id exactly, for example \`glm-5.3\`. Generic OpenAI does not offer a model list, so typos are a common cause. Correct the name in settings and start a new chat. AnythingLLM collects anonymous usage telemetry by default, without document contents or chats. Set \`DISABLE\_TELEMETRY='true'\` in the server \`.env\`, or turn it off in the app under Privacy, if you want no data to leave your environment besides model requests. Yes. Each workspace can override the system LLM with its own provider and model. You can, for example, use GLM-5.3 for a legal workspace with long contracts and gpt-oss-120b for a general team workspace. AnythingLLM ships with a built-in local embedder and vector database, which keep your document embeddings on your own machine. That is usually the simplest choice; the LLM provider setting on this page covers chat and agents. [Ask a question](https://inference.eu/contact/) ## Related [All integrations](https://inference.eu/integrations/) - [Open WebUIConnect Open WebUI to inference.eu as an OpenAI-compatible connection and give your team a self-hosted chat on open-weight models processed in the EU.Chat apps](https://inference.eu/integrations/open-webui/) - [LibreChatAdd inference.eu as a custom endpoint in librechat.yaml and give your organisation a self-hosted chat with agents on open-weight models processed in the EU.Chat apps](https://inference.eu/integrations/librechat/) - [LlamaIndexConnect LlamaIndex to inference.eu with the OpenAILike class and run RAG pipelines and agents on open-weight models processed in EU data centres.Frameworks](https://inference.eu/integrations/llamaindex/) ## Move your AI workloads to EU infrastructure Tell us which models you use and roughly how much traffic you have. We reply within one working day with pricing and next steps. - Reply within one working day - Pricing for your volume - DPA ready to sign