--- title: "Haystack with EU-hosted models · OpenAIChatGenerator setup" description: "Point Haystack's OpenAIChatGenerator at inference.eu with api_base_url and run RAG pipelines and agents on open-weight models processed in the EU." canonical: "https://inference.eu/integrations/haystack/" language: "en" --- Frameworks · Haystack # Use Haystack with open-source models hosted in the EU Haystack, the open-source framework from deepset, builds search, RAG and agent pipelines from composable components. Its \`OpenAIChatGenerator\` takes a custom \`api\_base\_url\`, so every pipeline that generates text can run on open-weight models at inference.eu, processed in European data centres. [Contact sales](https://inference.eu/contact/)[Setup](#setup) 1. 01API format: OpenAI 2. 02EU data centres 3. 03Covered by one DPA ## Set it up in a few minutes 1. 1 Create an API key in your inference.eu account and store it as `INFERENCE_EU_API_KEY`. 2. 2 Install Haystack with `pip install haystack-ai`. 3. 3 Create `OpenAIChatGenerator` with `api_key=Secret.from_env_var("INFERENCE_EU_API_KEY")`, `api_base_url` set to the inference.eu base URL and an inference.eu `model` id. 4. 4 Add the generator to your pipeline, connected to a `ChatPromptBuilder` or used inside an `Agent`. 5. 5 Run the pipeline once with a test question to confirm the connection. python ``` from haystack.components.generators.chat import OpenAIChatGenerator from haystack.dataclasses import ChatMessage from haystack.utils import Secret generator = OpenAIChatGenerator( api_key=Secret.from_env_var("INFERENCE_EU_API_KEY"), api_base_url="https://api.inference.eu/v1", model="gpt-oss-120b", ) result = generator.run([ChatMessage.from_user("Summarise GDPR Article 28 in two sentences.")]) print(result["replies"][0].text) ``` ## What Haystack is Haystack is an open-source Python framework from deepset for building production pipelines around language models. You connect components such as retrievers, rankers, prompt builders and generators into a pipeline graph, which can be serialised to YAML, versioned and deployed. Its `Agent` component adds tool-calling loops on top of any chat generator. Generators are the components that call a language model. `OpenAIChatGenerator` speaks the OpenAI Chat Completions format and accepts a custom endpoint through `api_base_url`, which is the documented way to point it at a compatible API such as inference.eu. ## How the configuration works Create `OpenAIChatGenerator` with `api_base_url` set to the inference.eu base URL ending in `/v1` and `model` set to an inference.eu id. Pass the key as `Secret.from_env_var("INFERENCE_EU_API_KEY")`: Haystack resolves it at run time and never writes the value into serialised pipelines. In YAML pipeline definitions the same settings appear under the generator's `init_parameters`, with `api_key` as an `env_var` secret. Model parameters such as `temperature` or `max_completion_tokens` go into `generation_kwargs`. To stream tokens, pass a `streaming_callback`, for example Haystack's `print_streaming_chunk`. ## Which models to choose For RAG pipelines that answer questions from retrieved documents, `gpt-oss-120b` is a fast default that stays close to the provided context. For long documents, many retrieved passages or answers that need careful synthesis, `glm-5.3` and `kimi-k3` are strong choices. For the `Agent` component and pipelines with tools, choose a model with dependable tool calling such as `qwen3-coder-480b-a35b-instruct`, `glm-5.3` or `kimi-k3`. Pipelines can contain several generators, so you can use a fast model for query rewriting or classification and a stronger one for the final answer. ## Troubleshooting A 401 error means the secret resolved to an empty or wrong key. Check that `INFERENCE_EU_API_KEY` is set in the process that runs the pipeline; `Secret.from_env_var` raises an error at run time if the variable is missing. A 404 usually means `api_base_url` has an extra path or the model id does not exist. If structured output fails, simplify the Pydantic model passed as `response_format` in `generation_kwargs` and validate the result. If an agent loops without calling tools, check the model supports tool calling. Serialised pipelines that still contain `OPENAI_API_KEY` as the secret name will send the wrong key; update the YAML. ## Keeping data in the EU With `api_base_url` pointed at inference.eu, the prompts and retrieved documents your pipelines send to the model are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros. A Haystack pipeline usually includes more than a generator: embedders, a document store, converters and possibly tracing. Each of these handles your documents too. Choose EU-hosted or self-hosted options for them, so that documents, embeddings and model requests all stay in Europe. ## Haystack Category Frameworks Maker deepset API format OpenAI Processing EU data centres Website [haystack.deepset.ai](https://haystack.deepset.ai/) Documentation [docs.haystack.deepset.ai](https://docs.haystack.deepset.ai/docs/openaichatgenerator) ## Haystack: common questions The questions buyers, developers and legal teams ask us most. Something missing? Ask us directly. Create \`OpenAIChatGenerator\` with \`api\_base\_url\`, \`api\_key\` and \`model\`. Haystack documents \`api\_base\_url\` for custom deployments, and every pipeline or agent that uses the generator then sends its requests to that endpoint instead of OpenAI. Use \`Secret.from\_env\_var("INFERENCE\_EU\_API\_KEY")\`. Haystack reads the value at run time and serialises only the variable name, so the key never ends up in pipeline YAML files, logs, notebooks or version control. gpt-oss-120b is a quick default for RAG answers grounded in retrieved documents. GLM-5.3 and Kimi K3 suit long documents and careful synthesis, and Qwen3 Coder 480B, GLM-5.3 and Kimi K3 work well with the \`Agent\` component and tools. Yes. Pass a \`streaming\_callback\`, such as Haystack's \`print\_streaming\_chunk\` or your own function, when you create the generator or call \`run\`. Each chunk is passed to the callback as it arrives from inference.eu. Yes. Pass an \`OpenAIChatGenerator\` configured for inference.eu as the agent's \`chat\_generator\`, together with your tools. Choose a model with dependable tool calling so that the agent loop finishes cleanly with a final answer. Yes. Pass a Pydantic model or JSON schema as \`response\_format\` in \`generation\_kwargs\`. How strictly the schema is followed depends on the model, so validate the reply and keep schemas simple for the most reliable results. Under the generator's \`init\_parameters\`, set \`api\_base\_url\` to the inference.eu base URL, \`model\` to the model id and \`api\_key\` to an \`env\_var\` secret naming \`INFERENCE\_EU\_API\_KEY\`. Load the pipeline as usual with \`Pipeline.loads\`. The secret resolved to the wrong key or none. Check the variable is set in the environment that runs the pipeline, and that serialised pipelines do not still reference \`OPENAI\_API\_KEY\` as the secret name. This page covers text generation. For embedders, choose a model whose hosting meets your requirements, ideally in the EU or self-hosted, so that your documents and queries stay in Europe together with the requests your generator sends. Haystack sends data to the components you configure. It also collects anonymous usage telemetry by default, which contains no documents or prompts; set \`HAYSTACK\_TELEMETRY\_ENABLED=False\` to turn it off. Review document stores and tracing tools for where they keep data. [Ask a question](https://inference.eu/contact/) ## Related [All integrations](https://inference.eu/integrations/) - [LlamaIndexConnect LlamaIndex to inference.eu with the OpenAILike class and run RAG pipelines and agents on open-weight models processed in EU data centres.Frameworks](https://inference.eu/integrations/llamaindex/) - [LangChainConnect LangChain's ChatOpenAI to inference.eu with base\_url and run chains, agents and LangGraph workflows on open-weight models processed in the EU.Frameworks](https://inference.eu/integrations/langchain/) - [Pydantic AIConnect Pydantic AI agents to inference.eu with OpenAICompatibleProvider and get typed, validated output from open-weight models processed in the EU.Frameworks](https://inference.eu/integrations/pydantic-ai/) ## Move your AI workloads to EU infrastructure Tell us which models you use and roughly how much traffic you have. We reply within one working day with pricing and next steps. - Reply within one working day - Pricing for your volume - DPA ready to sign