Use Haystack with open-source models hosted in the EU

Haystack is an open-source Python framework from deepset for building production pipelines around language models. You connect components such as retrievers, rankers, prompt builders and generators into a pipeline graph, which can be serialised to YAML, versioned and deployed. Its Agent component adds tool-calling loops on top of any chat generator.

Generators are the components that call a language model. OpenAIChatGenerator speaks the OpenAI Chat Completions format and accepts a custom endpoint through api_base_url, which is the documented way to point it at a compatible API such as inference.eu.

Create OpenAIChatGenerator with api_base_url set to the inference.eu base URL ending in /v1 and model set to an inference.eu id. Pass the key as Secret.from_env_var("INFERENCE_EU_API_KEY"): Haystack resolves it at run time and never writes the value into serialised pipelines.

In YAML pipeline definitions the same settings appear under the generator's init_parameters, with api_key as an env_var secret. Model parameters such as temperature or max_completion_tokens go into generation_kwargs. To stream tokens, pass a streaming_callback, for example Haystack's print_streaming_chunk.

For RAG pipelines that answer questions from retrieved documents, gpt-oss-120b is a fast default that stays close to the provided context. For long documents, many retrieved passages or answers that need careful synthesis, glm-5.3 and kimi-k3 are strong choices.

For the Agent component and pipelines with tools, choose a model with dependable tool calling such as qwen3-coder-480b-a35b-instruct, glm-5.3 or kimi-k3. Pipelines can contain several generators, so you can use a fast model for query rewriting or classification and a stronger one for the final answer.

A 401 error means the secret resolved to an empty or wrong key. Check that INFERENCE_EU_API_KEY is set in the process that runs the pipeline; Secret.from_env_var raises an error at run time if the variable is missing. A 404 usually means api_base_url has an extra path or the model id does not exist.

If structured output fails, simplify the Pydantic model passed as response_format in generation_kwargs and validate the result. If an agent loops without calling tools, check the model supports tool calling. Serialised pipelines that still contain OPENAI_API_KEY as the secret name will send the wrong key; update the YAML.

With api_base_url pointed at inference.eu, the prompts and retrieved documents your pipelines send to the model are processed in EU data centres and never used for training. We sign a data processing agreement under Article 28 GDPR, publish our subprocessor list and bill in euros.

A Haystack pipeline usually includes more than a generator: embedders, a document store, converters and possibly tracing. Each of these handles your documents too. Choose EU-hosted or self-hosted options for them, so that documents, embeddings and model requests all stay in Europe.