How do I get JSON output from an open model?

Pass `response_format` with `type: "json_schema"`, a `name`, your JSON Schema and `strict: true` in a chat completions request. The model's reply in `message.content` is then JSON that matches the schema. For a looser guarantee, use `type: "json_object"` and describe the fields in the prompt.

Is this compatible with OpenAI structured outputs?

Yes. `response_format` with `json_schema` and `json_object` follows the OpenAI chat completions API, and `text.format` follows the Responses API. SDK helpers that build the schema from a Pydantic or Zod model, such as `client.chat.completions.parse` in Python, work when pointed at inference.eu.

What is the difference between JSON mode and structured outputs?

JSON mode guarantees that the output is valid JSON but not which keys it contains. Structured outputs with a `json_schema` guarantee that the output follows your schema: required fields present, types correct and no extra properties. Use structured outputs whenever you know the shape you need.

Which JSON Schema features are supported?

Objects, arrays, strings, numbers, integers, booleans, `null`, `enum`, `required`, nested objects, `additionalProperties: false` and descriptions all work. Very complex constructs can be rejected with a 400 error that names the problem. Keep schemas flat and explicit for the most reliable results.

Can I use Pydantic models with structured outputs?

Yes. In Python, define a Pydantic model and pass it as `response_format` to `client.chat.completions.parse`. The SDK converts it to a JSON Schema, sends it, and returns a parsed object in `message.parsed`. Zod does the same in TypeScript with the SDK's helper.

Why is my JSON cut off?

The model reached `max_tokens` before it finished, and `finish_reason` will be `length`. Raise `max_tokens`, shorten long text fields in the schema, or extract fewer items per request. Reasoning models need extra room because their thinking also counts towards the limit.

Can structured outputs be combined with tool calling?

Yes. Tools can use `strict: true` in their function definitions, so their arguments follow the parameter schema, and the final answer can use `response_format`. A common setup lets the model call lookup tools first and then return a structured summary.

Does structured output make the answer correct?

No. It guarantees that the answer has the right shape, not that every value is true. A model can still misread a document or guess a value. Allow `null` for missing data, validate important fields in code, and review a sample of results by hand.

Structured outputs and JSON schema · OpenAI-compatible, EU

extract_invoice.py

import json, os
from openai import OpenAI

client = OpenAI(base_url="https://api.inference.eu/v1", api_key=os.environ["INFERENCE_EU_API_KEY"])

schema = {
    "type": "object",
    "properties": {
        "supplier": {"type": "string"},
        "invoice_number": {"type": "string"},
        "issue_date": {"type": "string", "description": "ISO 8601 date"},
        "currency": {"type": "string", "enum": ["EUR", "GBP", "CHF", "PLN", "SEK"]},
        "total": {"type": "number"},
        "vat_id": {"type": ["string", "null"]},
    },
    "required": ["supplier", "invoice_number", "issue_date", "currency", "total", "vat_id"],
    "additionalProperties": False,
}

reply = client.chat.completions.create(
    model="gpt-oss-120b",
    messages=[
        {"role": "system", "content": "Extract invoice fields. Use null when a field is missing."},
        {"role": "user", "content": open("invoice.md").read()},
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "invoice", "schema": schema, "strict": True},
    },
)
invoice = json.loads(reply.choices[0].message.content)

How structured outputs work

Add response_format with type: "json_schema" and a json_schema object containing a name, your schema written as JSON Schema, and strict: true. While the model generates, the API restricts each token to those that keep the output valid against the schema. The result in message.content is a JSON string you can parse directly.

This is more reliable than asking for JSON in the prompt. The model cannot add a sentence before the opening brace, leave out a required field or invent a property you did not define. You still describe the task in the prompt; the schema controls the shape, the prompt controls the content.

JSON mode and schema design

response_format: {"type": "json_object"} is the looser option, called JSON mode. The output is guaranteed to be valid JSON, but not to follow any particular structure, so mention the expected fields in the prompt. Prefer json_schema whenever you know the shape.

Keep schemas simple and explicit. List every property in required and set additionalProperties to false, which strict mode expects. Represent optional values as a type that allows null, as with vat_id in the example, so the model can say "not found" instead of guessing. Use enum for fixed categories, and add a description to fields whose meaning is not obvious from the name.

A practical extraction pipeline

A common pattern turns documents into data. Convert each PDF to Markdown with the document conversion tool, send the text with a short system prompt and your schema, and parse the result. Validate the parsed object with the same schema in your code, then check business rules the schema cannot express, such as totals adding up.

For thousands of documents, put the same requests into a batch job. Keep a field for the model's confidence or a short notes string if reviewers need to know why a value was chosen. With the Responses API, the same schema goes into text.format instead of response_format.

Errors and edge cases

Very large or deeply nested schemas, and unusual keywords such as complex pattern expressions or oneOf with overlapping branches, may be rejected with a 400 error that names the unsupported part. Split such schemas or simplify them.

The schema guarantees the format, not the facts. A model can still put a wrong date in a valid date field, so check important values. If generation stops at max_tokens, the JSON is cut off and will not parse; check finish_reason and raise the limit. Reasoning models think before they write the JSON, so leave room for both. Refusals for unsafe requests are returned as text, not as schema output.

Structured outputs: JSON that matches your schema

How structured outputs work

JSON mode and schema design

A practical extraction pipeline

Errors and edge cases

Structured outputs: common questions

Related

Move your AI workloads to EU infrastructure