> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# LLM Providers
> Configure LLM providers for text generation and reasoning in Cognee
LLM (Large Language Model) providers handle text generation, reasoning, and structured output tasks in Cognee. You can choose from cloud providers like OpenAI and Anthropic, or run models locally with Ollama.
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
## Supported Providers
Cognee supports multiple LLM providers:
* **OpenAI** — GPT models via OpenAI API (default)
* **Azure OpenAI** — GPT models via Azure OpenAI Service
* **Google Gemini** — Gemini models via Google AI
* **Anthropic** — Claude models via Anthropic API
* **AWS Bedrock** — Models available via AWS Bedrock
* **Groq** — Fast inference via Groq API (via LiteLLM)
* **Ollama** — Local models via Ollama
* **LM Studio** — Local models via LM Studio
* **HuggingFace** — Models via HuggingFace Inference API or Inference Endpoints
* **llama.cpp** — Local models via llama-cpp-python (in-process or server mode)
* **Custom** — OpenAI-compatible endpoints (like vLLM, OpenRouter, DeepInfra, company-internal)
* **MCP Sampling** — Reuse the host harness's LLM via MCP `sampling/createMessage` (no `LLM_API_KEY`; only when Cognee runs as an MCP server under a host that grants sampling)
**Model names are not an allowlist.** Any model reachable through an OpenAI-compatible endpoint can be configured — see [Custom Providers](#custom-providers) for the generic path and its compatibility requirements.
**LLM/Embedding Configuration**: If you configure only LLM or only embeddings, the other defaults to OpenAI. Cognee rejects this mismatch up front — `add()` and `remember()` fail with `ProviderConfigMismatchError` before any ingestion work happens. Configure both LLM and embeddings, or keep a working OpenAI API key for the side you leave at its defaults — see [LLM/Embedding Configuration](/setup-configuration/overview#configuration-workflow).
## Choosing a Model
Cognee always uses **two** models together: an **LLM** for entity/relationship extraction and reasoning, and an **embedding model** for semantic search. An embedding model is mandatory — every `cognify` run writes vectors to a [vector store](/setup-configuration/vector-stores), and [recall](/core-concepts/main-operations/recall) depends on them. If you only set one, the other silently falls back to OpenAI (see the warning above).
A small, fast model is the right default. Cognee ships with one (`openai/gpt-5-mini`) and the examples on this page use comparable light models such as `gpt-4o-mini`. Knowledge-graph extraction is many short, schema-constrained calls per document rather than a few long ones, so a light model keeps cost and latency low while handling most workloads well.
Reach for a more powerful model when:
* Your sources are dense or domain-specific (legal, medical, scientific) and you need higher-fidelity entities and relationships.
* You use a [custom graph model](/guides/custom-graph-model) or [ontology](/guides/ontology-support) with a complex schema the model must populate accurately.
* A light model produces noisy or incomplete graphs on your data.
Extraction relies on [structured output](/setup-configuration/structured-output-backends), so very small or weak models may return malformed JSON or lower-quality graphs. If a small local model struggles, try a stronger one or adjust the [instructor mode](#llm-instructor-modes).
The embedding model is lightweight — defaults like `nomic-embed-text` (Ollama) or `all-MiniLM-L6-v2` ([Fastembed](/setup-configuration/embedding-providers#fastembed-local), CPU-only) run comfortably on a CPU or a small GPU and rarely dominate resource use.
The **LLM** is the constraint for local setups. The [Local Setup guide](/guides/local-setup) defaults to an 8B model (`llama3.1:8b`); as a rough guide, an 8B model quantized to 4-bit needs roughly 6 GB of free VRAM, while larger or less-quantized models need proportionally more. If a model does not fit, it spills to system RAM and CPU, which still works but is much slower — for [llama.cpp](#llama-cpp-local) you can tune `LLAMA_CPP_N_GPU_LAYERS` to offload only as many layers as fit. Limited VRAM does not change graph quality; it mainly affects how fast `cognify` runs, since extraction issues many sequential LLM calls per document. With low VRAM, prefer a smaller LLM and lower `EMBEDDING_BATCH_SIZE` (see [Embedding Providers](/setup-configuration/embedding-providers#batch-size)) over a large model that does not fit.
`llama3.1:8b` is the recommended Ollama default because it hits a practical sweet spot for Cognee's workload: it follows instructions and produces valid [structured output](/setup-configuration/structured-output-backends) reliably (the Ollama provider defaults to [`json_schema_mode`](#llm-instructor-modes), so Ollama 0.5+ enforces the schema rather than just being asked for JSON), while staying small enough to run on modest hardware (\~6 GB VRAM at 4-bit). Other validated options: `llama3.2:3b` for lightweight or resource-constrained environments, and `qwen2.5:14b` as a mid-size alternative; larger tags (`llama3.1:70b`, `llama3.3`, `qwen2.5:32b`) improve extraction fidelity on dense or domain-specific sources at proportionally higher resource cost. For the full classification — recommended and problematic models, and what Cognee logs at startup for each — see [Model Support Warning](#model-support-warning) in the Ollama setup guide below; on problematic models the failures surface as `InstructorRetryException` errors or empty graphs.
## Configuration
Set these environment variables in your `.env` file:
* `LLM_PROVIDER` — The provider to use: `openai`, `azure`, `anthropic`, `gemini`, `mistral`, `bedrock`, `ollama`, `llama_cpp`, `custom`, `mcp-sampling`. Optional — when it is unset, Cognee infers it from the `LLM_MODEL` prefix (see the note below)
* `LLM_MODEL` — The specific model to use
* `LLM_API_KEY` — Your API key for the provider (not used by `mcp-sampling`)
* `LLM_ENDPOINT` — Custom endpoint URL (for Azure, Ollama, or custom providers)
* `LLM_API_VERSION` — API version (for Azure OpenAI)
* `LLM_TEMPERATURE` — Sampling temperature, sent with every LLM call when you set it explicitly — and, on local inference servers (`ollama`, `llama_cpp`, LM Studio), also when you don't: there an unset value sends `0.0`. On every other provider, leaving it unset sends no temperature at all and the provider's own default applies (see [Temperature and Seed](#temperature-and-seed))
* `LLM_SEED` — Sampling seed for reproducible outputs, sent when set (provider support varies)
* `LLM_MAX_COMPLETION_TOKENS` — Maximum tokens per request (optional)
* `LLM_INSTRUCTOR_MODE` — Structured-output mode override for Instructor-backed LLM calls (optional)
* `LLM_EXTRACTION_*`, `LLM_SUMMARIZATION_*`, `LLM_QUERY_*` — Optional per-stage overrides that route individual pipeline stages to different models/providers (see [Per-Stage Model Routing](#per-stage-model-routing))
A preflight LLM connection test can time out at 30s, especially against smaller models. Workaround: add `COGNEE_SKIP_CONNECTION_TEST=true` to your `.env`.
**Why do model names have a prefix like `gemini/` or `openrouter/`?**
Cognee routes all LLM requests through [LiteLLM](https://docs.litellm.ai/docs/providers), which uses provider prefixes to identify the correct API endpoint. For example, Google lists their model as `gemini-2.0-flash`, but in Cognee you must write `gemini/gemini-2.0-flash`. This prefix tells LiteLLM to use the Gemini API. The same applies to custom providers — `openrouter/`, `hosted_vllm/`, `lm_studio/`, etc. See each provider section below for the correct format.
### Provider Inference
`LLM_PROVIDER` is optional. When you don't set it, Cognee infers the provider from the prefix of `LLM_MODEL` — `anthropic/claude-3-5-sonnet` resolves to `anthropic`, `gemini/gemini-2.0-flash` to `gemini`, and so on.
Precedence, highest first:
1. An explicit `llm_provider` passed in Python (for example to `cognee.config.set_llm_config()`).
2. The `LLM_PROVIDER` environment variable.
3. Inference from the `LLM_MODEL` prefix.
An explicitly set provider always wins, even when it disagrees with the model prefix — that is what makes the Azure recipe below (`LLM_PROVIDER="openai"` with `LLM_MODEL="azure/gpt-4o-mini"`) keep working. A model id with no `/` (for example `llama3.1:8b`) has no prefix to infer from, so it falls back to the default `openai` unless you set `LLM_PROVIDER` yourself.
Inference only recognises prefixes Cognee has an adapter for: `openai`, `azure`, `anthropic`, `gemini`, `mistral`, `bedrock`, `ollama`, `llama_cpp`, `custom`, `mcp-sampling`.
Inference resolves `LLM_PROVIDER` *from* the model name. The opposite direction also exists, but only on the `litellm_native` [structured output backend](/setup-configuration/structured-output-backends): once the provider is known, a model id LiteLLM cannot route on its own — a bare `llama3.1:8b`, or a namespaced `library/phi4` — is re-qualified with that provider's prefix before the request is sent. See [Model-name qualification](/setup-configuration/structured-output-backends#configuration) for which providers are prefixed.
Inference runs once, when the configuration is loaded from the environment. Setting `llm_model` later in Python via `cognee.config.set_llm_config()` does **not** re-trigger it — the provider stays whatever it already was (the default `openai` unless configured otherwise). When you configure the model in Python instead of through `LLM_MODEL`, set `llm_provider` explicitly in the same call.
Any other prefix raises `ProviderNotDeducibleError` — Cognee refuses to guess rather than silently falling back to OpenAI. This includes LiteLLM-routed prefixes Cognee has no adapter of its own for, such as `openrouter/`, `groq/`, or `deepseek/`. For those, set `LLM_PROVIDER="custom"`, as every recipe in [Custom Providers](#custom-providers) already does — with `custom`, the prefix is passed straight through to LiteLLM for routing.
## Provider Setup Guides
OpenAI is the default provider and works out of the box with minimal configuration.
```dotenv theme={null}
LLM_PROVIDER="openai"
LLM_MODEL="gpt-4o-mini"
LLM_API_KEY="sk-..."
# Optional overrides
# LLM_ENDPOINT=https://api.openai.com/v1
# LLM_API_VERSION=
# LLM_MAX_COMPLETION_TOKENS=16384
```
Use Azure OpenAI Service with your own deployment.
```dotenv theme={null}
LLM_PROVIDER="openai"
LLM_MODEL="azure/gpt-4o-mini"
LLM_ENDPOINT="https://.openai.azure.com/openai/deployments/gpt-4o-mini"
LLM_API_KEY="az-..."
LLM_API_VERSION="2024-12-01-preview"
```
Cognee routes Gemini requests through [LiteLLM](https://docs.litellm.ai/docs/providers/gemini). There are two ways to reach Gemini models: the **Google AI Studio** API (a single API key) or **Vertex AI** (Google Cloud project + service-account credentials).
The simplest setup. Get an API key from [Google AI Studio](https://aistudio.google.com/apikey) and use the `gemini/` model prefix.
```dotenv theme={null}
LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-2.0-flash"
LLM_API_KEY="AIza..."
# Optional
# LLM_ENDPOINT=https://generativelanguage.googleapis.com/
# LLM_API_VERSION=v1beta
```
This path talks to the Gemini REST API directly and needs no extra Google packages.
Use Vertex AI when your models are served through a Google Cloud project. Vertex routes through LiteLLM's `vertex_ai/` prefix and authenticates with Google Cloud credentials instead of an API key.
```dotenv theme={null}
LLM_PROVIDER="gemini"
LLM_MODEL="vertex_ai/gemini-2.0-flash"
LLM_API_KEY="." # placeholder; Vertex auth uses credentials below
# Google Cloud project + region (read by LiteLLM)
VERTEXAI_PROJECT="your-gcp-project-id"
VERTEXAI_LOCATION="us-central1"
# Path to your service-account key file (Application Default Credentials)
GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
```
**Installation**: Vertex AI requires the Google client libraries, which Cognee does not bundle by default. Install them with:
```bash theme={null}
uv pip install google-cloud-aiplatform
```
`GOOGLE_APPLICATION_CREDENTIALS` points at a service-account JSON key. If you run inside Google Cloud (or after `gcloud auth application-default login`), you can omit it and rely on [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
Use Anthropic's Claude models for reasoning tasks.
```dotenv theme={null}
LLM_PROVIDER="anthropic"
LLM_MODEL="claude-sonnet-4-5-20250929"
LLM_API_KEY="sk-ant-..."
```
Groq provides fast inference for open models. Cognee routes Groq requests through [LiteLLM](https://docs.litellm.ai/docs/providers/groq) using the `groq/` model prefix.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="groq/llama-3.3-70b-versatile"
LLM_API_KEY="gsk_..."
```
**Installation**: Install the Groq dependency:
```bash theme={null}
pip install cognee[groq]
```
**Popular Groq models** (use with the `groq/` prefix):
* `groq/llama-3.3-70b-versatile`
* `groq/llama3-8b-8192`
* `groq/mixtral-8x7b-32768`
* `groq/gemma2-9b-it`
See the [Groq model list](https://console.groq.com/docs/models) for all available models. Your Groq API key can be created in the [Groq Console](https://console.groq.com/keys).
**No endpoint needed**: The `LLM_ENDPOINT` variable is not required for Groq — LiteLLM resolves the Groq API endpoint automatically from the `groq/` prefix.
Use models available on AWS Bedrock for various tasks. For Bedrock specifically, you will need to
also specify some information regarding AWS.
```dotenv theme={null}
LLM_API_KEY=""
LLM_MODEL="eu.amazon.nova-lite-v1:0"
LLM_PROVIDER="bedrock"
LLM_MAX_COMPLETION_TOKENS="16384"
AWS_REGION=""
AWS_ACCESS_KEY_ID=""
AWS_SECRET_ACCESS_KEY=""
AWS_SESSION_TOKEN=""
# Optional parameters
#AWS_BEDROCK_RUNTIME_ENDPOINT="bedrock-runtime.eu-west-1.amazonaws.com"
#AWS_PROFILE_NAME=""
```
There are **multiple ways of connecting** to Bedrock models. Cognee picks the first one it finds, in this order:
1. Using an API key and region. Simply generate your key on AWS, and put it in the `LLM_API_KEY` env variable. If `LLM_API_KEY` is set, it takes precedence over the credential and profile methods below, so leave it empty when you want to use those.
2. Using AWS Credentials. You can only specify `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, no need for the `LLM_API_KEY`.
In this case, if you are using temporary credentials (e.g. `AWS_ACCESS_KEY_ID` starting with `ASIA...`, such as those issued by `aws sso login` or `aws sts assume-role`), then you also
must specify the `AWS_SESSION_TOKEN`. All three values expire and must be refreshed when AWS rotates them.
3. Using AWS profiles. `AWS_PROFILE_NAME` is the **name** of a profile (for example `default` or `my-sso-profile`), not a path to a file or to a folder. Cognee hands the name to boto3, which resolves the credentials through the standard AWS chain using the shared config and credentials files at `~/.aws/config` and `~/.aws/credentials` (override their locations with the `AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` env variables). This is the recommended path for **AWS SSO**: run `aws sso login --profile ` first, then set `AWS_PROFILE_NAME` to that profile name and Cognee will use the temporary SSO credentials boto3 caches for it — no need to copy the `ASIA...` keys into your `.env`.
4. Using an **ambient IAM role**. If `LLM_API_KEY`, the `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` pair, and `AWS_PROFILE_NAME` are all unset, Cognee passes no credentials to Bedrock and boto3 resolves them from the default AWS credential chain — an EC2 instance profile, ECS task role, or EKS IRSA service-account role. Unlike most providers, `bedrock` does not require `LLM_API_KEY`, so no missing-key error is raised; `LLM_PROVIDER`, `LLM_MODEL`, and `AWS_REGION` are enough to authenticate to Bedrock (an [embedding provider](/setup-configuration/embedding-providers) still needs its own configuration). Since resolution is first-match, remove any stale key, credential, or profile values from your `.env` and shell — they take precedence over the role.
**Installation**: Install the required dependency:
```bash theme={null}
pip install cognee[aws]
```
**Model Name**
The name of the model might differ based on the region (the name begins with **eu** for Europe, **us** of USA, etc.)
See the [AWS Bedrock Integration](/integrations/aws-bedrock-integration) guide for the full setup walkthrough.
Run models locally with Ollama for privacy and cost control.
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434/v1"
LLM_API_KEY="ollama"
```
`LLM_API_KEY="ollama"` is a placeholder required by the client library — Ollama itself does not validate it.
**Installation**: Install Ollama from [ollama.ai](https://ollama.ai) and pull your desired model:
```bash theme={null}
ollama pull llama3.1:8b
```
**Namespaced tags and Hugging Face GGUFs**: Ollama model names are not always bare tags — they can be namespaced (`library/phi4`), and a GGUF pulled from Hugging Face keeps its full path. Both work as `LLM_MODEL` on the `litellm_native` [structured output backend](/setup-configuration/structured-output-backends), which qualifies them to `ollama/…` for routing:
```bash theme={null}
ollama pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
```
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF"
LLM_ENDPOINT="http://localhost:11434/v1"
LLM_API_KEY="ollama"
STRUCTURED_OUTPUT_FRAMEWORK="litellm_native"
```
Setting `LLM_PROVIDER="ollama"` is required here, not optional: `library` and `hf.co` are not prefixes [provider inference](#provider-inference) recognises, so leaving `LLM_PROVIDER` unset raises `ProviderNotDeducibleError` at configuration load.
**Zero-API-key setup**: To avoid falling back to OpenAI for embeddings, you must also configure the embedding provider to use a local backend. See the [Local Setup guide](/guides/local-setup) for a complete `.env` example using Ollama or Fastembed for both LLM and embeddings.
### Model Support Warning
When `LLM_PROVIDER="ollama"`, Cognee classifies the configured `LLM_MODEL` against a built-in support matrix as the LLM configuration is loaded, and logs a warning for models it has not validated for structured graph extraction. The check is **advisory only** — nothing is blocked and no exception is raised, so `cognify()` runs either way. The line appears the first time the LLM configuration is loaded in the process.
| Classification | Models | Logged |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Recommended | `llama3`, `llama3.1`, `llama3.2`, `llama3.3`, and any `qwen2.5` tag of 14B or larger (`qwen2.5:14b`, `qwen2.5:32b`, …) | Nothing |
| Problematic | `mistral`, `phi3`, `phi3.5`, and `qwen2.5` below 14B or with no parseable size in its tag (`qwen2.5:7b`, `qwen2.5:latest`, bare `qwen2.5`) | A warning that the model has known limitations for structured graph extraction (schema validation errors, silent drops) and a suggestion to switch to a validated model |
| Unknown | Every model not in the matrix, including `gemma2` | A warning that the model has not been validated and that extraction quality may vary |
Matching ignores an `ollama/` prefix and everything after the `:` in a tag, so `llama3.1`, `llama3.1:8b`, and `ollama/llama3.1:70b` are all classified as `llama3.1`. `qwen2.5` is the exception: its tag is parsed for a parameter count, and that count is what decides between recommended and problematic. A custom Modelfile tag (see below) inherits the classification of its base name — `llama3.1:8b-8k` is still `llama3.1`.
Each distinct `LLM_MODEL` value warns at most once per process, so you see the line once even though the configuration is read many times. The message points at [`docs/ollama_models.md`](https://github.com/topoteretes/cognee/blob/dev/docs/ollama_models.md) in the repo, which carries a similar matrix plus troubleshooting notes.
### Known Issues
* **`ValidationError` on import (`Missing: [...]`)**: Cognee validates the LLM variables as an all-or-nothing group when `LLM_PROVIDER="ollama"` — if you set any of `LLM_MODEL`, `LLM_ENDPOINT`, or `LLM_API_KEY`, you must set all three. Setting only some raises `Value error, You have set some but not all of the required environment variables for LLM usage`.
The check reads the **resolved** LLM configuration rather than the process environment, so values loaded from a `.env` file count exactly the same as exported environment variables. A blank or whitespace-only value counts as unset, and `LLM_MODEL` counts as set only if you actually supply it — leaving it at its default (`openai/gpt-5-mini`) does not satisfy the group, so setting just `LLM_ENDPOINT` and `LLM_API_KEY` fails with `Missing: ['LLM_MODEL']`.
The embedding variables are **no longer** part of this check: `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL` and `EMBEDDING_DIMENSIONS` are owned by the embedding configuration and can be set independently of one another. Setting them together is still the recommended configuration for a local Ollama setup, since Ollama embeddings need `HUGGINGFACE_TOKENIZER` for token counting and an explicit `EMBEDDING_DIMENSIONS` avoids a wrong auto-derived vector size:
```dotenv theme={null}
EMBEDDING_PROVIDER="ollama"
EMBEDDING_MODEL="nomic-embed-text:latest"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embed"
EMBEDDING_DIMENSIONS="768"
HUGGINGFACE_TOKENIZER="nomic-ai/nomic-embed-text-v1.5"
```
See [Embedding Providers → Ollama](/setup-configuration/embedding-providers#ollama-local) for model-to-tokenizer mappings and how to find the right `HUGGINGFACE_TOKENIZER` value.
* **`NoDataError` with mixed providers**: Using Ollama as LLM and OpenAI as embedding provider may fail with `NoDataError`. Workaround: configure both LLM and embeddings to the same local provider (see the local setup guide above).
* **Audio transcription is not supported**: `AudioLoader` relies on a Whisper-compatible transcription endpoint. Cognee's Ollama adapter does not provide one, so audio ingestion will fail when `LLM_PROVIDER="ollama"`.
### Context Window (`num_ctx`) and Custom Modelfiles
If `cognify()` returns HTTP **500 errors** while the same model answers fine when you run `ollama run ` in a terminal, the usual cause is **context-window truncation**, not a connection problem.
Ollama's default context window can be much smaller than Cognee's extraction window. Cognee sizes extraction chunks from `LLM_MAX_COMPLETION_TOKENS` (default `16384`) — up to roughly half that per chunk — so the entity-extraction prompts it sends can be far larger than a short terminal prompt. When a prompt exceeds `num_ctx`, Ollama may truncate it, the model can return malformed or empty structured output, and Instructor's parse failure can surface as a 500.
`LLM_MODEL` is just an Ollama model tag, so the fix is to point it at a model whose `num_ctx` is large enough. Create a custom [Modelfile](https://docs.ollama.com/modelfile):
```dockerfile theme={null}
FROM llama3.1:8b
PARAMETER num_ctx 8192
```
Build the tag and reference it in your `.env`:
```bash theme={null}
ollama create llama3.1:8b-8k -f Modelfile
```
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b-8k"
LLM_ENDPOINT="http://localhost:11434/v1"
LLM_API_KEY="ollama"
```
A larger `num_ctx` uses more memory. If you can't raise it, lower `LLM_MAX_COMPLETION_TOKENS` instead so Cognee builds smaller chunks that fit the model's existing context window.
### Connection Troubleshooting
If you see `cannot connect to host` or `connection refused` errors, the most common causes are an unreachable endpoint, the wrong protocol, or a Docker networking mismatch.
**Default endpoint protocol**
Ollama's local server speaks plain **HTTP**, not HTTPS. Cognee does not add TLS by default — the protocol is determined entirely by the scheme in `LLM_ENDPOINT` and `EMBEDDING_ENDPOINT`. Use `https://` only if you have placed Ollama behind a TLS-terminating reverse proxy (Caddy, nginx, Traefik, etc.). For a local Ollama setup, use:
| Variable | Value |
| ----------------------------- | ---------------------------------- |
| `LLM_ENDPOINT` (Ollama) | `http://localhost:11434/v1` |
| `EMBEDDING_ENDPOINT` (Ollama) | `http://localhost:11434/api/embed` |
The Ollama embedding engine builds a secure SSL context for outgoing requests, but it is only applied when the endpoint URL uses `https://` — plain HTTP requests are not upgraded.
**`localhost` vs `host.docker.internal`**
Inside a Docker container, `localhost` refers to the container itself, not your host machine where Ollama is running. If Cognee runs in Docker and Ollama runs on the host, use `host.docker.internal` instead:
```dotenv theme={null}
LLM_ENDPOINT="http://host.docker.internal:11434/v1"
EMBEDDING_ENDPOINT="http://host.docker.internal:11434/api/embed"
```
`host.docker.internal` is available on Docker Desktop (macOS/Windows) and on Linux when the `host-gateway` mapping is configured in `docker-compose.yml`. On Linux without that mapping, use `--network host` or the Docker bridge IP.
**Other common causes**
* **Ollama not running**: verify with `curl http://localhost:11434/api/tags` from the same machine and network namespace Cognee is running in.
* **Wrong port**: the default Ollama port is `11434`. If you started Ollama with `OLLAMA_HOST=0.0.0.0:`, match that port in `LLM_ENDPOINT`.
* **Missing path suffix**: the LLM endpoint must end in `/v1` (OpenAI-compatible chat completions), and the embedding endpoint must end in `/api/embed`. Pointing either at the bare host (e.g. `http://localhost:11434`) will fail.
* **Bind address**: Ollama binds to `127.0.0.1` by default. To accept connections from other machines or Docker containers via a LAN IP, start it with `OLLAMA_HOST=0.0.0.0:11434`.
* **Inside Docker Compose**: if the LLM or embedding endpoint runs on your host machine, `localhost` inside the container points back to the container itself. Use `host.docker.internal` on Docker Desktop (macOS/Windows), or add a `host-gateway` mapping in `docker-compose.yml` on Linux.
```dotenv theme={null}
LLM_ENDPOINT="http://host.docker.internal:11434/v1"
EMBEDDING_ENDPOINT="http://host.docker.internal:11434/api/embed"
```
If the service runs in the same Compose project, use the Compose **service name** instead of `localhost` for any `DB_HOST`, `VECTOR_DB_URL`, or `GRAPH_DATABASE_URL` setting.
Use models from HuggingFace via the [HuggingFace Inference API](https://huggingface.co/docs/api-inference/index) (serverless) or dedicated [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index).
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="huggingface/mistralai/Mistral-7B-Instruct-v0.3"
LLM_API_KEY="hf_..."
```
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="huggingface/mistralai/Mistral-7B-Instruct-v0.3"
LLM_ENDPOINT="https://..aws.endpoints.huggingface.cloud/v1/"
LLM_API_KEY="hf_..."
```
**Installation**: Install the HuggingFace extra to enable the HuggingFace tokenizer used for chunking:
```bash theme={null}
pip install cognee[huggingface]
```
**Model names**: Use the full HuggingFace model repo ID after the `huggingface/` prefix (e.g., `huggingface/mistralai/Mixtral-8x7B-Instruct-v0.1`). Not all models on HuggingFace support the text generation inference API — check the model card for compatibility. The model is routed through [LiteLLM](https://docs.litellm.ai/docs/providers/huggingface).
Run models locally with LM Studio for privacy and cost control.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="lm_studio/magistral-small-2509"
LLM_ENDPOINT="http://127.0.0.1:1234/v1"
LLM_API_KEY="."
LLM_INSTRUCTOR_MODE="json_schema_mode"
```
**Installation**: Install LM Studio from [lmstudio.ai](https://lmstudio.ai/) and download your desired model from
LM Studio's interface.
Load your model, start the LM Studio server, and Cognee will be able to connect to it.
**Set up instructor mode**: `LLM_INSTRUCTOR_MODE` controls how Cognee asks the model for structured output. LM Studio models often work best with `json_schema_mode`. For more detail, see [LLM Instructor Modes](#llm-instructor-modes) below and [Structured Output Backends](/setup-configuration/structured-output-backends).
### Complete `.env` (LLM + embeddings on one LM Studio server)
Configure the embedding side too, or embeddings still default to OpenAI — succeeding silently against api.openai.com if `OPENAI_API_KEY` is set in your environment, or failing on the missing key. Load both a chat model and an embedding model in LM Studio, then point Cognee at the same base URL:
```dotenv theme={null}
# LLM — routed through LiteLLM, so the model needs the lm_studio/ prefix
LLM_PROVIDER="custom"
LLM_MODEL="lm_studio/magistral-small-2509"
LLM_ENDPOINT="http://127.0.0.1:1234/v1"
LLM_API_KEY="."
LLM_INSTRUCTOR_MODE="json_schema_mode"
# Embeddings — talks to /v1/embeddings directly, model id used verbatim
EMBEDDING_PROVIDER="openai_compatible"
EMBEDDING_MODEL="text-embedding-nomic-embed-text-v1.5"
EMBEDDING_ENDPOINT="http://127.0.0.1:1234/v1"
EMBEDDING_API_KEY="."
EMBEDDING_DIMENSIONS="768"
```
Both endpoints are the **base** URL ending in `/v1` — not `/v1/chat/completions` or `/v1/embeddings`. `LLM_API_KEY`/`EMBEDDING_API_KEY` are placeholders LM Studio does not validate. Set `EMBEDDING_DIMENSIONS` to your embedding model's real output size (`768` for `nomic-embed-text-v1.5`), otherwise it falls back to `3072` and the first vector-store write fails with a shape mismatch.
The embedding half can also be routed through LiteLLM with `EMBEDDING_PROVIDER="custom"` and an `lm_studio/`-prefixed model — see [Embedding Providers → LM Studio](/setup-configuration/embedding-providers#lm-studio-local) and [Valid EMBEDDING\_PROVIDER values and endpoint URL forms](/setup-configuration/embedding-providers#valid-embedding_provider-values-and-endpoint-url-forms) for when to pick which.
Run models locally using [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) for full offline inference.
Cognee supports two setup modes:
* **Local mode** — Load a `.gguf` model directly in-process
* **Server mode** — Connect to a running `llama-cpp-python` server over HTTP
**Installation**: Install the required dependency:
```bash theme={null}
pip install cognee[llama-cpp]
```
**Choosing a mode**: Use local mode for the simplest setup with no separate server process. Use server mode if you want to share one model across multiple processes or run the model on another machine.
Load a GGUF model file directly. No server setup required.
```dotenv theme={null}
LLM_PROVIDER="llama_cpp"
LLAMA_CPP_MODEL_PATH="/path/to/your/model.gguf"
# Optional: context window size (default: 2048)
LLAMA_CPP_N_CTX=4096
# Optional: GPU layers to offload (default: 0 = CPU only, -1 = all layers on GPU)
LLAMA_CPP_N_GPU_LAYERS=35
# Optional: chat format (default: chatml)
LLAMA_CPP_CHAT_FORMAT="chatml"
```
**GPU acceleration**: Set `LLAMA_CPP_N_GPU_LAYERS=-1` to offload all layers to GPU, or set a positive integer to offload a specific number of layers. Leave it at `0` for CPU-only inference.
**Concurrency**: In local in-process mode the model is loaded once and shared across calls. Because the underlying `llama_cpp.Llama` instance is not thread-safe, Cognee serializes concurrent structured-output calls (such as the per-chunk extraction that `cognify()` fans out) on that single instance. This means in-process requests are processed one at a time rather than in parallel; if you need parallel decoding, run a `llama-cpp-python` server and use **Server Mode** instead.
Connect to a running `llama-cpp-python` server. Start the server separately:
```bash theme={null}
python -m llama_cpp.server --model /path/to/your/model.gguf --port 8000
```
Then configure Cognee to connect to it:
```dotenv theme={null}
LLM_PROVIDER="llama_cpp"
LLM_ENDPOINT="http://localhost:8000/v1"
LLM_API_KEY="."
LLM_MODEL="your-model-name"
```
Use any OpenAI-compatible endpoint — OpenRouter, vLLM, a company-internal gateway, or other services.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="openai/"
LLM_ENDPOINT="https:///v1"
LLM_API_KEY=""
# Optional: fallback provider for content policy violations
# FALLBACK_MODEL=openrouter/openai/gpt-4o-mini
# FALLBACK_ENDPOINT=https://openrouter.ai/api/v1
# FALLBACK_API_KEY=or-...
```
See [Fallback Provider](#fallback-provider) in Advanced Options for full details.
**Custom Provider Prefixes**: When using `LLM_PROVIDER="custom"`, you must include the correct provider prefix in your model name. Cognee forwards requests to [LiteLLM](https://docs.litellm.ai/docs/providers), which uses these prefixes to route requests correctly.
`LLM_PROVIDER="custom"` is **required** for these recipes, not just conventional: Cognee cannot infer a provider from a prefix it has no adapter for, so omitting the line raises `ProviderNotDeducibleError`. See [Provider Inference](#provider-inference).
Common prefixes include:
* `hosted_vllm/` — vLLM servers
* `openrouter/` — OpenRouter
* `lm_studio/` — LM Studio
* `openai/` — OpenAI-compatible APIs
See the [LiteLLM providers documentation](https://docs.litellm.ai/docs/providers) for the full list of supported prefixes.
**Any OpenAI-compatible model works** — the providers listed on this page are examples, not an allowlist. Cognee does not validate `LLM_MODEL` against a set of known models: the value is passed through to LiteLLM, which forwards the request to whatever `LLM_ENDPOINT` you configure. A model that was released after your Cognee version, or one served only from a private gateway, is configured with the same template shown at the top of this section.
**Compatibility requirements**
* **OpenAI-compatible chat completions**: the endpoint must expose a `/v1/chat/completions` route that accepts `system` and `user` messages. `LLM_ENDPOINT` is the base URL (usually ending in `/v1`).
* **Exact model id**: whatever follows the LiteLLM prefix must be the id your server accepts — typically the id returned by the endpoint's `/v1/models`.
* **Structured output**: graph extraction requires the model to return JSON matching a schema. The `custom` provider defaults to the `json_mode` [instructor mode](#llm-instructor-modes); if extraction returns malformed JSON, try `tool_call` or `json_schema_mode`. Weaker models may produce lower-quality graphs even when the transport works — see [Structured Output Backends](/setup-configuration/structured-output-backends).
* **An API key value**: `custom` always sends one, so set `LLM_API_KEY="."` if your server does not authenticate.
* **Embeddings are separate**: the model does not need to serve embeddings — configure those independently under [Embedding Providers](/setup-configuration/embedding-providers).
**Token limits for unknown models**: a model missing from LiteLLM's registry has no known output limit, so your `LLM_MAX_COMPLETION_TOKENS` is used as-is — lower it if the model's real output limit is smaller. See [Max Completion Tokens](#max-completion-tokens) under Advanced Options for how the ceiling is applied.
Below are examples for common providers and patterns:
Use DeepSeek's models for reasoning and chat via their OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="deepseek/deepseek-chat"
LLM_ENDPOINT="https://api.deepseek.com/v1"
LLM_API_KEY="sk-..."
```
Get your API key from [platform.deepseek.com](https://platform.deepseek.com/api_keys). The `deepseek/` prefix tells LiteLLM to route to the DeepSeek API.
**Popular DeepSeek models** (use with the `deepseek/` prefix):
* `deepseek/deepseek-chat` — DeepSeek-V3 (general chat and instruction following)
* `deepseek/deepseek-reasoner` — DeepSeek-R1 (chain-of-thought reasoning)
**Structured output**: DeepSeek's API is OpenAI-compatible, so the default `json_mode` for `custom` providers works well. If you encounter issues with structured output, try setting `LLM_INSTRUCTOR_MODE="tool_call"`.
Use Moonshot AI's Kimi models via their OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="moonshot/moonshot-v1-32k"
LLM_ENDPOINT="https://api.moonshot.cn/v1"
LLM_API_KEY="sk-..."
```
Get your API key from [platform.moonshot.cn](https://platform.moonshot.cn/console/api-keys). The `moonshot/` prefix tells LiteLLM to route to the Moonshot AI API.
**Available Kimi models** (use with the `moonshot/` prefix):
* `moonshot/moonshot-v1-8k` — 8k context window
* `moonshot/moonshot-v1-32k` — 32k context window
* `moonshot/moonshot-v1-128k` — 128k context window (for long documents)
Use [OpenRouter](https://openrouter.ai) to access hundreds of models from a single API endpoint.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="openrouter/deepseek/deepseek-r1"
LLM_ENDPOINT="https://openrouter.ai/api/v1"
LLM_API_KEY="sk-or-..."
```
Get your API key from [openrouter.ai/keys](https://openrouter.ai/keys). Browse all available models at [openrouter.ai/models](https://openrouter.ai/models) — prefix the model slug with `openrouter/`.
**Example models** (use with the `openrouter/` prefix):
* `openrouter/deepseek/deepseek-r1` — DeepSeek R1 via OpenRouter
* `openrouter/openai/gpt-4o-mini` — GPT-4o Mini via OpenRouter
**Model ids change — confirm before you copy.** OpenRouter adds and retires models continuously, and the free (`:free`) tier rotates fastest of all, so a slug that worked last month may return a model-not-found error today. Check the live catalogue rather than trusting an example:
```bash theme={null}
curl -s https://openrouter.ai/api/v1/models | jq -r '.data[].id'
```
**Embeddings need their own configuration.** These variables set only the LLM. If you leave `EMBEDDING_*` untouched it stays on the OpenAI defaults, so the LLM connects fine and ingestion fails later at embedding time on a missing or invalid OpenAI key. OpenRouter serves embedding models too — see [OpenRouter embeddings](/setup-configuration/embedding-providers#custom-providers) — or point `EMBEDDING_*` at OpenAI or a local provider.
Use DeepInfra to access open-source models via their OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="deepinfra/meta-llama/Meta-Llama-3-8B-Instruct"
LLM_ENDPOINT="https://api.deepinfra.com/v1/openai"
LLM_API_KEY=""
```
Find your model name in the [DeepInfra model catalog](https://deepinfra.com/models). The `deepinfra/` prefix tells LiteLLM to route to DeepInfra.
Any internal LLM server that exposes an OpenAI-compatible REST API (e.g., a corporate vLLM deployment, internal TGI server, or private OpenRouter proxy) can be used with the `custom` provider.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="openai/"
LLM_ENDPOINT="https://llm.internal.example.com/v1"
LLM_API_KEY=""
```
The model prefix you use (`openai/`, `hosted_vllm/`, etc.) determines which LiteLLM adapter handles the request. For most OpenAI-compatible servers, `openai/` works best. Set `LLM_API_KEY` to whatever bearer token your server requires (use `.` if no auth is needed).
Use vLLM for high-performance model serving with OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="hosted_vllm/"
LLM_ENDPOINT="https://your-vllm-endpoint/v1"
LLM_API_KEY="."
```
**Example with Gemma:**
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="hosted_vllm/gemma-3-12b"
LLM_ENDPOINT="https://your-vllm-endpoint/v1"
LLM_API_KEY="."
```
**Important**: The `hosted_vllm/` prefix is required for LiteLLM to correctly route requests to your vLLM server. The model name after the prefix should match the model ID returned by your vLLM server's `/v1/models` endpoint.
To find the correct model name, see [their documentation](https://docs.litellm.ai/docs/providers/vllm).
When Cognee runs **as an MCP server** (`cognee-mcp`) inside a host that grants the MCP `sampling` capability, `LLM_PROVIDER="mcp-sampling"` delegates completions to the host's own model through `sampling/createMessage`. No `LLM_API_KEY` is required.
```dotenv theme={null}
LLM_PROVIDER="mcp-sampling"
# LLM_MODEL is a preference hint only — the host chooses the actual model
LLM_MODEL="host-default"
```
**Preconditions**: This provider only works while Cognee is running as an MCP server inside a host process that granted the `sampling` capability to the client. If Cognee is not running under such a host — or the host did not grant sampling — the adapter fails closed with `MCPSamplingUnavailableError` before issuing any request. Treat that error as a configuration/capability issue: set `LLM_PROVIDER` to a provider with credentials, or run inside a sampling-granting host.
**Host support varies.** Not every MCP host grants the `sampling` capability. For example, as of early 2026 Claude Code does not yet grant sampling ([anthropics/claude-code#1785](https://github.com/anthropics/claude-code/issues/1785)). Check your host's MCP documentation.
**Completions only.** MCP sampling covers text completions — it does not provide embeddings, audio transcription, or image description. Because vector search needs embeddings, you must still configure an [embedding provider](/setup-configuration/embedding-providers) (audio transcription returns nothing and image description raises `NotImplementedError`).
**Structured output.** The MCP protocol returns free text only, so Cognee produces structured output by embedding the response model's JSON Schema in the prompt and running a bounded validate/repair loop (up to 5 attempts) before raising an error. Plain-string responses pass through unchanged.
Background tasks (such as the `cognify` tasks launched from within a request) inherit the host MCP session automatically via the SDK's per-request context, so no changes to `cognee-mcp` server code are needed.
## Advanced Options
LLM configuration is read at runtime from your environment/`.env` — it is **not** stored per dataset. Changing `LLM_PROVIDER` / `LLM_MODEL` therefore works fine on top of a dataset you have already processed; nothing about the existing data blocks the switch.
**What is not affected.** Already-processed data is left untouched: the entities and relationships in your [graph store](/setup-configuration/graph-stores) and the embeddings in your [vector store](/setup-configuration/vector-stores) are neither re-computed nor invalidated. Cognee does not re-run past extraction, and vectors depend on the [embedding model](/setup-configuration/embedding-providers), not the LLM, so [recall](/core-concepts/main-operations/recall) over existing data keeps working.
**What is affected.** The new LLM applies only to *future* work:
* Subsequent [`cognify`](/core-concepts/main-operations/legacy-operations/cognify) / [`memify`](/core-concepts/main-operations/legacy-operations/memify) runs — new data is extracted and summarized with the new model.
* Query-time reasoning during [`search`](/core-concepts/main-operations/legacy-operations/search) (e.g. `GRAPH_COMPLETION`) — answers are generated by the new model over the *existing* graph and vectors.
This means a graph can mix output from different LLMs: nodes written by the old model stay as-is, and only newly cognified data reflects the new one. If you want the whole dataset to reflect the new model's extraction quality, re-process it: run [`cognify`](/core-concepts/main-operations/legacy-operations/cognify) with `incremental_loading=False` to force a full reprocess, or empty the dataset, re-[`add`](/core-concepts/main-operations/legacy-operations/add) the source data, and run `cognify` again. Simply re-running `cognify` is not enough — it skips already-processed data by default.
Changing the **embedding** model is different: existing vectors were written with the old embeddings and become inconsistent with new ones. To change embeddings on an existing dataset you must re-process it — run `cognify` with `incremental_loading=False`, or delete and re-add the data — and if the new model has a different `EMBEDDING_DIMENSIONS`, remove the existing vector collections first (e.g. with [`prune`](/python-api/prune), which wipes **all** datasets). See [Embedding Providers](/setup-configuration/embedding-providers).
By default Cognee uses a single model — the base `LLM_*` settings — for every stage of the pipeline. You can optionally route individual stages to different models or providers by setting stage-specific env var groups. Because **extraction runs once per chunk and typically dominates token spend**, it is often worth routing a cheaper or local model there while keeping a stronger model for summarization and query-time reasoning.
**Stages and their env groups**
| Env group | Stage it controls |
| --------------------- | ------------------------------------------------------------------ |
| `LLM_EXTRACTION_*` | Entity/relationship extraction during `cognify()` (runs per chunk) |
| `LLM_SUMMARIZATION_*` | Text summarization during `cognify()` |
| `LLM_QUERY_*` | Query-time completion during `search()` |
Each group accepts the same fields as the base `LLM_*` config, with the stage name in place of the leading `LLM`:
* `LLM__MODEL`
* `LLM__PROVIDER`
* `LLM__ENDPOINT`
* `LLM__API_KEY`
* `LLM__API_VERSION`
**Fallback to base config**: any stage field you leave unset (empty or absent) falls back to the corresponding base `LLM_*` value, so you only set what you want to override. If you set no stage overrides at all, the effective config is identical to a single-model setup — **default single-model behavior is unchanged**.
**Only those five fields are overridden.** Everything else (temperature, `LLM_MAX_COMPLETION_TOKENS`, instructor mode, the rate-limit settings) is inherited from the base config — there are no `LLM__*` equivalents for them. One default is re-derived rather than inherited: unless you set `LLM_RATE_LIMIT_REQUESTS` yourself, a stage routed to a local inference server resolves the [lower local RPM budget](#rate-limiting) of `10` instead of the `60` derived for a cloud base provider — in the example below, extraction resolves `10` while summarization and query keep `60`.
**Example** — route extraction to a local Ollama model while summarization and query keep using OpenAI:
```dotenv theme={null}
# Base config (used for any stage field left unset)
LLM_PROVIDER="openai"
LLM_MODEL="openai/gpt-5-mini"
LLM_API_KEY="sk-..."
# Extraction → local Ollama (cheap, high-volume)
LLM_EXTRACTION_MODEL="ollama_chat/llama3.1"
LLM_EXTRACTION_PROVIDER="ollama"
LLM_EXTRACTION_ENDPOINT="http://localhost:11434"
LLM_EXTRACTION_API_KEY=""
# Summarization and query keep the base model (set explicitly if you want a different one)
LLM_SUMMARIZATION_MODEL="openai/gpt-5-mini"
LLM_SUMMARIZATION_PROVIDER="openai"
LLM_QUERY_MODEL="openai/gpt-5-mini"
LLM_QUERY_PROVIDER="openai"
```
The stage-level budget shapes that stage's *resolved config* only — the limiter that paces dispatch is built once, process-wide, from the base configuration, so if you route high-volume extraction to a local server, set `LLM_RATE_LIMIT_REQUESTS` explicitly to what that server can absorb. The re-derivation is also one-way: a *local* base provider resolves `10`, and a stage routed from there to a cloud provider keeps `10` rather than returning to `60`.
No SDK or pipeline call signatures change when you enable per-stage routing. Each stage transparently gets its own cached client derived from its effective config, so concurrent stages can use different models safely.
When using the Instructor structured-output framework (opt-in via `STRUCTURED_OUTPUT_FRAMEWORK=instructor`; the default is `litellm_native`), Cognee instructs the model to return structured data in a specific way. The `LLM_INSTRUCTOR_MODE` environment variable controls which strategy is used.
Each provider has a built-in default that matches its API capabilities. Override it only when the default doesn't work for your specific model.
**Available modes:**
| Mode | Description | When to use |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `json_schema_mode` | Passes the full JSON Schema of the expected output in the request and enforces strict schema compliance. | OpenAI models that support the `response_format` / structured-output feature (e.g. GPT-4o). Also works well with Bedrock and with Ollama 0.5+, which can enforce a JSON Schema server-side. |
| `json_mode` | Instructs the model to return any valid JSON object. Instructor then validates and coerces it to the target schema. | Gemini, Generic/Custom endpoints, and any model that supports `response_format: json_object` but not strict schema enforcement. |
| `anthropic_tools` | Uses Anthropic's native tool-calling API to extract structured data. | Anthropic Claude models only. Leverages first-class tool-use support for reliable extraction. |
| `mistral_tools` | Uses Mistral's native tool-calling API to extract structured data. | Mistral models only. Mirrors the OpenAI function-calling interface provided by Mistral. |
| `tool_call` | Uses the generic OpenAI-style function/tool-calling API to define the schema as a callable tool. | OpenAI-compatible APIs that support function calling but not strict JSON schema output. |
| `md_json` | Asks the model to return JSON wrapped in a Markdown code block. Instructor extracts the block and validates it. | Models that reliably format code blocks but may not support `json_mode` (e.g. some self-hosted models). |
**Per-provider defaults (from source code):**
| Provider (`LLM_PROVIDER`) | Default mode |
| ------------------------------------ | ------------------ |
| `openai` (and Azure OpenAI) | `json_schema_mode` |
| `anthropic` | `anthropic_tools` |
| `gemini` | `json_mode` |
| `bedrock` | `json_schema_mode` |
| `mistral` | `mistral_tools` |
| `ollama` | `json_schema_mode` |
| `custom` (generic OpenAI-compatible) | `json_mode` |
**Example — override the mode:**
```dotenv theme={null}
LLM_INSTRUCTOR_MODE="json_schema_mode"
```
Override the default only when the model you are using requires a different mode. For example, LM Studio models typically need `json_schema_mode` even though the `custom` provider defaults to `json_mode`.
**Ollama's default changed to `json_schema_mode`.** It was previously `json_mode`, which sent `response_format: {"type": "json_object"}` — that asks for *some* JSON and passes the Pydantic schema to the model as prompt text only, so validity is checked after the fact. With `json_schema_mode`, the schema is sent as a decoder constraint that Ollama enforces, which cuts first-attempt validation failures and the resulting `InstructorRetryException` on local models (measured 2/5 → 5/5 valid first attempts with `llama3.1:8b` and `max_retries=0`).
Ollama has supported JSON-schema structured outputs since **0.5**. If you run an older Ollama, set the previous behavior back explicitly:
```dotenv theme={null}
LLM_INSTRUCTOR_MODE="json_mode"
```
Control the randomness of LLM responses with the `LLM_TEMPERATURE` and `LLM_SEED` environment variables.
| Variable | Default | Description |
| ----------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `LLM_TEMPERATURE` | unset — provider default applies (`0.0` on local inference servers) | Sampling temperature. `0.0` = deterministic / focused output. Higher values (e.g. `0.7`–`1.0`) produce more varied, creative responses. |
| `LLM_SEED` | unset — no seed sent | Sampling seed for reproducible outputs. Provider support varies. |
**Both are sent when you set them.** If you leave `LLM_TEMPERATURE` unset, Cognee sends no `temperature` at all and the provider's own default applies (OpenAI's is `1.0`) — it does not fall back to `0.0`. Same for `LLM_SEED`. Local inference servers are the one exception for temperature; see the note below.
```dotenv theme={null}
LLM_TEMPERATURE=0.0
LLM_SEED=42
```
**When to adjust**: setting `LLM_TEMPERATURE=0.0` is recommended for knowledge-graph extraction because it produces consistent, structured output; add `LLM_SEED` on top when you need runs to be reproducible. Raise the temperature only if you need more variety in generated text (e.g. conversational responses or creative summarisation).
Under the hood, both values are merged into `LLM_ARGS`, the provider kwargs Cognee sends with each call. A `temperature` or `seed` key given directly in `LLM_ARGS` wins over `LLM_TEMPERATURE` / `LLM_SEED`, so existing `LLM_ARGS='{"temperature": 0}'` setups keep working unchanged.
**Deterministic default on local inference servers.** When `LLM_TEMPERATURE` is not set explicitly, Ollama, llama.cpp, and LM Studio are sent `temperature: 0.0` rather than nothing — they accept the field, and leaving it out meant extraction ran at whatever the model itself defaults to (`1.0` for several Ollama models). vLLM is treated as a regular provider here and still gets no temperature when the variable is unset, as do all hosted providers. The precedence is unchanged, so both escape hatches still work: set `LLM_TEMPERATURE` to the value you want, or put a `temperature` key in `LLM_ARGS` to override the dedicated field entirely.
gpt-5 models — including the default `openai/gpt-5-mini` — reject any temperature other than their own default. Setting `LLM_TEMPERATURE` to something else (`0.0`, for instance) on a gpt-5 model makes every generation call fail. That restriction is why Cognee sends nothing on hosted providers unless you opt in: leave it unset, or set it only alongside a model that accepts custom temperatures. It does not apply to the local servers above, which is why they are exempt from that rule.
`LLM_MAX_COMPLETION_TOKENS` sets the maximum number of tokens an LLM call may **generate** per request (passed to the provider as `max_tokens`/`max_completion_tokens`).
| Variable | Default | Description |
| --------------------------- | ------- | ------------------------------------------------------------------------ |
| `LLM_MAX_COMPLETION_TOKENS` | `16384` | Per-request output-token ceiling, and an input to automatic chunk sizing |
**Observable impact:**
* **Truncation.** If extraction or summarisation responses are larger than this ceiling, the provider stops generating mid-response. With [structured output](/setup-configuration/structured-output-backends) this surfaces as malformed/incomplete JSON or, with some local models, HTTP 500 errors. Raise the value if you see truncated output.
* **Effective value is clamped.** When the model is in [LiteLLM's](https://docs.litellm.ai/docs/providers) model registry, Cognee uses `min(model limit from LiteLLM's registry, LLM_MAX_COMPLETION_TOKENS)`. Setting it far above the registry limit has no effect — "higher" is not automatically "better".
* **Chunk size, cost and latency.** Extraction chunks are sized as `min(EMBEDDING_MAX_COMPLETION_TOKENS, LLM_MAX_COMPLETION_TOKENS // 2)` — so this value also caps how much text goes into each `cognify` chunk. A larger value means fewer, larger chunks (fewer LLM calls but more tokens per call); a smaller value means more, smaller chunks (more calls, finer-grained extraction). See [Chunkers](/core-concepts/further-concepts/chunkers) for how chunk size shapes the graph.
**Tuning guidance:** the default `16384` is a good starting point for cloud models. Lower it for **local models with a small context window** so chunks fit (see the Ollama [`num_ctx`](#ollama-local) note). Raise it only if your model supports a larger output window and you observe truncated extraction.
Control client-side throttling for LLM calls to manage API usage and costs.
**The limiter starts off, but switches itself on when your provider shows signs of overload.** Cognee dispatches at full speed until a request comes back rate limited, times out, or returns HTTP 429/503/529 — then it logs one warning and paces every dispatch with the RPM budget below for a 15-minute cooldown. Set `LLM_RATE_LIMIT_ENABLED="true"` to pace from the first request instead, or `AUTO_RATE_LIMIT="false"` to stay unbounded no matter what the provider reports.
**Defaults:**
| Variable | Default | Meaning |
| ------------------------- | --------------------------------------- | --------------------------------------------------------------------------- |
| `AUTO_RATE_LIMIT` | `true` | Turn the limiter on automatically once the provider shows overload evidence |
| `LLM_RATE_LIMIT_ENABLED` | `false` | Pace every call from the start, without waiting for overload evidence |
| `LLM_RATE_LIMIT_REQUESTS` | `60` (`10` for local inference servers) | Max requests per interval |
| `LLM_RATE_LIMIT_INTERVAL` | `60` | Interval in seconds |
The cloud defaults (60 requests / 60 seconds) allow 1 request/second on average. Adjust both values to match your provider's tier limit.
**Lower default budget for local inference servers.** When `LLM_RATE_LIMIT_REQUESTS` is not set explicitly, Ollama, llama.cpp, and LM Studio get `10` requests per interval instead of `60` — they process requests near-serially, so the cloud default would still flood them. vLLM is treated as a regular provider: continuous batching absorbs concurrency like a cloud endpoint, so it keeps the `60` default. An explicitly configured `LLM_RATE_LIMIT_REQUESTS` always wins, whatever the provider.
**How auto rate limiting behaves:**
* **What counts as evidence**: a rate-limit error, a timeout (how overwhelmed local servers usually surface — they never send rate limits), or HTTP `429`, `503`, or `529`. Evidence is looked for through the whole exception cause chain, so a provider error wrapped by Instructor still counts. Slow responses on their own are not evidence.
* **Cooldown window**: the limiter stays engaged for 900 seconds (15 minutes). Further evidence inside the window extends it; once the window lapses quietly, behavior returns to whatever you configured.
* **One warning per episode**: the first piece of evidence in an episode logs a warning naming the cause and the budget being applied; evidence inside an active window extends it silently. A fresh episode after a quiet cooldown warns again.
* **Opting out**: `AUTO_RATE_LIMIT="false"` disables the automatic engagement entirely. `LLM_RATE_LIMIT_ENABLED="true"` keeps the limiter on from the start, independent of the auto behavior.
**How it works:**
* **Client-side limiter**: Cognee paces outbound LLM calls before they reach the provider
* **Moving window**: Spreads allowance across the time window for smoother throughput
* **Per-process scope**: In-memory limits don't share across multiple processes/containers
* **Retries are paced too**: adapters enter the limiter inside their retry loop, so retried attempts are throttled alongside first attempts
* **Auto-applied**: Works with all providers (OpenAI, Gemini, Anthropic, Ollama, Custom)
**Sizing guidance:**
Set `LLM_RATE_LIMIT_REQUESTS` to your provider's RPM (requests per minute) limit, and `LLM_RATE_LIMIT_INTERVAL` to `60`. To leave headroom, use \~80–90% of the advertised limit. Check your provider's dashboard for your current tier limits.
Each `cognify()` call issues multiple LLM requests (entity extraction, summarization, etc.) per document chunk — plan for several requests per chunk, not one.
**Example configurations for common provider tiers**
These examples target chat/completions-style LLM endpoints, such as OpenAI models like `gpt-4o-mini`.
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="450"
LLM_RATE_LIMIT_INTERVAL="60"
```
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="4500"
LLM_RATE_LIMIT_INTERVAL="60"
```
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="45"
LLM_RATE_LIMIT_INTERVAL="60"
```
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="13"
LLM_RATE_LIMIT_INTERVAL="60"
```
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="60"
LLM_RATE_LIMIT_INTERVAL="60"
```
Always verify your exact tier limits in your provider's dashboard — limits vary by model, tier, and region. The examples above are approximations for common tiers and may change.
Cognee supports a primary-plus-fallback model configuration that automatically retries a failed request against a secondary provider. This is useful when your primary provider may reject certain content, and you want a fallback to handle those cases gracefully.
**When the fallback triggers**
The fallback is invoked only on **content policy violations** from the primary provider:
* `ContentFilterFinishReasonError` — the provider's output filter blocked the response
* `ContentPolicyViolationError` — the request was rejected for policy reasons
* `InstructorRetryException` containing "content management policy"
The fallback does **not** activate for network errors, rate limits, or authentication failures.
**Supported providers**
Fallback is available when `LLM_PROVIDER` is set to `openai` or `custom`. Other providers (Anthropic, Gemini, Mistral, Bedrock, Ollama) do not currently support the fallback chain.
**Configuration**
Set these three variables alongside your primary LLM configuration:
```dotenv theme={null}
# Primary provider
LLM_PROVIDER="openai"
LLM_MODEL="openai/gpt-4o-mini"
LLM_API_KEY="sk-..."
# Fallback provider (used only on content policy violations)
FALLBACK_MODEL="openrouter/openai/gpt-4o-mini"
FALLBACK_ENDPOINT="https://openrouter.ai/api/v1"
FALLBACK_API_KEY="or-..."
```
For `LLM_PROVIDER="custom"`, all three fallback variables (`FALLBACK_MODEL`, `FALLBACK_ENDPOINT`, `FALLBACK_API_KEY`) must be set. If any is missing, Cognee raises a `ContentPolicyFilterError` instead of falling back.
For `LLM_PROVIDER="openai"`, only `FALLBACK_MODEL` and `FALLBACK_API_KEY` are required. If set, `FALLBACK_ENDPOINT` is now forwarded to the OpenAI adapter and routes the fallback request to that base URL; if omitted, the fallback request uses the default OpenAI endpoint.
**Variable reference**
| Variable | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------ |
| `FALLBACK_MODEL` | Model identifier for the fallback provider (use LiteLLM prefix format, e.g. `openrouter/openai/gpt-4o-mini`) |
| `FALLBACK_ENDPOINT` | Base URL for the fallback provider's API (required for `custom`, optional for `openai`) |
| `FALLBACK_API_KEY` | API key for the fallback provider |
Structured-output LLM calls (`acreate_structured_output`, used internally for entity extraction, summarization, and other graph-building steps) are wrapped in a shared retry policy that retries transient failures with exponential backoff.
**How long a failing call persists**
A call is allowed to give up only once **both** of these floors are met:
| Floor | Value | Meaning |
| -------------------- | ------- | ------------------------------------------------------------------- |
| Minimum attempts | `2` | At least two attempts are made before failing. |
| Minimum elapsed time | `~240s` | At least \~240 seconds of wall-clock time must pass before failing. |
Because both conditions must hold, a call against an unstable or rate-limited provider can keep retrying for up to a few minutes before it finally errors out. Backoff between attempts is exponential with jitter (starting around 8 seconds, capped near 128 seconds).
These floors are internal defaults shared across the OpenAI, Azure OpenAI, Anthropic, Gemini, Mistral, Ollama, Llama.cpp, Custom, and BAML structured-output paths. They are **not** environment-configurable.
Bedrock uses a separate retry path: its structured-output adapter relies on the Bedrock rate-limit/sleep retry wrapper and Instructor's Bedrock retry setting instead of the shared `~240s` retry floor.
Some errors are treated as non-transient and are **not** retried — they fail immediately: authentication errors, model-not-found errors, cancellations, payment/budget exhaustion, and quota/billing exhaustion. That includes the shared retry paths for OpenAI, Azure OpenAI, Anthropic, Gemini, Mistral, Ollama, Llama.cpp, Custom, and BAML, so interrupted jobs and worker shutdowns stop promptly rather than waiting out the backoff window. Bedrock uses a separate retry path, but cancellations still unwind immediately there as well.
**Quota / billing exhaustion is terminal.** When a provider reports that its quota or billing limit is exhausted, retrying cannot help, so the call fails fast instead of spinning through the retry window. The raw provider error is converted at the single `acreate_structured_output` choke point into an actionable `LLMQuotaExceededError` (provider- and framework-agnostic). The following provider wordings are classified as terminal:
| Pattern | Provider |
| --------------------------- | ----------------------------------------------- |
| `insufficient_quota` | OpenAI / Azure OpenAI (billing quota exhausted) |
| `quota_exceeded` | Generic provider quota-exhaustion code |
| `billing hard limit` | OpenAI (monthly hard limit reached) |
| `credit balance is too low` | Anthropic (prepaid credits exhausted) |
| `out of credits` | Generic |
Transient **per-minute rate limits** stay retryable. The patterns above are deliberately narrow: the bare phrase "exceeded your current quota" is intentionally **not** matched, because Gemini free tier uses it for recoverable `RESOURCE_EXHAUSTED` limits (OpenAI's terminal case is still caught via `insufficient_quota`). Monitoring and alerting should treat `LLMQuotaExceededError` as a terminal condition and respond by checking the provider billing/quota dashboard, raising the limit, or switching credentials — not by retrying.
**Operational note**: when a provider is flaky, expect higher tail latency and additional API calls (and therefore cost) while retries play out. This persistent retry improves resilience for transient failures but does not mask genuine misconfiguration such as a bad API key.
`LLM_ENDPOINT` overrides the base URL Cognee uses to reach the LLM. Use it to point at an Azure deployment, a local server, an OpenAI-compatible proxy, or a company-internal gateway. For routing all outbound traffic through a corporate HTTP proxy without rewriting the endpoint, use the standard `HTTPS_PROXY` / `HTTP_PROXY` environment variables.
**Per-provider `LLM_ENDPOINT` semantics**
| `LLM_PROVIDER` | How `LLM_ENDPOINT` is used | Required? |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `openai` | Passed to LiteLLM as `api_base`. Omit to use OpenAI's default (`https://api.openai.com/v1`). Set to point at a compatible gateway or proxy. | Optional |
| `azure` | Azure resource endpoint (e.g., `https://.openai.azure.com`). The deployment is selected by `LLM_MODEL`. | Required |
| `gemini` | Passed to LiteLLM as `api_base`. Omit to use the provider's default. | Optional |
| `mistral` | `LLM_ENDPOINT` is currently not used for generation; Cognee uses the default Mistral provider endpoint. | Not applicable |
| `ollama` | OpenAI-compatible endpoint of your Ollama server (typically `http://localhost:11434/v1`). | Required |
| `custom` | Base URL of your OpenAI-compatible server (vLLM, OpenRouter, LM Studio, internal gateway, etc.). | Required |
| `llama_cpp` | Required only in server mode (URL of the `llama-cpp-python` server). Ignored in local in-process mode. | Server mode only |
| `anthropic` | Not read. Anthropic's SDK has its own internal base URL. To route through a proxy, use `HTTPS_PROXY`. | Not applicable |
| `bedrock` | Not read. Use `AWS_BEDROCK_RUNTIME_ENDPOINT` to override the Bedrock endpoint. | Not applicable |
**Routing through a corporate HTTP/HTTPS proxy**
Cognee's LLM transport is built on the `openai`, `anthropic`, `httpx`, and `litellm` Python clients, all of which honor the standard proxy environment variables. Set them in your shell or `.env` before starting Cognee:
```dotenv theme={null}
HTTPS_PROXY="http://proxy.corp.example.com:8080"
HTTP_PROXY="http://proxy.corp.example.com:8080"
# Optional: hosts that should bypass the proxy
NO_PROXY="localhost,127.0.0.1,.internal.example.com"
```
This is the right approach when the LLM provider's public URL is correct but your network blocks direct egress. No Cognee config change is needed — outbound LLM, embedding, and HTTP loader calls all pick up these variables automatically.
**Troubleshooting "not connected / cannot reach LLM"**
* **`LLM_ENDPOINT` typos** — values are stripped of surrounding quotes, but a missing scheme (`http://` / `https://`) or trailing path segment will surface as a connection error. For OpenAI-compatible endpoints, the URL must end in `/v1` (or whatever the server exposes).
* **Preflight timeout** — Cognee runs a 30s connection test at startup. If your proxy adds latency or your local model is slow to warm up, set `COGNEE_SKIP_CONNECTION_TEST=true` to skip it.
* **Provider mismatch** — if `LLM_ENDPOINT` points at a non-OpenAI server but `LLM_PROVIDER="openai"`, Cognee will hit the wrong route. For OpenAI-compatible third parties, use `LLM_PROVIDER="custom"` with the correct LiteLLM model prefix (see [Custom Providers](#custom-providers) above).
* **TLS interception** — if your corporate proxy uses its own CA, set `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE` to the CA bundle path so Python's HTTP clients trust the proxy certificate.
## Notes
* If `EMBEDDING_API_KEY` is not set, Cognee falls back to `LLM_API_KEY` for embeddings
* Rate limiting helps manage API usage and costs
* Structured output frameworks ensure consistent data extraction from LLM responses
Configure embedding providers for semantic search
Return to setup configuration overview
Set up SQLite or Postgres for metadata storage