cognify()

Transform raw data into a structured knowledge graph

cognee.cognify()

async def cognify(
    datasets: Union[str, list[str], list[UUID]] = None,
    user: User = None,
    graph_model: BaseModel = KnowledgeGraph,
    chunker = TextChunker,
    chunk_size: int = None,
    chunks_per_batch: int = None,
    config: Config = None,
    vector_db_config: dict = None,
    graph_db_config: dict = None,
    run_in_background: bool = False,
    incremental_loading: bool = True,
    custom_prompt: Optional[str] = None,
    data_per_batch: int = 20,
    llm_config: Optional[LLMConfig] = None,
    embedding_config: Optional[EmbeddingConfig] = None,
    dry_run: bool = False,
    raise_on_error: bool = True,
    chunk_attachment: Optional[Literal["direct", "all"]] = None,
    extractor: Optional[Literal["llm", "gliner_demo", "gliner"]] = None,
    summary_method: Optional[Literal["llm", "from_extraction"]] = None,
)

Description

Transform ingested data into a structured knowledge graph.

This is the core processing step in Cognee that converts raw text and documents into an intelligent knowledge graph. It analyzes content, extracts entities and relationships, and creates semantic connections for enhanced search and reasoning.

Prerequisites:

• LLM_API_KEY: Must be configured to extract entities and build the graph with an LLM; extractor="gliner_demo" extracts the graph and the chunk summaries with the local GLiNER demo model instead, and with no key configured at all the auto default resolves to that extractor on its own — see LLM-free extraction with GLiNER
• Data Added: Must have data previously added via cognee.add()
• Vector Database: Must be accessible for embeddings storage
• Graph Database: Must be accessible for relationship storage

Input Requirements:

• Datasets: Must contain data previously added via cognee.add()
• Content Types: Works with any text-extractable content including:
  • Natural language documents
  • Structured data (CSV, JSON)
  • Code repositories
  • Academic papers and technical documentation
  • Mixed multimedia content (with text extraction)

Processing Pipeline:

1. Document Classification: Identifies document types and structures
2. Text Chunking: Breaks content into semantically meaningful segments
3. Entity Extraction: Identifies key concepts, people, places, organizations
4. Relationship Detection: Discovers connections between entities
5. Graph Construction: Builds semantic knowledge graph with embeddings
6. Content Summarization: Creates hierarchical summaries for navigation

Graph Model Customization: The graph_model parameter allows custom knowledge structures:

• Default: General-purpose KnowledgeGraph for any domain
• Custom Models: Domain-specific schemas (e.g., scientific papers, code analysis)
• Ontology Integration: Pass an ontology resolver via config (or set the ONTOLOGY_FILE_PATH environment variable) for predefined vocabularies. An optional "ontology_mode" key in the same ontology_config picks how strictly the ontology is applied — "annotate" (default: enrich only) or "strict" (drop extracted entities the ontology does not ground, and their edges). It overrides the ONTOLOGY_MODE environment variable for this call; unrecognized values warn and fall back to "annotate".

Args: datasets: Dataset name(s) or dataset uuid to process. Processes all available data if None.

• Single dataset: “my_dataset”
• Multiple datasets: [“docs”, “research”, “reports”]
• None: Process all datasets for the user user: User context for authentication and data access. Uses default if None. graph_model: Pydantic model defining the knowledge graph structure. Defaults to KnowledgeGraph for general-purpose processing. chunker: Text chunking strategy (TextChunker, LangchainChunker).
  • TextChunker: Paragraph-based chunking (default, most reliable)
  • LangchainChunker: Recursive character splitting with overlap Determines how documents are segmented for processing. chunk_size: Maximum tokens per chunk. Auto-calculated based on LLM if None. Formula: min(embedding token limit, llm_max_completion_tokens // 2). A value above what the embedding model accepts is lowered to that limit with a warning. Default limits: ~512-8192 tokens depending on models. Smaller chunks = more granular but potentially fragmented knowledge. chunks_per_batch: Number of chunks to be processed in a single batch in Cognify tasks. vector_db_config: Custom vector database configuration for embeddings storage. graph_db_config: Custom graph database configuration for relationship storage. run_in_background: If True, starts processing asynchronously and returns immediately. If False, waits for completion before returning. Background mode recommended for large datasets (>100MB). Use pipeline_run_id from return value to monitor progress. custom_prompt: Optional custom prompt string to use for entity extraction and graph generation. If provided, this prompt will be used instead of the default prompts for knowledge graph extraction. The prompt should guide the LLM on how to extract entities and relationships from the text content. dry_run: If True, return a stage-level estimate of LLM token usage and rough cost without making LLM calls or writing graph results. The estimate covers all data in the selected dataset(s); an incremental run may process fewer items. chunk_attachment: How widely each chunk links into the graph extracted from it. Accepts "direct", "all", or None; omitting it is the same as "direct". Requires a custom DataPoint graph_model. See Chunk attachment. extractor: Which implementation fills the extract-and-summarize step of the standard pipeline. Accepts "llm", "gliner_demo" (alias "gliner"), or None; the explicit argument wins over the GRAPH_EXTRACTOR setting and None resolves to that setting — "auto" by default, which is "llm" with a usable LLM key and "gliner_demo" without. See LLM-free extraction with GLiNER. summary_method: How the standard pipeline writes each chunk’s TextSummary. "llm" makes one LLM call per chunk; "from_extraction" joins the chunk’s own extracted types and relation texts with no LLM call, and a chunk whose extraction produced no relations gets no summary at all. Accepts "llm", "from_extraction", or None; the explicit argument wins over the SUMMARY_METHOD setting ("llm" by default). See Chunk summaries from extraction.

Returns: Union[dict, list[PipelineRunInfo], DryRunEstimate]:

• Blocking mode: Dictionary mapping dataset_id -> PipelineRunInfo with:
  • Processing status (completed/failed/in_progress)
  • Extracted entity and relationship counts
  • Processing duration and resource usage
  • Error details if any failures occurred
• Background mode: List of PipelineRunInfo objects for tracking progress
  • Use pipeline_run_id to monitor status
  • Check completion via pipeline monitoring APIs

Next Steps: After successful cognify processing, use search functions to query the knowledge:

import cognee
from cognee import SearchType

# Process your data into knowledge graph
await cognee.cognify()

# Query for insights using different search types:

# 1. Natural language completion with graph context
insights = await cognee.search(
    "What are the main themes?",
    query_type=SearchType.GRAPH_COMPLETION
)

# 2. Get entity relationships and connections
relationships = await cognee.search(
    "connections between concepts",
    query_type=SearchType.GRAPH_COMPLETION
)

# 3. Find relevant document chunks
chunks = await cognee.search(
    "specific topic",
    query_type=SearchType.CHUNKS
)

Advanced Usage:

# Custom domain model for scientific papers
class ScientificPaper(DataPoint):
    title: str
    authors: List[str]
    methodology: str
    findings: List[str]

await cognee.cognify(
    datasets=["research_papers"],
    graph_model=ScientificPaper,
)

# Ground extraction in an ontology (there is no `ontology_file_path` argument;
# pass a resolver through `config` instead).
from cognee.modules.ontology.rdf_xml.RDFLibOntologyResolver import RDFLibOntologyResolver
from cognee.modules.ontology.ontology_config import Config

config: Config = {
    "ontology_config": {
        "ontology_resolver": RDFLibOntologyResolver(ontology_file="scientific_ontology.owl"),
        # Optional; overrides ONTOLOGY_MODE. "annotate" (default) enriches only,
        # "strict" drops extracted entities the ontology does not ground.
        "ontology_mode": "annotate",
    }
}
await cognee.cognify(datasets=["research_papers"], config=config)

# Background processing for large datasets
run_info = await cognee.cognify(
    datasets=["large_corpus"],
    run_in_background=True
)
# Check status later with run_info.pipeline_run_id

Environment Variables: Required:

• LLM_API_KEY: API key for your LLM provider

Optional (same as add function):

• LLM_PROVIDER, LLM_MODEL, VECTOR_DB_PROVIDER, GRAPH_DATABASE_PROVIDER
• AUTO_RATE_LIMIT: Turn the rate limiter on automatically when the provider shows overload evidence (default: True)
• LLM_RATE_LIMIT_ENABLED: Enable rate limiting from the first request (default: False)
• LLM_RATE_LIMIT_REQUESTS: Max requests per interval (default: 60; 10 for local inference servers)

Optional (provenance ledger — see Provenance ledger):

• PROVENANCE_TRACKING: Append the opt-in provenance-ledger task to the pipeline (default: False)

Optional (extractor — see LLM-free extraction with GLiNER):

• GRAPH_EXTRACTOR: Which implementation fills the extract-and-summarize step — auto (default), llm, or gliner_demo, with gliner kept as an alias for gliner_demo. auto runs the LLM path when a usable LLM key is configured and the GLiNER demo model otherwise
• GLINER_AUTO_INSTALL: Install the GLiNER runtime (gliner2 + PyTorch) on the first cognify that resolves to gliner_demo and finds it missing (default: True). false raises KeylessExtractorNotInstalledError instead
• GLINER_TORCH_INDEX_URL: Package index the auto-install takes PyTorch from, and only PyTorch (default: https://download.pytorch.org/whl/cpu)
• GLINER_INFERENCE_THREADS: How many GLiNER model batches run at once on one shared, loaded model (default: 0 — size it to the machine). 1 runs one batch at a time; N pins the concurrency. The extracted graph is identical at every setting

Optional (chunk summaries — see Chunk summaries from extraction):

• SUMMARY_METHOD: How each chunk’s summary is written — llm (default, one LLM call per chunk) or from_extraction (built from the chunk’s extracted graph, no LLM call)

Parameters

datasets · Union[str, list[str], list[UUID]] · default: "None"

Dataset name(s) or UUID(s) to process. Processes all datasets if not specified.

user · User · default: "None"

User performing the operation.

graph_model · BaseModel · default: "KnowledgeGraph"

Pydantic model defining the knowledge graph schema. Defaults to KnowledgeGraph.

chunker · Any · default: "TextChunker"

Text chunking strategy class.

chunk_size · int · default: "None"

Maximum size of text chunks in tokens. A value above what the embedding model accepts is lowered to that limit with a warning; None auto-calculates it as min(embedding token limit, LLM_MAX_COMPLETION_TOKENS // 2). See Max Completion Tokens.

chunks_per_batch · int · default: "None"

Number of chunks to process per LLM batch.

config · Config · default: "None"

Override the full Cognee config for this run.

vector_db_config · dict · default: "None"

Override vector database configuration.

graph_db_config · dict · default: "None"

Override graph database configuration.

run_in_background · bool · default: "False"

If true, return immediately and process in background.

incremental_loading · bool · default: "True"

If true, skip already-processed data. The skip runs whenever this or data_cache is true, so a full reprocess requires both to be False. See Incremental loading and deduplication.

data_cache · bool · default: "True"

Companion flag to incremental_loading — either one being true enables the already-processed skip for a data item.

custom_prompt · Optional[str] · default: "None"

Custom system prompt for entity/relationship extraction.

data_per_batch · int · default: "20"

Number of data items per processing batch.

llm_config · Optional[LLMConfig] · default: "None"

LLM settings to install into the current async context for this graph-building operation. When omitted, Cognee uses the active context config or global LLM config. Import LLMConfig from cognee.infrastructure.llm.config.

embedding_config · Optional[EmbeddingConfig] · default: "None"

Embedding settings to install into the current async context for this graph-building operation. When omitted, Cognee uses the active context config or global embedding config. Import EmbeddingConfig from cognee.infrastructure.databases.vector.embeddings.config.

dry_run · bool · default: "False"

If true, return a DryRunEstimate of LLM token usage and rough cost instead of running the pipeline. No LLM calls are made and no graph results are written. See Dry-run cost estimation.

raise_on_error · bool · default: "True"

If true, a blocking run whose pipeline errored raises CognifyFailedError instead of returning an errored PipelineRunInfo. Set False to keep the previous swallow-and-return behavior. Ignored when run_in_background=True. See Failed runs.

chunk_attachment · Optional[Literal["direct", "all"]] · default: "None"

How widely each chunk links into the graph extracted from it. "all" links the chunk to every stored node reachable from its extracted root; "direct" and omitting it keep the single-root linkage. Requires a custom DataPoint graph_model. See Chunk attachment.

extractor · Optional[Literal["llm", "gliner_demo", "gliner"]] · default: "None"

Which implementation fills the extract-and-summarize step of the standard pipeline. "gliner_demo" builds the graph and the chunk summaries with the local GLiNER2 demo model instead of the LLM; "gliner" is accepted as an alias for it. Omitting it (or passing None) falls back to the GRAPH_EXTRACTOR setting, which defaults to "auto" — the LLM path with a usable LLM key, the GLiNER demo without one. Any other value raises ValueError. See LLM-free extraction with GLiNER.

summary_method · Optional[Literal["llm", "from_extraction"]] · default: "None"

How the standard pipeline writes each chunk’s TextSummary. "from_extraction" builds it from the chunk’s own extracted types and relations instead of calling the LLM. Omitting it (or passing None) falls back to the SUMMARY_METHOD setting, which defaults to "llm". Any other value raises ValueError: Unknown summary_method ...; expected 'llm' or 'from_extraction'. See Chunk summaries from extraction.

temporal_cognify has been removed. Passing it is still accepted for compatibility but ignored without a warning, and the default pipeline runs. That pipeline already extracts dates as Timestamp nodes, which SearchType.TEMPORAL reads. See Temporal Knowledge Graph.

Failed runs

A blocking cognify() whose pipeline ends in an error raises CognifyFailedError instead of returning a run info with status PipelineRunErrored:

import cognee
from cognee.modules.pipelines.exceptions import CognifyFailedError

try:
    await cognee.cognify(datasets=["my_dataset"])
except CognifyFailedError as error:
    print(error)                 # Cognify failed for dataset 'my_dataset': AuthenticationError: … | Pass raise_on_error=False …
    error.dataset_name           # "my_dataset"
    error.error_class            # class name of the underlying exception, e.g. "AuthenticationError"
    error.error_message          # PII-scrubbed message of the underlying error

The same exception type covers both failure paths: a run that finished with an errored run info, and a run-level exception that escaped the pipeline (an authentication error from the LLM provider, for example). Already-typed Cognee API errors — PermissionDeniedError, DatasetNotFoundError, and other CogneeApiError subclasses — pass through unwrapped, so existing except clauses keep matching. When several datasets are processed, the first errored one raises, and its name is in dataset_name. On the run-level-exception path there is no run info to read the name from, so dataset_name is the stringified datasets argument ("['my_dataset']") rather than a single resolved name.

Pass raise_on_error=False to get the previous behavior:

from cognee.modules.pipelines.models.PipelineRunInfo import get_errored_run_info

result = await cognee.cognify(datasets=["my_dataset"], raise_on_error=False)
errored = get_errored_run_info(result)
if errored:
    print(errored.error_class, errored.error_message)

On the errored-run-info path raise_on_error=False hands back the {dataset_id: PipelineRunInfo} mapping as before; when the failure was a run-level exception, it re-raises the original exception untouched.

Background runs (run_in_background=True) never raise here — the call returns as soon as the run starts. Their failures land on the pipeline run record, and a later recall() surfaces them through the build_failed marker.

The REST /cognify route calls cognify() with raise_on_error=False, and the /remember route passes the same flag through remember(), so their responses (including the errored run-info bodies and the 409 behavior) are unchanged. update() also passes raise_on_error=False, but reports an errored run as its own status: "failed" result rather than the run-info mapping.

Chunk attachment

With a custom DataPoint graph_model, a chunk gets exactly one contains edge into the graph extracted from it — to the root the LLM filled in — and every other extracted entity is reachable only by walking down from that root. (The default KnowledgeGraph path already links each extracted entity to its chunk, which is why this argument does not apply to it.) Pass chunk_attachment="all" to link the chunk once to every stored node reachable from that root instead, so any extracted entity is one hop from the chunk it came from:

from typing import List
import cognee
from cognee.low_level import DataPoint

class Activity(DataPoint):
    name: str
    metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}

class Person(DataPoint):
    name: str
    likes: List[Activity] | None = None
    metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}

class PeopleGraph(DataPoint):
    people: List[Person]

await cognee.cognify(graph_model=PeopleGraph, chunk_attachment="all")
# Every Person *and* every Activity now sits one `contains` edge from its chunk,
# instead of only the PeopleGraph root.

Value	Chunk links to
None (omitted)	Same as "direct" — existing behavior, so existing runs are unaffected.
"direct"	The extracted root; or, when that root is a transparent container, the children that replaced it.
"all"	Every node the run stores from that root — the root itself included, unless it is a transparent container and so never stored. Each node is linked once.

Behavior notes:

• Requires a custom DataPoint graph_model. Passing it with the default KnowledgeGraph (or a subclass of it) raises a ValueError — that path already attaches every extracted entity to its chunk, so there is nothing to widen.
• Rejected up front. An invalid value, a non-DataPoint graph_model, or a connection to a remote instance via serve() each raise a ValueError before any pipeline work starts. cognify() forwards unknown keyword arguments into the LLM call, so an unusable value has to raise rather than be silently dropped.
• Permitted with dry_run=True. The estimate covers the two LLM-heavy stages only and is genuinely unchanged by attachment, so a dry run behaves exactly as it does without the argument.
• cognify()-only. It is not a remember() keyword and not a field on the REST POST /api/v1/cognify body.
• Standard-routed items only, exactly like graph_model — DLT-source manifests and code files run their own task lists and ignore both.
• Independent of metadata["transparent"]. Transparency is a property of the model; attachment is a property of the run. They compose.
• Cost of "all". Edge indexing embeds one EdgeType per distinct edge text, and a contains edge’s text is "<chunk label> contains <node label>." — so a model that yields N nodes per chunk means roughly N extra embedded rows per chunk.

LLM-free extraction with GLiNER

With a usable LLM key, cognify() calls the LLM twice per chunk batch: once to extract entities and relationships, once to summarize. Pass extractor="gliner_demo" to fill that same extract-and-summarize step with the local GLiNER2 demo model instead — no LLM call is made for extraction or for summaries. Everything else about the run is unchanged: documents are still classified and chunked the same way, and add_data_points still embeds what it writes, so an embedder is still required (with nothing configured and no key, that too resolves to a local fastembed model).

This is also what an install with no LLM key at all does on its own: GRAPH_EXTRACTOR defaults to auto, which resolves to gliner_demo whenever no usable key is configured. The open-source extractor is a demo of cognee’s enterprise GLiNER extraction, and selecting it logs one notice per process saying so. See Recall Without an LLM Key for the end-to-end keyless round trip.

Its runtime — gliner2 and PyTorch — is recommended but no longer required up front: a run that needs it and does not find it installs it itself, once, before the pipeline starts. See Installing the runtime below.

The first extraction downloads the model fastino/gliner2.5-base-v1 (~750 MB) into the Hugging Face hub cache, and logs a WARNING naming the model, its approximate size and that directory while it does, so a first call that takes minutes reads as a download rather than a hang; a cached load logs at INFO. Move the cache with HF_HOME.

import cognee

await cognee.add("Tim Cook is the chief executive officer of Apple Inc.")

# Per call — the explicit argument wins over GRAPH_EXTRACTOR
await cognee.cognify(extractor="gliner_demo")

Or select it for the whole process:

GRAPH_EXTRACTOR=gliner_demo   # default: auto

The cognify config is cached for the lifetime of the process, so set GRAPH_EXTRACTOR in your .env or environment before the first cognify() call. The per-call extractor argument is read on every call and always wins; passing anything other than "llm", "gliner_demo", "gliner", or None raises ValueError.

Installing the runtime

The recommended install for GLiNER users is still the extra:

uv pip install "cognee[gliner]"

But PyTorch is not a cognee dependency — an install that uses an LLM key never downloads it — and a missing runtime is no longer a hard stop. When a cognify() resolves to gliner_demo and gliner2/torch are not importable, cognee installs the gliner extra into the running environment itself, once, before the pipeline starts: CPU-only PyTorch from GLINER_TORCH_INDEX_URL first, and only when no torch build (CPU or GPU) is already importable, then the rest of the extra from your default index. That is roughly 200 MB to download and 800 MB on disk, announced by a WARNING naming the environment it is installing into and the index it takes PyTorch from, followed by the installer’s progress lines, a still installing status every 15 seconds, and a closing tip to install the extra so it is not downloaded again. The install runs in a worker thread, so the event loop keeps serving other work — but it is awaited: the pipeline starts only once torch and gliner2 import. Concurrent installs, in one process or across several, serialize on a lock file next to the environment, and only the caller that actually installs does the work.

The install uses whichever installer the environment has: pip when it is importable, otherwise uv — which is what a uv-created virtual environment usually needs, since those ship without pip. uv is looked for beyond PATH, because a process started outside a shell (an IDE, an MCP client, a service, a venv’s python run directly) often has a PATH without it: the UV environment variable (uv run sets it) when it points at an existing file, then PATH, then the running interpreter’s scripts directory, XDG_BIN_HOME, ~/.local/bin, ~/.cargo/bin, and finally the WinGet links directory under %LOCALAPPDATA% on Windows or /opt/homebrew/bin and /usr/local/bin elsewhere. If your uv lives somewhere else — installed into an unrelated Python’s scripts directory, say — point UV at the binary and the install will find it. Only an environment with no pip, where no uv turns up in any of those places, fails up front with this environment has neither pip nor uv — and there the fix is the extra or an LLM_API_KEY, as below.

It is awaited after cognify()’s own argument checks and after the serve() remote route, so a call that raises one of the errors under Limitations, or that runs against a remote instance, never installs anything locally.

Setting	Effect
GLINER_AUTO_INSTALL=false	Never install. A run that needs the missing runtime raises KeylessExtractorNotInstalledError instead, with the extra (or setting LLM_API_KEY) as the fix. Use it where the environment is built ahead of time — the cognee Docker image already bakes the runtime in.
GLINER_TORCH_INDEX_URL	Index the PyTorch step fetches from, default https://download.pytorch.org/whl/cpu. It is the only package taken from that index.

A failed install raises GlinerInstallError (a CogneeConfigurationError) whose remediation is that same pip install "cognee[gliner]". The second step pins every distribution already installed, so an install that would have to replace a package this process has imported fails rather than breaking the running interpreter.

The extra takes PyTorch from PyPI, which on Linux is the CUDA build — what you want on a GPU host, and several GB larger than the CPU wheel the first-use install fetches.

Concurrent model batches

Once the model is loaded, cognee runs its model batches concurrently on that one shared instance — the model itself is never duplicated, and PyTorch’s own intra-op thread count is left untouched, so the extracted graph is identical at every setting. GLINER_INFERENCE_THREADS decides how many batches are in flight at once. Its default, 0, sizes that to the machine: half of PyTorch’s thread count, capped by memory at one batch plus one more for every 2 GB of available memory beyond a 4 GB reserve (a batch of 16 windows holds roughly 1.7 GB of activations). Inside a container, both halves also respect the cgroup limits — a Docker --cpus/--memory or Kubernetes CPU/memory limit lowers the CPU count and the available memory used. The result is never lower than 1, and the chosen size is logged once as GLiNER inference: N concurrent model batch(es).

GLINER_INFERENCE_THREADS=0   # default: size to the machine
GLINER_INFERENCE_THREADS=1   # one batch at a time
GLINER_INFERENCE_THREADS=4   # pin the concurrency

Two caveats. The pool is process-wide, not per-pipeline: it is sized once on first use and shared by every pipeline in the process, so concurrent documents queue behind one another instead of multiplying the memory in flight. And the memory half of the auto-sizing assumes the default batch size of 16 windows — a task list built with a larger get_gliner_demo_tasks(gliner_batch_size=...) holds proportionally more activations per batch and should set GLINER_INFERENCE_THREADS explicitly.

What GLiNER extracts

GLiNER is a span-labelling model, not a generative one, so the extraction schema is a closed list of entity and relation labels rather than a free-form prompt. Cognee builds that list from your ontology when one is configured (ONTOLOGY_FILE_PATH, or a resolver passed through config), and otherwise from built-in label banks. Expect the resulting graph to differ from an LLM-extracted one, and the summaries to be short and deterministic rather than written prose.

Limitations

extractor="gliner_demo" produces the generic KnowledgeGraph, and it is rejected up front — before any pipeline task runs — wherever the run could not honour it:

Combination	Behavior
Custom graph_model	Raises ValueError — only the llm extractor fills a custom schema.
dry_run=True	Raises ValueError — the estimator prices the LLM extraction pipeline and has no cost model for GLiNER.
Connected to a remote instance via serve()	Passing extractor at all — including "llm" — raises ValueError, because the remote call carries no extractor field. A GRAPH_EXTRACTOR setting is not forwarded either, but it raises nothing: the remote silently runs its own default. Call cognee.disconnect() to choose the extractor locally.
Unrecognized **kwargs	Raises ValueError naming them, instead of forwarding them to an LLM call that no longer happens.
custom_prompt	Accepted but ignored, with a custom_prompt is ignored when extractor='gliner_demo'. warning — there is no prompt to replace.

Every row except the serve() one reads the resolved extractor, so a GRAPH_EXTRACTOR=gliner_demo setting — or the auto default resolving to it on a keyless install — raises the same errors as the explicit argument. The serve() check is the exception: it looks only at whether you passed extractor.

Running with no LLM key at all

Two related behaviors make a fully LLM-free remember() → recall() loop possible once extraction is LLM-free, and one input stays outside it:

• The provider preflight skips its LLM half when the pipeline ahead runs no LLM task, so add() / remember() no longer fail on a missing LLM_API_KEY. add() always skips it — it stages data and makes no LLM call of its own — and remember() skips it whenever the extractor it resolves is LLM-free. Embedding configuration is still checked. See Provider consistency preflight.
• recall() without a query_type resolves to SearchType.CHUNKS when no usable LLM is configured, because nothing could write a completion answer. See Auto-routing.
• Media files are the exception on the write side. Images, audio, and video have no LLM-free path, so ImageLoader, AudioLoader, and VideoLoader raise LLMAPIKeyNotSetError themselves — before reading the file — when no usable LLM is configured. Text, document, and code files ingest with no key at all. See Loaders.

Pair the extractor with a local embedding provider (for example fastembed), and optionally set AUTO_FEEDBACK=false — the per-turn feedback analysis is an LLM call of its own, but it checks for a usable key first and returns an empty analysis instead of failing, so turning it off is an optimization rather than a requirement. Anything ending in _COMPLETION still needs an LLM to write the answer, so those search types remain unavailable without a key.

Two guides walk through this end to end: Custom GLiNER Extraction drives the extractor’s task list directly, with your own entity and relation labels and a stats object reporting what it kept, and Recall Without an LLM Key runs a full remember() → recall() loop with no key present, on the extractor and embedder the auto default resolves to.

Chunk summaries from extraction

The standard pipeline writes one TextSummary per chunk, and by default that costs one LLM call per chunk on top of extraction. Set summary_method="from_extraction" to build the summary from the graph the chunk’s extraction already produced instead — no LLM call is made for summaries. The default, "llm", is unchanged.

import cognee

await cognee.add("Tim Cook is the chief executive officer of Apple Inc.")

# Per call — the explicit argument wins over SUMMARY_METHOD
await cognee.cognify(summary_method="from_extraction")

Or select it for the whole process:

SUMMARY_METHOD=from_extraction   # default: llm

The cognify config is cached for the lifetime of the process, so set SUMMARY_METHOD in your .env or environment before the first cognify() call. The per-call summary_method argument is read on every call and always wins; anything other than "llm", "from_extraction", or None raises ValueError: Unknown summary_method ...; expected 'llm' or 'from_extraction'.

What the summary contains

One Type: name, name line per extracted type, in first-seen order, followed by the chunk’s relation texts, one per line and deduplicated. A relation’s text is the description the extraction gave it; when it has none, storage’s fallback label — Source relationship name Target. — is used instead. The default KnowledgeGraph stores entity and type names lowercased, so for the chunk above that is roughly:

person: tim cook
company: apple inc.
Tim Cook is the chief executive officer of Apple Inc.

Only the chunk’s own extracted nodes and relations are used, so a summary never picks up edges another chunk in the same batch attached to the same entities. Custom graph_models are supported: the chunk is walked the way it is stored, with the chunk, its document and its node sets dropped. The summary keeps the same id it has on the LLM path (uuid5(chunk.id, "TextSummary")), so it still pairs with its chunk, and it is embedded like any other summary.

Chunks with no relations get no summary

When a chunk’s extraction produced no relations there is nothing to summarize, so no TextSummary is written for it. The chunk itself is still stored and embedded. Two read-side consequences follow:

• SearchType.SUMMARIES searches the summary collection, so such a chunk is not among its results.
• The hybrid retriever still returns the chunk — it ranks on the chunk’s own vector — but has no summary text to pair with it. Because the pairing contributes nothing for that chunk, it can rank differently than it would with an LLM-written summary.

Cost and sequencing

On the "llm" path extraction and summarization run concurrently, as an extraction call and a summarization call for each chunk. "from_extraction" has to wait: the summary is built from what extraction returns, so extraction completes first and then the summaries are assembled locally. You trade that concurrency for one fewer LLM call per chunk. summarization_model is unused on this path — nothing prompts the LLM for a summary.

Scope and restrictions

summary_method applies to the default LLM-extractor task list only. The gliner_demo extractor builds its own deterministic summaries already and is not affected by this setting.

Combination	Behavior
Connected to a remote instance via serve()	Passing summary_method at all — including "llm" — raises ValueError, because the remote call carries no summary_method field. A SUMMARY_METHOD setting is not forwarded either, but raises nothing. Call cognee.disconnect() to choose the summary method locally.
dry_run=True	Supported. "from_extraction" makes the chunk_summarization stage report no calls and no cost — see Dry-run cost estimation.

Dry-run cost estimation

Pass dry_run=True to preview the LLM token usage and rough USD cost of a cognify() run without making any LLM calls or writing graph results. This is useful for budgeting a large dataset before committing to the run.

import cognee

estimate = await cognee.cognify(datasets=["my_dataset"], dry_run=True)
print(estimate)                    # human-readable summary table
print(estimate.estimated_cost_usd) # e.g. 0.012345
print(estimate.total_tokens)       # input + output tokens across stages
print(estimate.to_dict())          # JSON-serializable dict

The estimate covers the two LLM-heavy stages of the default pipeline — structured_graph_extraction and chunk_summarization — reusing the real document classifier, chunker, and prompt templates so chunk and call counts track an actual run. The estimate follows the resolved summary method: with summary_method="from_extraction" (or SUMMARY_METHOD=from_extraction), the chunk_summarization stage reports zero calls, tokens, and cost, because that path makes no LLM call for summaries.

The returned DryRunEstimate exposes:

Field	Type	Description
operation	str	"cognify" for this call.
model	str	The configured LLM model the estimate is priced against.
chunks	int	Number of chunks that would trigger LLM calls.
chunk_tokens	int	Total input tokens across those chunks.
input_tokens / output_tokens / total_tokens	int	Aggregate token counts across all stages.
estimated_cost_usd	float	Rough total cost across all stages.
skipped_items	int	Items excluded from estimation (e.g. audio/image items, DLT row chunks, code files).
warnings	list[str]	Notes about approximations (reasoning-model output allowance, skipped items, or missing pricing entries).
stages	list	Per-stage breakdown (name, calls, input_tokens, output_tokens, total_tokens, estimated_cost_usd).

Behavior notes:

• Datasets are resolved read-only. Unlike a normal run, a dry run never creates a missing dataset, so estimating a typo’d dataset name fails loudly instead of silently creating one.
• Estimates are upper bounds for re-runs. With incremental_loading=True, a real run skips already-processed documents, so a dry run may over-estimate a re-run.
• Not supported while connected to a remote Cognee instance via serve() — it raises a ValueError.
• chunk_attachment is allowed alongside dry_run. It changes only how the graph is wired at write time, and the estimated stages make no extra LLM calls because of it, so the estimate is the same with or without it. See Chunk attachment.
• Unknown models emit a warning rather than reporting a $0 cost when no pricing entry exists for the configured model.
• Code files are counted as skipped, at zero cost. Items that cognify() would route down the code graph pipeline are separated out before any document is read, so they are never chunked and contribute no tokens or cost. They are folded into skipped_items and reported in warnings as Skipped N code file(s) because they run the deterministic code graph pipeline — no LLM calls. See Loaders for which extensions take that route.

Processing Pipeline

When you call cognify(), data goes through these stages:

1. Document classification — identify content type
2. Text chunking — split into manageable segments
3. Entity extraction — identify entities using the LLM
4. Relationship detection — find connections between entities
5. Graph construction — build the knowledge graph
6. Summarization — generate summaries of content
7. Provenance recording (opt-in, off by default) — append an audit-ledger entry for every document, chunk, entity, and relationship this run produced. Enabled with PROVENANCE_TRACKING=true; when off, the task list is identical to the pipeline above. See Provenance ledger.

Provenance ledger

When PROVENANCE_TRACKING is enabled, cognify() splices one extra task (record_provenance) into the default pipeline. It runs last, after the graph and embeddings have been written, so node ids are persisted and stable. For each item the run produced it appends document → chunk → entity → relationship lineage entries to the append-only provenance_entries table in the relational database, committing all entries for one task invocation as a single chained transaction.

There is no cognify() argument for this feature; it is configured entirely through CognifyConfig:

PROVENANCE_TRACKING=true   # default: false

The cognify config is cached for the lifetime of the process, so set this in your .env or environment before the first cognify() call. Changing it mid-process has no effect.

Behavior notes:

• Failure is never fatal. The task returns its input unchanged and swallows all of its own errors, logging a warning (Provenance recording failed; ingestion unaffected: ...) instead of failing the run. Missing pipeline context or ids degrade to entries with a source_ref_key of None rather than raising. A swallowed failure rolls the whole batch back, so those entries are simply absent — they never claimed sequence numbers, the hash chain stays intact, and verify_chain() still reports valid. Verification proves the stored entries were not tampered with, not that everything an ingestion produced was recorded, so watch that warning if you depend on the ledger being complete.
• Ledger keys are dataset-scoped. Cognee entity ids are deterministic and the ledger lives in the shared relational database, so every node-derived ledger id is prefixed with the dataset id ("{dataset_id}:{raw_id}"), and relationship ids are built from the prefixed endpoints. Two datasets mentioning the same entity name keep separate version chains.
• There is one chain, and one writer at a time. Dataset scoping applies to ledger keys and version chains, not to the hash chain itself: every entry in the table shares a single sequence_id sequence, and each commit takes a ledger-wide write lock (a Postgres advisory lock, so it serializes across processes too). Concurrent cognify() runs — several datasets, several workers — therefore queue behind each other for their ledger commits, which is why entries are batched one transaction per task invocation.
• Custom graph_model schemas are covered. DataPoints that do not follow the default made_from / is_part_of / contains shape are walked with the same traversal add_data_points uses, so every node and edge is still recorded.
• The table must exist. It ships as Alembic revision b8c1d3e5f7a9; run migrations (cognee.run_migrations(), or alembic upgrade head) before enabling the flag on an existing deployment. The migration is idempotent — it is a no-op if the table is already present.

Entries are read back programmatically through ProvenanceManager, which exposes track_entity, track_chunk, track_relationship, get_provenance, get_lineage, trace_lineage, revision_history, invalidate, verify_chain, check, and get_statistics:

from cognee.modules.provenance import get_provenance_manager

manager = get_provenance_manager()

report = await manager.verify_chain()
print(report["valid"], report["total_entries"], report["broken_links"])

Each entry carries a SHA-256 checksum over its canonical JSON plus the previous entry’s checksum, linked by a unique sequence_id, so verify_chain() detects deletion, reordering, and single-field edits. It streams the ledger in sequence order by keyset pagination rather than materializing it client-side, so it is safe to run against a large table.

Contradiction detection

cognify() no longer checks new facts against stored ones. Reviewing conflicting facts is now an opt-in improve stage: call improve(review_conflicts=True), or set IMPROVE_REVIEW_CONFLICTS=true to include it in the automatic improve that runs after remember(). See Improve.

If you used the old options:

• CONTRADICTION_DETECTION is ignored. When it is set, cognee logs a warning once per process pointing to improve(review_conflicts=True).
• CONTRADICTION_CONFIDENCE_THRESHOLD and CONTRADICTION_MAX_FACTS are no longer read.
• Passing functional_relationships= to cognify() raises TypeError.
• contradicts and superseded edges written by earlier versions stay in the graph and are ignored, so no migration is needed.

Examples

import cognee

# Process all datasets
await cognee.cognify()

# Process a specific dataset
await cognee.cognify(datasets=["my_dataset"])

# Process in background
await cognee.cognify(datasets=["large_dataset"], run_in_background=True)

# Use a custom graph model
from pydantic import BaseModel

class MyGraph(BaseModel):
    nodes: list
    edges: list

await cognee.cognify(graph_model=MyGraph)

# Custom extraction prompt
await cognee.cognify(
    custom_prompt="Extract all technical concepts and their relationships."
)

# Pin specific entity and relationship types
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""

await cognee.cognify(custom_prompt=custom_prompt)

When custom_prompt is set, it fully replaces the default graph extraction prompt (see GRAPH_PROMPT_PATH) for the entity/relationship extraction step, so you can constrain exactly which entity types and relationship labels the LLM produces. For a step-by-step walkthrough, see the Custom Prompts guide.

Further details

Background Execution

When run_in_background=True, cognify() starts the processing pipeline as an async background task and returns immediately. The return shape is the same as blocking mode — a dict mapping dataset_id → PipelineRunInfo — but each entry has status PipelineRunStarted instead of PipelineRunCompleted, and the knowledge graph construction continues in the background.

import cognee

# Start processing without waiting for completion
run_info = await cognee.cognify(
    datasets=["large_corpus"],
    run_in_background=True
)

# run_info is a dict of {dataset_id: PipelineRunInfo}
for dataset_id, info in run_info.items():
    print(info.pipeline_run_id)  # UUID to track this run
    print(info.dataset_id)       # Dataset being processed
    print(info.status)           # Initial status (e.g. "PipelineRunStarted")

The returned PipelineRunInfo fields relevant for monitoring:

Field	Type	Description
pipeline_run_id	UUID	Unique identifier for this pipeline run
dataset_id	UUID	The dataset being processed
dataset_name	str	Name of the dataset
status	str	Current status of the run

Possible status values: PipelineRunStarted, PipelineRunYield, PipelineRunCompleted, PipelineRunAlreadyCompleted, PipelineRunErrored.

A PipelineRunErrored entry additionally carries error_class and error_message (PII-scrubbed) describing the root cause. These are the fields a raise_on_error=False caller reads — see Failed runs. payload is still a repr() string, but it now holds the repr of that same root cause instead of the generic PipelineRunFailedError it carried before; unlike error_message it is not PII-scrubbed, so log error_class / error_message rather than payload.

A sixth status, PipelineRunProgress, exists only on the WebSocket channel described below. cognify() never returns or yields it, so SDK callers that want in-flight progress should poll GET /api/v1/datasets/status/progress instead.

Monitoring progress via WebSocket (REST API)

When using the REST API, subscribe to real-time pipeline updates for a run_in_background=True cognify or add run with the WebSocket endpoint:

WebSocket: /cognify/subscribe/{pipeline_run_id}

A background remember run does not publish to this endpoint, so poll its dataset status instead.

Authentication: The handshake accepts exactly the credentials the HTTP API accepts — an API key header (X-Api-Key), a bearer Authorization header, or the auth cookie. The configured authentication backends are tried in registration order and the first one that resolves an active user wins. When REQUIRE_AUTHENTICATION is off, an unauthenticated handshake falls back to the default user, so single-user deployments can connect without credentials just as their HTTP routes do.

Browsers cannot set custom headers when opening a WebSocket, so both the bearer and API key schemes also accept the credential as a ?token= query parameter. The fallback applies only to WebSocket connections — plain HTTP requests still require the header.

A WebSocket handshake is itself an HTTP request, so a ?token= query string can end up in access logs. Uvicorn’s own access/error logs redact it automatically, but a reverse proxy or load balancer in front of Cognee (nginx, AWS ALB) logs the full request path by default — redact the token query parameter there if you terminate WebSocket traffic through one. Prefer a header wherever your client can set one.

Usage example (JavaScript):

const pipelineRunId = "your-pipeline-run-id-uuid";

// Same-origin browser client with an auth cookie — nothing extra needed:
const ws = new WebSocket(`ws://your-server/cognify/subscribe/${pipelineRunId}`);

// Otherwise pass the bearer token or API key as a query parameter:
// const ws = new WebSocket(
//     `ws://your-server/cognify/subscribe/${pipelineRunId}?token=${token}`
// );

ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    console.log("Status:", data.status);
    console.log("Run ID:", data.pipeline_run_id);
    // data.payload contains the current graph data for the dataset
};

ws.onclose = () => {
    // Server closes the connection when processing completes (status: PipelineRunCompleted)
    console.log("Pipeline run finished");
};

Most WebSocket messages have this shape:

{
    "pipeline_run_id": "uuid-string",
    "status": "PipelineRunYield",
    "payload": { /* current graph data for the dataset */ }
}

In-flight progress messages. While a run is executing, the connection also carries PipelineRunProgress messages — one each time a processed result exits the run’s task chain — so a backgrounded run signals it is alive and moving instead of going quiet until the terminal event:

{
    "pipeline_run_id": "uuid-string",
    "status": "PipelineRunProgress",
    "completed_items": null,
    "total_items": null,
    "current_stage": "add_data_points",
    "stage_index": 4,
    "stage_total": 4
}

Field	Type	Description
current_stage	string | null	Name of the task most recently entered in the emitting item’s task chain
stage_index	int | null	1-based position of that task in the chain
stage_total	int | null	Total number of tasks in the chain
completed_items	null	Always null on this channel — see below
total_items	null	Always null on this channel — see below

Read the stage fields for what they are, not as a stage-by-stage tracker: tasks stream results through the chain, so a message only fires once a result has passed through every stage. By then the whole chain has been entered, which means current_stage in practice always names the chain’s final task (add_data_points on the default cognify pipeline) and stage_index equals stage_total. Treat these messages as a liveness heartbeat that also tells you the chain’s length, and get progress numbers from the polling endpoint below.

Two more things to handle in a client:

• Progress messages carry no payload key. Ticks are frequent, and the graph snapshot that the other statuses attach is deliberately not recomputed for each one. Read data.payload only when data.status is not PipelineRunProgress.
• completed_items and total_items arrive as null here. The N-of-M file counts are the polling endpoint’s signal. For a “3 of 10 files” bar, read GET /api/v1/datasets/status/progress, which returns {status, progress} per dataset with completed_items, total_items, and current_stage.

The server closes the WebSocket with one of these codes:

Code	Meaning
1000	The run reached PipelineRunCompleted and its final payload was sent
1008	Rejected: not authenticated, pipeline_run_id is not a valid UUID, no such run, or no read permission on the run’s dataset. Also sent mid-stream if the dataset is deleted or read access is revoked while streaming
1011	The server hit an internal failure while streaming the run

Each 1008 close carries a reason naming which of those it was. A retry replays the same rejection, so clients should stop rather than reconnect.

The run’s update queue is consumed, not observed: there is one subscriber per run. A second client subscribing to the same pipeline_run_id steals events from the first. Authorization is checked before the queue is touched at all, so a rejected caller cannot disturb the real subscriber.

When to use background mode

• Large datasets (>100 MB) where blocking would time out HTTP connections
• API integrations where you want to return a job ID to the caller immediately
• Parallel processing of multiple datasets without waiting for each

For small datasets or scripts, the default blocking mode (run_in_background=False) is simpler and returns the final result directly.