Cognify

Transform ingested data into a knowledge graph.

cognify() is a legacy operation. In Cognee v1.0, most users should use remember() instead, which replaces the add() + cognify() + memify() workflow with a single call.

What is the cognify operation

The .cognify operation takes the ingested data with Add and turns plain text into structured knowledge: chunks, embeddings, summaries, nodes, and edges that live in Cognee’s vector and graph stores. It prepares your data for downstream operations like Search.

• Transforms ingested data: builds chunks, embeddings, and summaries
• Graph creation: extracts entities and relationships to form a knowledge graph
• Vector indexing: makes everything searchable via embeddings
• Dataset-scoped: runs per dataset, respecting ownership and permissions

.cognify can be run multiple times as the dataset grows, and Cognee will skip what’s already processed. Read more about Incremental loading in Examples and details

What happens under the hood

The .cognify pipeline is made of six ordered Tasks, plus two optional tasks you can switch on. Each task takes the output of the previous one and moves your data closer to becoming a searchable knowledge graph.

1. Classify documents — wrap each ingested file as a Document object with metadata and optional node sets
2. Check permissions — enforce that you have write access to the target dataset
3. Extract chunks — split documents into smaller pieces (paragraphs, sections)
4. Extract graph — use LLMs to identify entities and relationships, inserting them into the graph DB
5. Summarize text — generate summaries for each chunk, stored as TextSummary DataPoints
6. Add data points — embed nodes and summaries, write them into the vector store, and update graph edges

Two further tasks are appended only when you opt in:

7. Record provenance (opt-in, off by default) — append an audit-ledger entry for every document, chunk, entity, and relationship this run produced. Enable it with PROVENANCE_TRACKING=true; when it is off, the pipeline is exactly the tasks above. See Provenance ledger.
8. Detect contradictions (opt-in, off by default) — compare the facts this run touched against the facts already stored around them and record each conflict as a contradicts edge. Enable it with CONTRADICTION_DETECTION=true; when it is off, the pipeline is exactly the tasks above. See Contradiction detection.

The result is a fully searchable, structured knowledge graph connected to your data.

After cognify finishes

When .cognify completes for a dataset:

• DocumentChunks exist in memory as the granular breakdown of your files
• Summaries are stored and indexed in the vector database for semantic search
• Knowledge graph nodes and edges are committed to the graph database
• Dataset metadata is updated with token counts and pipeline status
• contradicts edges may also be present if you enabled contradiction detection — each one records the two conflicting facts, the reason, and a confidence score
• Your dataset is now query-ready: you can run Search or graph queries immediately

Because cognify() calls the LLM for entity extraction and summarization, it can fail when the configured LLM provider (or LiteLLM proxy) reports that its token budget is exhausted. In that case it raises LLMPaymentRequiredError, which the API surfaces as HTTP 402 (Payment Required) with body {"error": "Token budget exhausted", "detail": "..."}. This error is terminal — Cognee does not retry budget-exhaustion failures — so treat a 402 as final for the request and prompt the user to top up their token budget rather than reissuing the call.

Examples and details

Pipeline tasks (detailed)

1. Classify documents

   • Turns raw Data rows into Document objects
   • Chooses the right document type (PDF, text, image, audio, etc.)
   • Attaches metadata and optional node sets

2. Check permissions

   • Verifies that the user has write access to the dataset

3. Extract chunks

   • Splits documents into DocumentChunks using a chunker
   • You can customize the chunk size and strategy — see Chunkers for details
   • Updates token counts in the relational DB

4. Extract graph

   • Calls the LLM to extract entities and relationships
   • Deduplicates nodes and edges, commits to the graph DB

5. Summarize text

   • Generates concise summaries per chunk
   • Stores them as TextSummary DataPoints for vector search

6. Add data points

   • Converts summaries and other DataPoints into graph + vector nodes
   • Embeds them in the vector store, persists in the graph DB

7. Record provenance (opt-in — set PROVENANCE_TRACKING=true)

   • Runs right after Add data points, where node ids are persisted and stable, and before the contradiction check, so the ledger never depends on contradiction edges
   • Appends document → chunk → entity → relationship lineage entries to the append-only provenance_entries table in the relational DB, one chained transaction per task invocation
   • Every entry carries a SHA-256 checksum linking it to the previous one, so verify_chain() detects deletion, reordering, and single-field edits
   • Fail-safe: it returns its input unchanged and swallows its own errors, so it can never break ingestion — which also means a failed batch is silently absent from the ledger
   • Read entries back through ProvenanceManager, and note the migration requirement — see Provenance ledger

8. Detect contradictions (opt-in — set CONTRADICTION_DETECTION=true)

   • Runs last, so both the new facts and the already-stored ones are persisted and comparable
   • Gathers the facts one hop from the entities this run touched (structural edges such as contains and made_from are skipped) and asks the LLM which pairs conflict
   • Logs a warning per conflict and writes a contradicts edge carrying first_fact, second_fact, reason, and confidence
   • Non-destructive and fail-safe: it only adds edges and swallows its own errors, so it can never break ingestion
   • Tunable with CONTRADICTION_CONFIDENCE_THRESHOLD (default 0.5) and CONTRADICTION_MAX_FACTS (default 500) — see Contradiction detection

Default extraction prompts

Cognee ships with several built-in system prompts for entity and relationship extraction, stored in cognee/infrastructure/llm/prompts/. The active prompt is controlled by the GRAPH_PROMPT_PATH environment variable (default: generate_graph_prompt.txt).

Prompt file	Use case	What it does
generate_graph_prompt.txt	Default balanced extraction	Extracts entities and relationships using the standard Cognee rules: basic node types, human-readable IDs, normalized dates, snake_case relationships, and coreference consistency.
generate_graph_prompt_simple.txt	Lightweight extraction	Uses a shorter, more compact rule set for straightforward graph extraction while keeping the same core conventions around node types, IDs, dates, and relationship naming.
generate_graph_prompt_strict.txt	Tighter schema control	Applies a more explicit prompt with named node categories, stronger relationship constraints, examples, and a strict instruction not to infer facts that are not present in the text.
generate_graph_prompt_guided.txt	More directed graph shaping	Adds guidance for edge direction, allows multi-word entity labels, and encourages logically implied facts when they improve graph clarity without repeating the same fact.

To switch to a different built-in prompt, set the environment variable:

GRAPH_PROMPT_PATH=generate_graph_prompt_strict.txt

Or configure it at runtime via cognee.config:

import cognee

cognee.config.llm_config.graph_prompt_path = "generate_graph_prompt_strict.txt"

If you need to use a custom prompt, refer to our Custom Prompts guide

Datasets and permissions

• Cognify always runs on a dataset
• You must have write access to the target dataset
• Permissions are enforced at pipeline start
• Each dataset maintains its own cognify status and token counts

Incremental loading and deduplication

incremental_loading=True is the default on cognee.add(), cognee.cognify(), and update(), as is its companion data_cache=True. The per-item skip runs whenever either flag is on, so disabling deduplication means passing both as False. Failure behavior is controlled by an environment variable, not a function parameter — see When a data item fails below.

The two flags give you two layers of deduplication:

Layer 1 — content-hash deduplication in add()

Before cognify() runs, add() already deduplicates by content hash within the dataset. Re-adding unchanged content is skipped at ingestion time, while changed content hashes differently and becomes a new record with its own id — the previous record stays. To replace a document’s content while keeping its id, use update().

For the full behavior and scenario table, see Hash-based deduplication on the Add page.

Layer 2 — pipeline-status tracking in cognify()

Before processing each data item, cognify() checks a pipeline_status field on the record. If the status for the cognify_pipeline in the current dataset is already COMPLETED, the item is skipped entirely — no LLM calls, no re-embedding, no graph writes.

Scenario	What happens
Same file added and cognified again (unchanged)	Skipped — pipeline_status is already COMPLETED
File content changes, then add() + cognify()	pipeline_status is reset by add(); cognify() reprocesses
New file added to an existing dataset, then cognify()	Only the new file is processed; existing ones are skipped
incremental_loading=False, data_cache=False passed to cognify()	All items are reprocessed regardless of previous status

Common usage patterns:

Appending new data to an existing dataset

You can grow a dataset over time without reprocessing what’s already there:

import cognee

# Initial load
await cognee.add("First document content", dataset_name="my_dataset")
await cognee.cognify(datasets=["my_dataset"])

# Later: add more data — existing items are skipped automatically
await cognee.add("Second document content", dataset_name="my_dataset")
await cognee.cognify(datasets=["my_dataset"])  # only processes the new document

Forcing a full reprocess

To reprocess everything regardless of status, pass both skip flags as False — the skip runs whenever either one is on:

await cognee.cognify(datasets=["my_dataset"], incremental_loading=False, data_cache=False)

This bypasses the pipeline-status check but does not re-ingest files — use cognee.datasets.empty_dataset() first if you also need to clear the stored data.

Re-ingesting a source that keeps growing

For a table, database, or CSV that gains rows between runs, two independent settings decide what happens. The write_disposition you pass to dlt ingestion decides what the staged snapshot contains on a re-run; incremental_loading / data_cache decide whether Cognee looks at that snapshot at all. write_disposition alone is not enough: a plain re-run is skipped before the grown snapshot is ever compared (see Re-Ingesting a Source), so turn off both skip flags on add() to force the ingestion layer to look:

await cognee.add(
    orders_resource,
    dataset_name="orders",
    primary_key="id",
    write_disposition="replace",
    incremental_loading=False,
    data_cache=False,
)
await cognee.cognify(datasets=["orders"])  # keep the default here

cognify() can keep its defaults: add() clears the record’s pipeline_status whenever the content hash changed, so the grown source is reprocessed while every unchanged document in the dataset is still skipped. Note that append also disables orphan cleanup, so rows deleted upstream stay in the graph.

When a data item fails

Deduplication never suppresses errors, and the behavior is not a function parameter: it is the RAISE_INCREMENTAL_LOADING_ERRORS environment variable, default true.

Value	Behavior
true (default)	The first failing item aborts the run — the remaining items are not processed
false	The error is logged and the remaining items keep processing before the run is marked failed

In both modes the add() / cognify() call returns a PipelineRunErrored run info rather than raising. A failed cognify() run also rolls back its partial artifacts so the next run resumes cleanly — see Crash recovery and stuck pipelines for how the rollback is scoped; add() has no rollback step. Cases where deduplication cannot identify rows are logged as warnings rather than raised: duplicate primary keys within a dlt table are reported and the last row loaded wins for foreign-key targeting.

Batching for faster processing

Two batching parameters control how much work Cognee runs at once during ingestion and graph building:

For new workflows, prefer remember(). It is the current API and accepts these batching controls for permanent-memory ingestion. Use add() and cognify() directly only when you need lower-level control over ingestion and graph building as separate legacy steps.

Parameter	Applies to	What it controls	Default
data_per_batch	remember() permanent memory; legacy add() / cognify()	Maximum number of data items processed concurrently within one dataset pipeline run	20
chunks_per_batch	remember() permanent memory; legacy cognify()	Number of chunks emitted to each chunk-level Cognify task batch, including graph extraction/summarization and data-point persistence	Default Cognify: 2000; temporal Cognify: 10; remember HTTP endpoint form default: 36

data_per_batch is the outer concurrency limit. Cognee schedules the data items in a dataset and uses a semaphore so at most this many items are processed at the same time. The default of 20 is a deliberate concurrency cap — pushing many more items through the pipeline at once overwhelms it regardless of backend. On the default SQLite backend there is an additional, sharper constraint to know about before raising it: every in-flight item writes its own Data row, and at high concurrency the parallel read-then-write transactions hit WAL snapshot-upgrade conflicts that fail immediately with database is locked, bypassing busy_timeout. Deployments on Postgres are not subject to that particular failure. Raise data_per_batch explicitly when you want more ingestion concurrency, and lower it if your deployment is memory-constrained or your model provider’s rate limits are tight.

chunks_per_batch is the inner chunk-task batch size. In the default Cognify pipeline, Cognee passes it as batch_size to the graph extraction/summarization task and to add_data_points. If you do not pass it directly, Cognee checks the chunks_per_batch value from CognifyConfig; when that is unset, the default Cognify pipeline uses 2000.

Where to configure them

• Pass data_per_batch and chunks_per_batch to permanent-memory remember() for the current API path.
• Use legacy add() / cognify() only when you intentionally split ingestion and graph building; data_per_batch applies to both, while chunks_per_batch applies to cognify().
• For the legacy /api/v1/cognify endpoint, both values are accepted in the JSON request body.
• For /api/v1/remember, chunks_per_batch is exposed as a multipart form field. The current remember endpoint does not expose data_per_batch as a form field, so tune data_per_batch through the Python SDK or the lower-level Cognify API when you need that control.

Tuning guidance

Scenario	Suggested starting point
Small local datasets	Keep the defaults. Lower values rarely help unless you are debugging provider limits.
Large datasets on a larger machine	Raise data_per_batch above the default to get more ingestion concurrency — safest on Postgres, since on SQLite high ingestion concurrency risks immediate database is locked failures. Increase chunks_per_batch gradually, for example to 100 or higher, while watching memory, database load, and provider rate limits.
Memory-constrained environments	Lower both values, for example data_per_batch=2 to 5 and chunks_per_batch=10 to 25, to reduce concurrent model calls and in-memory intermediate results.
Faster ingestion with many independent files	Increase data_per_batch first, because it controls how many data items can move through the pipeline concurrently.
Faster graph extraction for long documents	Tune chunks_per_batch, because it controls chunk-level batches after documents are split.

Larger batches can improve throughput by keeping the pipeline, model provider, and databases busier, but they also increase memory pressure and can hit LLM, embedding, or database rate limits sooner. The best values depend on document size, chunk count, model/provider limits, embedding batch behavior, graph/vector database capacity, and the CPU/RAM available to the Cognee process.

How entity and relationship names are determined

During the Extract graph step, Cognee asks the LLM to turn each chunk into graph nodes and edges. The names and types in that graph are inferred from your content rather than fixed in advance.

Element	Fields
Node (vertex)	id - graph-local handle the model assigns, used only to wire up that chunk’s edges; name - human-readable label; type - semantic category such as Person or Organization; description - short summary
Edge (relationship)	source_node_id, target_node_id, relationship_name - a free-text verb phrase such as works_at or produces

The extraction prompt instructs the model to:

• Capture entities, names, nouns, and implied mentions exhaustively
• Form relationships as (start_node, relationship_name, end_node) triplets using explicit and inferred connections
• Avoid duplicates and overly generic terms

That means the resulting graph schema emerges from the data you ingest. Different datasets, prompts, or LLMs can produce slightly different node types and relationship names for similar content.

How extracted nodes become graph nodes. The id the model returns is only a local handle, used to wire up that chunk’s edges. The identity of the stored node is derived from its name (normalized to lowercase, with spaces turned into underscores and apostrophes stripped), so every mention of the same name — across chunks, across documents, and across later .cognify runs — resolves to the same Entity node instead of creating a duplicate.

Three consequences worth knowing:

• If a single extracted chunk graph contains several distinct nodes that share one name, they are not merged: each gets a deterministic chunk-scoped id so they stay separate.
• An extracted edge whose endpoint is not among that chunk’s extracted nodes is dropped, rather than creating a placeholder node for the missing endpoint.
• Entity nodes written before Cognee moved to name-derived ids keep their old ids, so re-running .cognify over data that is already in the graph can create a second node alongside the existing one. Re-process those datasets from scratch if you need the ids to line up.

If you need tighter control over naming, use an OWL ontology or a custom graph model. See Ontologies and Custom Graph Model.

Inspect extracted graph schema

Once .cognify finishes, the graph schema is inspectable because the extracted node types and relationship names now exist in the graph store.

Python SDK

Use the graph engine directly to inspect the stored nodes and edges:

from cognee.infrastructure.databases.graph import get_graph_engine

graph_engine = await get_graph_engine()

# Returns all nodes and edges
nodes, edges = await graph_engine.get_graph_data()

# Inspect unique node types
node_types = {props.get("type") for _, props in nodes if props.get("type")}
print("Node types:", node_types)

# Inspect unique relationship names
relationship_names = {rel_name for _, _, rel_name, _ in edges}
print("Relationship names:", relationship_names)

get_graph_data() returns:

• Nodes as (node_id: str, properties: dict)
• Edges as (source_id: str, target_id: str, relationship_name: str, properties: dict)

If you only need aggregate information, inspect graph metrics instead:

metrics = await graph_engine.get_graph_metrics()
# Returns: num_nodes, num_edges, mean_degree, edge_density,
#          num_connected_components (int), and
#          sizes_of_connected_components (list[int])

metrics = await graph_engine.get_graph_metrics(include_optional=True)
print(metrics)

HTTP server mode

When you run the Cognee HTTP server, you can inspect graph data through the dataset graph endpoint:

GET /api/v1/datasets/{dataset_id}/graph

To explore the same graph visually, use the Graph Visualization guide.

Re-cognify after schema changes

If you update your data model (e.g., add new entity fields or relationships) and want to reprocess existing data:

1. Delete the dataset first, then re-add and re-cognify:

   # Clear existing processed data
   await cognee.datasets.empty_dataset(dataset_id=my_dataset.id)
   
   # Re-add source files
   await cognee.add(source_files, dataset_name="my_dataset")
   
   # Re-cognify with the updated schema
   await cognee.cognify()
   

2. Alternatively, use Memify for additive enrichment — it runs extraction and enrichment tasks over the existing graph without re-ingesting data. This is useful when you want to add new derived facts without reprocessing from scratch.

.cognify skips already-processed data by default. Simply re-running .cognify on unchanged files will not pick up schema changes. You must delete and re-add the data, or use memify for enrichment.

Final outcome

• Vector database contains embeddings for summaries and nodes
• Graph database contains entities and relationships
• Relational database tracks token counts and pipeline run status
• Your dataset is now ready for Search (semantic or graph-based)

Checking indexing status

If you are using the current v1.0 API, see Remember for the recommended indexing-status workflow built around remember() and recall().

If you are working directly with legacy .cognify() or the MCP cognify_status tool, the same dataset status primitives still apply:

• cognee.datasets.get_status([dataset_id])
• GET /api/v1/datasets/status?dataset=<dataset-uuid>
• GET /api/v1/activity/pipeline-runs?dataset_id=<dataset-uuid>

In MCP mode, cognify_status(dataset_name="main_dataset") provides a text summary of recent cognify runs.

LLM call count and cost estimation

The default cognify pipeline makes 2 LLM calls per chunk:

1. Graph extraction — identifies entities and relationships from the chunk text
2. Summarization — generates a concise summary of the chunk

Estimating total calls

The number of chunks depends on your document size and the configured chunk_size:

chunks          = ceil(document_tokens / chunk_size)
total_llm_calls = chunks × 2

When chunk_size is not set explicitly, Cognee auto-calculates it as:

chunk_size = min(embedding_model_max_tokens, llm_max_tokens ÷ 2)

With typical defaults (e.g., gpt-4o-mini + text-embedding-3-small) this usually falls in the 1 024 – 8 192 token range. See Chunkers for details.

Example estimates at chunk_size = 1024:

Document size	Chunks	LLM calls
100 tokens	1	2
1 000 tokens	1	2
10 000 tokens	10	20

Tips for reducing API usage

• Increase chunk_size — fewer, larger chunks mean fewer calls:
  await cognee.cognify(chunk_size=4096)
  
• Skip summarization — use a custom pipeline that omits the summarize_text task, reducing calls to 1 per chunk.
• Pace requests from the start — Cognee already turns the RPM limiter on by itself once a provider reports overload (see Rate Limiting), but you can set LLM_RATE_LIMIT_ENABLED=true to pace every call from the first request and avoid bursting your provider quota when processing many chunks in parallel.

Concurrent search while cognify is running

You can run search while a cognify pipeline is active — there is no global lock that blocks one from the other.

Cognee’s locks are session-level: they serialize short read-modify-write operations (such as update_qa or add_feedback) within the same (session_id, operation) pair. They do not apply across cognify and search.

Within a single process, the default LanceDB vector store uses an asyncio lock per adapter instance to serialize concurrent write coroutines, so interleaved searches and writes within the same worker are safe.

Across multiple processes, sharing the same LanceDB data directory is not supported — Cognee’s embedded stores are single-process. If two workers do open the same directory, LanceDB’s commit-conflict detection surfaces errors like:

CommitConflict: Too many concurrent writers

Seeing this error means two processes are writing to the same embedded vector store. Route all writes through one Cognee process, or switch to an external vector store such as PGVector — see the single-process callout in remember.

For single-process deployments (the default), concurrent search during cognify works without any special configuration.

Add

First bring data into Cognee

Search

Query embeddings or graph structures built by Cognify

Memify

Enrich your graph with derived facts after cognify