add()

Ingest data into the Cognee knowledge base

cognee.add()

async def add(
    data: Union[BinaryIO, list[BinaryIO], str, list[str], DataItem, list[DataItem]],
    dataset_name: str = 'main_dataset',
    user: User = None,
    node_set: Optional[List[str]] = None,
    vector_db_config: dict = None,
    graph_db_config: dict = None,
    dataset_id: Optional[UUID] = None,
    preferred_loaders: Optional[List[Union[str, dict[str, dict[str, Any]]]]] = None,
    incremental_loading: bool = True,
    data_per_batch: Optional[int] = 20,
    llm_config: Optional[LLMConfig] = None,
    embedding_config: Optional[EmbeddingConfig] = None,
)

Description

Add data to Cognee for knowledge graph processing.

This is the first step in the Cognee workflow - it ingests raw data and prepares it for processing. The function accepts various data formats including text, files, urls and binary streams, then stores them in a specified dataset for further processing.

Prerequisites:

• LLM_API_KEY: add() stages data and makes no LLM call of its own, so its provider preflight never checks the LLM half — a missing key is not an error here for any input, including a file whose loader has not been resolved yet. Embedding configuration is checked either way. Whether a key is needed is decided where the LLM is actually used: the graph build that follows needs one unless it runs on the gliner_demo extractor, which is what the auto default resolves to when no key is configured; and image, audio, and video files need one during ingestion itself, because their loaders transcribe or describe the file with the LLM. A media file added with no usable LLM raises LLMAPIKeyNotSetError from the loader — before it reads the file — rather than from the preflight. See Loaders for the message and Recall Without an LLM Key for the keyless path.
• Database Setup: Relational and vector databases must be configured
• User Authentication: Uses default user if none provided (created automatically)

Supported Input Types:

• Text strings: Direct text content (str) - any string that is not a file://, s3://, http:// or https:// URL and does not point to an existing local file. An absolute-looking string that is not an existing file (for example "/remember to call the dentist") is ingested as text content on every platform, including Windows.
• File paths: Local file paths as strings in these formats:
  • Absolute paths: “/path/to/document.pdf” (treated as a file reference only when the file exists)
  • File URLs: “file:///path/to/document.pdf” or “file://relative/path.txt”
  • S3 paths: “s3://bucket-name/path/to/file.pdf”
• Binary file objects: File handles/streams (BinaryIO)
• Code repository URLs: "https://github.com/<owner>/<repo>" (or a gitlab.com project, or any http(s) URL ending in .git) is shallow-cloned and ingested as ONE code-repo item that cognify() runs through the enola code graph pipeline (cross-file edges), plus the repository’s documents — the same result as adding a local code project directory. Requires ALLOW_HTTP_REQUESTS and git on PATH. See Code repository URLs below.
• Lists: Multiple files or text strings in a single call

Supported File Formats:

• Text files (.txt, .md, .csv)
• PDFs (.pdf)
• Images (.png, .jpg, .jpeg) - transcribed via vision models, with an optional local OCR pass
• Audio files (.mp3, .wav) - transcribed to text
• Code files (.py, .js, .ts, etc.) - parsed for structure and content
• Office documents (.docx, .pptx)

See the Supported File Formats table below for the full list grouped by loader, including which formats require optional extras.

Workflow:

1. Data Resolution: Resolves file paths and validates accessibility
2. Content Extraction: Extracts text content from various file formats
3. Dataset Storage: Stores processed content in the specified dataset
4. Metadata Tracking: Records file metadata, timestamps, and user permissions
5. Permission Assignment: Grants user read/write/delete/share permissions on dataset

Args: data: The data to ingest. Can be:

• Single text string: “Your text content here”
• Absolute file path: “/path/to/document.pdf” (must exist; otherwise the string is ingested as text)
• File URL: “file:///absolute/path/to/document.pdf” or “file://relative/path.txt”
• S3 path: “s3://my-bucket/documents/file.pdf”
• List of mixed types: [“text content”, “/path/file.pdf”, “file://doc.txt”, file_handle]
• Binary file object: open(“file.txt”, “rb”)
• url: A web link url (https or http); a GitHub/GitLab repository URL is cloned and indexed as a code graph instead of fetched as a page dataset_name: Name of the dataset to store data in. Defaults to “main_dataset”. Create separate datasets to organize different knowledge domains. user: User object for authentication and permissions. Uses default user if None. Default user: “[e-mail]” (created automatically on first use). Users can only access datasets they have permissions for. node_set: Optional list of node identifiers used to organize and filter data points in the knowledge graph. Not an access-control mechanism. vector_db_config: Optional configuration for vector database (for custom setups). graph_db_config: Optional configuration for graph database (for custom setups). dataset_id: Optional specific dataset UUID to use instead of dataset_name. extraction_rules: Optional dictionary of rules (e.g., CSS selectors, XPath) for extracting specific content from web pages using BeautifulSoup tavily_config: Optional configuration for Tavily API, including API key and extraction settings soup_crawler_config: Optional configuration for BeautifulSoup crawler, specifying concurrency, crawl delay, and extraction rules.

Returns: PipelineRunInfo: Information about the ingestion pipeline execution including:

• Pipeline run ID for tracking
• Dataset ID where data was stored
• Processing status and any errors
• Execution timestamps and metadata

Next Steps: After successfully adding data, call cognify() to process the ingested content:

import cognee

# Step 1: Add your data (text content or file path)
await cognee.add("Your document content")  # Raw text
# OR
await cognee.add("/path/to/your/file.pdf")  # File path

# Step 2: Process into knowledge graph
await cognee.cognify()

# Step 3: Search and query
results = await cognee.search("What insights can you find?")

Example Usage:

# Add a single text document
await cognee.add("Natural language processing is a field of AI...")

# Add multiple files with different path formats
await cognee.add([
    "/absolute/path/to/research_paper.pdf",        # Absolute path to an existing file
    "file://relative/path/to/dataset.csv",         # Relative file URL
    "file:///absolute/path/to/report.docx",        # Absolute file URL
    "s3://my-bucket/documents/data.json",           # S3 path
    "Additional context text",                      # Raw text content
    "/remember to call the dentist"                 # No such file -> stored as raw text
])

# Add to a specific dataset
await cognee.add(
    data="Project documentation content",
    dataset_name="project_docs"
)

# Add a single file
await cognee.add("/home/user/documents/analysis.pdf")

# Add a single url and bs4 extract ingestion method
extraction_rules = {
    "title": "h1",
    "description": "p",
    "more_info": "a[href*='more-info']"
}
await cognee.add("https://example.com",extraction_rules=extraction_rules)

# Add a single url and tavily extract ingestion method
Make sure to set TAVILY_API_KEY = YOUR_TAVILY_API_KEY as a environment variable
await cognee.add("https://example.com")

# Add a single url and keenable extract ingestion method
Make sure to set KEENABLE_API_KEY = YOUR_KEENABLE_API_KEY as a environment variable
(Tavily takes precedence if both keys are set.)
await cognee.add("https://example.com")

# Add multiple urls
await cognee.add(["https://example.com","https://books.toscrape.com"])

Environment Variables: Required:

• LLM_API_KEY: API key for your LLM provider (OpenAI, Anthropic, etc.)

Optional:

• LLM_PROVIDER: “openai” (default), “anthropic”, “gemini”, “ollama”, “mistral”, “bedrock”
• LLM_MODEL: Model name (default: “openai/gpt-5.6-luna”)
• DEFAULT_USER_EMAIL: Custom default user email
• DEFAULT_USER_PASSWORD: Custom default user password
• VECTOR_DB_PROVIDER: “lancedb” (default), “chromadb”, “pgvector”
• GRAPH_DATABASE_PROVIDER: “kuzu” (default), “neo4j”
• TAVILY_API_KEY: YOUR_TAVILY_API_KEY
• KEENABLE_API_KEY: YOUR_KEENABLE_API_KEY
• COGNEE_REPOS_DIR: Directory that repository URLs are shallow-cloned into (default: ~/.cognee/repos)

Changed files are refused

add() creates documents; it never updates one. A file that already exists in the dataset — the same local path, or the same filename for an upload — with different content raises DocumentUpdateRequiredError (HTTP 409 on POST /api/v1/add). The error names every such file with its data_id and tells you how to replace it: the SDK message gives the exact cognee.update(data_id=..., data=..., dataset_id=...) call, and the HTTP message gives the PATCH /api/v1/update?data_id=...&dataset_id=... request. update() keeps the document’s id and replaces its graph in place.

• The whole request is refused. The check runs once over every item before ingestion starts, so a batch with one changed file writes nothing.
• Identical content is still a no-op, as before.
• Raw text never conflicts: a text string is named by its content hash, so changed text is a new document.
• Pinned items are exempt: DLT and code-repository manifests, and update()’s own re-add, name their document explicitly.
• Only your own files match: the lookup is scoped to the dataset, owner, and tenant, and s3:// objects and repository URLs are not checked.

Parameters

data · Union[BinaryIO, list[BinaryIO], str, list[str], DataItem, list[DataItem]] · required

Data to ingest. Accepts text strings, file paths (local, S3, or URLs), binary file objects, DataItem objects, or lists of any of these.

DataItem is a lightweight wrapper that lets you attach per-item metadata, human-readable label, and an optional stable data_id. Import it from cognee.tasks.ingestion.data_item:

from cognee.tasks.ingestion.data_item import DataItem

item = DataItem(
    data="/path/to/report.pdf",
    label="q4-earnings-report",
    external_metadata={"title": "Q4 Financial Report", "author": "Jane Smith"},
)
await cognee.add(item)

# Mix items with different labels / metadata in a list
await cognee.add([
    DataItem("Contract text …", label="contract-2024"),
    DataItem("Meeting notes …", external_metadata={"source": "CRM"}),
])

label and external_metadata are stored on the relational Data record. label is not propagated into the knowledge graph automatically.

Pass literal_text=True (default False) to force a string data value to be stored verbatim as a text document instead of being resolved as a locator: with the flag set, a value such as "https://example.com/report" or "/tmp/report.pdf" is ingested as that text rather than fetched or read from disk. Use it for text you did not author — a synced document’s body, for instance — where a value that happens to look like a URL or path must not be followed. The flag holds on every storage path: the default add() path and its incremental pre-save, items with a pinned data_id, and cognee.update().

One check still runs ahead of it: the changed-file refusal above resolves a string as a local path without consulting literal_text, so a literal string without a pinned data_id that names an existing local file, or a directory containing one, that the dataset already holds with different content can still raise DocumentUpdateRequiredError.

external_metadata is copied onto every chunk the item produces at chunking time, so it travels with the chunk on both the graph node and the vector payload. It is never embedded and never part of the chunk id, so it stays invisible to matching and ranking — you cannot search by a metadata value. What it enables is surfacing: HYBRID_COMPLETION can return allowlisted keys alongside the passages it retrieves. Because the copy is stamped when chunks are created, items ingested before this behavior existed carry no metadata on their chunks until they are updated with cognee.update() — re-adding unchanged content is a no-op.

Use node_set when you need tags that flow into the graph and can be used for scoped queries.

dataset_name · str · default: "'main_dataset'"

Name of the dataset to add data to.

user · User · default: "None"

User performing the operation. Uses default user if not provided.

node_set · Optional[List[str]] · default: "None"

List of node set names to associate with the data. Must be None or a list of non-empty strings — pass ["projectA"] even for a single name. Any other shape (a string, an empty or whitespace-only entry, a non-string entry, or a name that cannot be encoded) raises InvalidNodeSetError (HTTP 400, from cognee.modules.engine.models.node_set) before anything is stored. The same rule applies, item by item, to a DataItem’s own node_set and to a node_set key in its external_metadata; with incremental loading (the default), other items in the same call may already be stored when one item is rejected, and an item skipped as already added is not checked at all.

vector_db_config · dict · default: "None"

Override vector database configuration for this operation.

graph_db_config · dict · default: "None"

Override graph database configuration for this operation.

dataset_id · Optional[UUID] · default: "None"

UUID of an existing dataset to add data to. Alternative to dataset_name.

preferred_loaders · Optional[List[Union[str, dict[str, dict[str, Any]]]]] · default: "None"

Custom loader configuration for specific file types.

incremental_loading · bool · default: "True"

If true, skip data that has already been ingested. The skip runs whenever this or data_cache is true, so re-ingesting an already-stored item requires both to be False. See Incremental loading and deduplication.

data_cache · bool · default: "True"

Companion flag to incremental_loading — either one being true enables the already-processed skip for a data item.

data_per_batch · Optional[int] · default: "20"

Number of data items to process per batch.

llm_config · Optional[LLMConfig] · default: "None"

LLM settings to install into the current async context for this ingestion operation. When omitted, Cognee uses the active context config or global LLM config. Import LLMConfig from cognee.infrastructure.llm.config.

embedding_config · Optional[EmbeddingConfig] · default: "None"

Embedding settings to install into the current async context for this ingestion operation. When omitted, Cognee uses the active context config or global embedding config. Import EmbeddingConfig from cognee.infrastructure.databases.vector.embeddings.config.

importance_weight · Optional[float] · default: "0.5"

Floating-point score stored on the Data record for retrieval ranking. Applied uniformly to all items in the batch. Use a higher value to make items more likely to surface in ranked results.

primary_key · str · default: "None"

Column name for primary key when ingesting dlt resources or database connection strings. Auto-detected if not specified. Also reaches CSV files: cognee folds it into the dlt_csv_loader entry of preferred_loaders, and setting it in both places raises ValueError. Can also be given inside dlt_config.

dlt_config · Optional[dict] · default: "None"

Options that apply only to dlt sources (connection strings, dlt resources, CSV files), grouped in one dict: primary_key, write_disposition, query, max_rows_per_table, column_value_columns, temporal_columns. An unknown key raises a pydantic ValidationError. The bare keyword arguments are still accepted, but an option given both in dlt_config and as a keyword argument raises ValueError. temporal_columns ({table: [column, ...]}, "*" wildcards, default {"*": ["*"]} from DLT_TEMPORAL_COLUMNS) selects which date cells become Timestamp edges; {} turns them off. See DLT Options.

write_disposition · str · default: "'replace'"

How to handle existing data for dlt sources: “replace” (drop and recreate), “merge” (upsert on primary_key), or “append” (always insert). Any other value raises InvalidDLTArgumentError. This controls dlt’s staging snapshot only — a growing source also needs the ingestion skip turned off, see Re-ingesting a source that keeps growing.

query · str · default: "None"

SQL query that filters what is ingested when using a database connection string as input. Accepts SELECT ... FROM <table> [WHERE ...], where the FROM target may be schema-qualified and may carry a table alias. A WHERE clause that references the alias, and a FROM clause containing JOIN, raise a ValueError — see Database Connection String for the accepted shapes, the rewrites, and the two detection caveats.

Supported Input Types

Type	Example
Text string	"Cognee is a knowledge graph platform."
File path	"/path/to/document.pdf"
S3 path	"s3://bucket/file.txt"
URL	"https://example.com/article"
Code repository URL	"https://github.com/owner/repo" (cloned and indexed as a code graph — see below)
Binary file	open("file.pdf", "rb")
Mixed list	["text", "/path/file.pdf", open("f.txt", "rb")]
dlt resource	@dlt.resource() decorated generator
CSV file	"/path/to/data.csv" (routed through dlt structured ingestion by the loader engine; flattened to text only with preferred_loaders=[{"csv_loader": {}}])
DB connection	"postgresql://user:pass@host/db"

Code Repository URLs

A string that names a whole git repository is recognised by shape and handled as code rather than as a web page: Cognee shallow-clones it (git clone --depth 1) and hands cognify() one code-repo item, which takes the CODE_REPO route (a single enola pass with cross-file edges) alongside the repository’s own documents. This is the same graph you get from adding a local code project directory.

# Indexed as a code graph — no content_type needed
await cognee.add("https://github.com/owner/repo", dataset_name="my_repo")
await cognee.cognify(datasets=["my_repo"])

# Same for remember()
await cognee.remember("https://github.com/owner/repo", dataset_name="my_repo")

Recognised as a repository:

• https://github.com/<owner>/<repo> — a repository root on github.com
• https://gitlab.com/<group>/<repo> — a project on gitlab.com, nested groups included
• Any http(s) URL whose last path segment ends in .git, on any host

An optional .git suffix, trailing slash, www. prefix, query string, or fragment is tolerated and stripped before cloning.

Not recognised (these stay on the ordinary web-page path):

• Deeper forge URLs such as /blob/, /tree/, /issues, /pull/, and GitLab’s /-/ pages
• Forge site pages such as https://github.com/topics/rust or https://gitlab.com/explore
• git@… and ssh://… specs — pass these explicitly through remember(..., content_type="code")

Requirements and behaviour:

• git must be on the server’s PATH, and ALLOW_HTTP_REQUESTS must not be false
• Clones land under COGNEE_REPOS_DIR (default ~/.cognee/repos), are reused across calls, and are refreshed with a best-effort git pull — budget disk space accordingly
• Cloning is credential-free. For a private repository, use remember(url, content_type="code", repo_credentials="<token>"), described in Code Graph
• A node_set passed with a repository (URL or local code project directory) tags every code graph node, the CodeRepository node included, not only the repository’s documents — see Code Graph

Supported File Formats

Loader	Extensions	Install extra
CodeLoader	.py .js .ts .go .rs .java and 25 more source-code extensions — see Loaders; these take the code graph pipeline instead of LLM extraction	— (built-in)
TextLoader	.txt .md .json .xml .yaml .yml .log	— (built-in)
DltCsvLoader	.csv — the default for every CSV; takes precedence over CsvLoader and routes rows through dlt structured ingestion	— (built-in)
CsvLoader	.csv — text flattening; reached only via preferred_loaders=[{"csv_loader": {}}]	— (built-in)
PyPdfLoader	.pdf	— (built-in)
ImageLoader	.png .jpg .jpe .jpeg .gif .webp .bmp .tif .tiff .heic .avif .ico .psd .apng .cr2 .dwg .xcf .jxr .jpx	— (built-in)
AudioLoader	.mp3 .wav .aac .flac .ogg .m4a .mid .amr .aiff	— (built-in)
UnstructuredLoader	.docx .doc .odt .xlsx .xls .ods .pptx .ppt .odp .rtf .html .htm .eml .msg .epub	pip install cognee[docs]
AdvancedPdfLoader	.pdf (layout-aware, preserves tables)	pip install cognee[docs]
BeautifulSoupLoader	.html	pip install cognee[scraping]

See Loaders for how to override the default loader selection or register custom loaders.

Examples

import cognee

# Add text
await cognee.add("Cognee builds knowledge graphs from your data.")

# Add a file
await cognee.add("/path/to/report.pdf", dataset_name="reports")

# Add multiple items to a named dataset
await cognee.add(
    ["First document text", "/path/to/second.pdf"],
    dataset_name="my_project",
)

# Add with custom node set
await cognee.add("Technical spec content", node_set=["engineering"])

# Add a public repository — cloned and indexed as a code graph, not scraped
await cognee.add("https://github.com/owner/repo", dataset_name="my_repo")

# Add structured data via dlt resource
import dlt

@dlt.resource()
def my_data():
    yield [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]

await cognee.add(my_data, dataset_name="people", primary_key="id")

# Add a CSV file (the loader engine routes it through dlt structured
# ingestion by default; dlt options reach the CSV loader)
await cognee.add(
    "/path/to/data.csv",
    dataset_name="csv_data",
    dlt_config={"primary_key": "id"},
)

# Add from a database with filtering
await cognee.add(
    "postgresql://user:pass@host/db",
    dataset_name="db_data",
    primary_key="id",
    query="SELECT * FROM users WHERE active = true",
)

For a complete guide on structured data ingestion with dlt, see the dlt integration page.