Dataset Management

Endpoints for creating, listing, and managing datasets

Datasets are the organizational unit for all data in Cognee Cloud. Each dataset maintains its own knowledge graph and vector store. See Datasets for the underlying concept.

List datasets

GET /api/v1/datasets/ — List all datasets accessible to the authenticated user.

curl https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
  -H "X-Api-Key: your-key"

Create a dataset

POST /api/v1/datasets/ — Create a new dataset or return the existing one if the name already exists.

curl -X POST https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
  -H "X-Api-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"name": "my_dataset"}'

Datasets are also created implicitly when you call add or remember with a dataset_name that does not yet exist.

Dataset status

GET /api/v1/datasets/status — Get the processing status of all datasets.

Returns the pipeline state for each dataset: whether cognify is pending, running, or completed.

Passing a dataset UUID you cannot read is a permission failure, not a conflict: the endpoint returns 403 Forbidden with the standard permission-denied body, and every UUID you pass must be readable — one inaccessible entry fails the whole request. A UUID that does not exist at all fails the same check, so it returns 403 too, not 404.

{
  "detail": "Request owner does not have necessary permission: [read] for all datasets requested. [PermissionDeniedError]"
}

A 409 is reserved for an unexpected failure while retrieving the statuses.

In-flight progress

GET /api/v1/datasets/status/progress — Get the same statuses, plus how far each running pipeline has got.

Takes the same selection parameters as /status: repeat dataset for each dataset UUID (omit it to cover every dataset you can read) and repeat pipeline to pick pipelines (omit it to default to cognify_pipeline).

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/status/progress?dataset=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
  -H "X-Api-Key: your-key"

Every value is an object {status, progress} rather than a bare status. The flat-versus-nested rule matches /status — flat for zero or one pipeline, nested per dataset and pipeline for more than one:

{
  "b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
    "status": "DATASET_PROCESSING_STARTED",
    "progress": {
      "completed_items": 3,
      "total_items": 10,
      "current_stage": "add_data_points"
    }
  }
}

{
  "b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
    "add_pipeline": { "status": "DATASET_PROCESSING_COMPLETED", "progress": null },
    "cognify_pipeline": {
      "status": "DATASET_PROCESSING_STARTED",
      "progress": { "completed_items": 3, "total_items": 10, "current_stage": "add_data_points" }
    }
  }
}

• completed_items / total_items: files finished out of files in the run. An item that errored still counts as finished, so completed_items always reaches total_items.
• current_stage: the task most recently entered in an item’s task chain. Results stream through the whole chain before they surface, so in practice this names the chain’s final task (add_data_points on the default cognify pipeline) once the first result lands, and is null before that. Treat it as “the run is moving”, not as a stage-by-stage position — completed_items/total_items are the fields to drive a progress bar from.

progress is null when there is nothing in flight to report — before the run’s first progress tick, and once the run reaches a terminal state, since the completed or errored record carries no progress snapshot. Treat null as “no progress information”, not as an error.

Progress ticks are throttled to roughly 20 database writes per run (the first and last file always persist), so on a large batch the numbers advance in steps rather than one file at a time.

/status is unchanged — it still returns bare status values in the same shape. Use /status/progress when you want the N-of-M numbers, and the cognify WebSocket when you want live in-flight updates while a run is executing; polling this endpoint is how a client recovers granular progress after a page refresh or a dropped subscription.

Error codes match /status: asking for a dataset you do not have read permission on returns 403 Forbidden with the PermissionDeniedError body, and a 409 means the progress could not be retrieved for an unexpected reason.

Per-item processing status

GET /api/v1/datasets/{dataset_id}/processing-status — Get which individual data items in one dataset have been processed by a pipeline, and which have not.

/status and /status/progress report the state of a run. This endpoint reports the state of the dataset’s items: which of them carry the per-item completion stamp for a pipeline. It is the endpoint to reach for when triaging an incremental load — after a partial or interrupted run, it names the files that still need work.

The optional pipeline query parameter selects the pipeline whose stamp is counted. It defaults to cognify_pipeline; add_pipeline and code_graph_pipeline are the other pipeline names Cognee stamps items with. An unknown name is not rejected — it simply reports every item as pending.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e/processing-status" \
  -H "X-Api-Key: your-key"

{
  "total": 3,
  "completed": 2,
  "pending": 1,
  "items": [
    { "id": "3f2b1a0c-5d6e-4f70-8192-a3b4c5d6e7f8", "name": "handbook.pdf", "completed": true },
    { "id": "7c8d9e0f-1a2b-4c3d-9e0f-1a2b3c4d5e6f", "name": "notes.md", "completed": true },
    { "id": "9a8b7c6d-5e4f-4302-b1a0-9f8e7d6c5b4a", "name": "report_q3.pdf", "completed": false }
  ]
}

• total: number of data items in the dataset.
• completed: items carrying the per-item completion stamp for the requested pipeline.
• pending: total - completed.
• items: one entry per data item, newest first — the same order as GET /api/v1/datasets/{dataset_id}/data, though this list is not paginated and always covers the whole dataset. Each id is the data_id accepted by DELETE /api/v1/datasets/{dataset_id}/data/{data_id} and by forget(data_id=...), so an item listed here as pending can be acted on directly.

Per-item errored state is not persisted, so it is not reported. A pending item may be untouched, in progress, or failed — the flag is “not stamped complete”, not “not started”. Use /status or /status/progress to tell whether a run is currently going, and this endpoint to tell which items it has finished.

A 404 is returned when the dataset does not exist or you do not have read permission on it, and a 409 when the status cannot be computed — the same codes as the other per-dataset routes such as GET /api/v1/datasets/{dataset_id}/data.

Graph summary

GET /api/v1/datasets/graph-summary — Get node and edge counts for each dataset.

Counts are computed once per dataset’s latest cognify run and cached, so this is much cheaper than the full graph endpoint when polling dataset sizes repeatedly.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary" \
  -H "X-Api-Key: your-key"

Pass one or more dataset_ids query parameters to summarize specific datasets. Omit it to summarize every dataset you have read access to.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary?dataset_ids=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
  -H "X-Api-Key: your-key"

Returns a list of summaries, one per dataset, each containing:

• datasetId: The dataset’s UUID
• pipelineRunId: The dataset’s latest cognify run, or null if it has never been cognified
• numNodes / numEdges: Graph size for that run
• computedAt: When the counts were cached, or null if they were not cached on this call. A null here has two causes, and they mean opposite things: either the last count attempt degraded (for example, the graph store was unavailable), in which case the counts are 0 placeholders and the next call retries; or a concurrent caller cached the same run first, in which case the counts you were served are exact. Read the counts themselves rather than treating every null as a failure.

A 409 is returned when the summary could not be built. This means the relational read failed — a single unreadable graph store does not cause it, since that dataset simply comes back with zero counts.

Live dataset updates

WS /api/v1/visualize/subscribe/{dataset_id} — Follow one dataset’s activity over a WebSocket instead of polling.

This is the push replacement for repeatedly calling GET /api/v1/visualize/live-events and refetching the graph payload just to notice that a cognify run finished. Authentication accepts the same credentials as every other endpoint — API key header, bearer header, or auth cookie, sent with the handshake — plus one WebSocket-only fallback: a browser cannot set headers when opening a WebSocket, so the same API key or bearer token can be passed as a ?token= query parameter instead. Prefer the header or cookie where you can send one — a handshake is an HTTP request, and its full URL (query string included) lands in the default access logs of common reverse proxies such as nginx or an AWS ALB, so deployments terminating WebSocket traffic behind one should redact the token parameter there. Cognee redacts it from uvicorn’s own logs. Nothing is ever read from client frames.

Every frame is a JSON object discriminated by kind:

• ready — {"kind": "ready", "dataset_id": str, "cursor": str | null}, sent once the connection is authorized, echoing the cursor it starts from
• live_events — {"kind": "live_events", "events": [...], "cursor": str}, roughly every 2s and only when the delta is non-empty. The events are exactly what GET /api/v1/visualize/live-events returns
• graph_grew — {"kind": "graph_grew", "pipeline_run_id": str}, within about 5s of a cognify run for this dataset completing. A run already complete when you connected is the baseline and is not announced
• heartbeat — {"kind": "heartbeat", "time": str}, every 15s or so, so a quiet stream is distinguishable from a dead one

Pass the cursor from the last live_events frame back as the since query parameter when reconnecting; omit it to start from every available event.

Close codes:

• 1008: not authenticated, or no read permission on this dataset. A retry replays the same rejection, so clients should stop. Permission is re-checked on every poll, so access revoked mid-stream closes the connection this way too
• 1011: the stream failed server-side; reconnecting is reasonable

A malformed since, or a dataset_id that is not a UUID, fails request validation before the connection is accepted. Depending on the ASGI server the client sees either a 1008 close or a plain HTTP rejection of the handshake, so treat a handshake that never opened as a client-side bug rather than a stream error.

Dataset data

GET /api/v1/datasets/{dataset_id}/data — List one page of data items in a dataset.

The response is a page, not the whole dataset. It carries at most limit items — 100 by default — ordered newest first (created_at descending, ties broken by id).

• limit (default 100, range 1–1000): maximum number of items in the page. A value outside the range fails request validation with 422.
• offset (default 0, maximum 1000000): number of items to skip. An offset above 1,000,000 fails request validation with 422.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e/data?limit=100&offset=0" \
  -H "X-Api-Key: your-key"

Each item also carries the label and metadata attached at upload time:

• label: The label given to this file via the labels field on add or remember, or null if none was set
• externalMetadata: The stored metadata object — the external_metadata entry you sent merged over loader-derived keys, plus a node_set key when one was passed at ingest. A file uploaded without metadata stores an empty object rather than null.
• dataSize: The stored size of the file in bytes, or null for a row written without one.

Paging through a dataset

The response is a bare JSON array: there is no X-Total-Count header and no truncation marker, so a full page is indistinguishable from a complete dataset — that is what /data/count is for. Raising limit only grows the page; it does not lift the cap. To read a whole dataset, advance offset until a short page comes back:

import httpx


async def list_all_data(dataset_id: str) -> list[dict]:
    rows: list[dict] = []
    seen: set[str] = set()
    offset, limit = 0, 1000

    async with httpx.AsyncClient(
        base_url="https://your-tenant.aws.cognee.ai",
        headers={"X-Api-Key": "your-key"},
    ) as client:
        while True:
            response = await client.get(
                f"/api/v1/datasets/{dataset_id}/data",
                params={"limit": limit, "offset": offset},
            )
            response.raise_for_status()
            page = response.json()

            # Deduplicate: a concurrent insert or delete can shift a row between pages.
            for row in page:
                if row["id"] not in seen:
                    seen.add(row["id"])
                    rows.append(row)

            if len(page) < limit:
                return rows
            offset += len(page)

Offset paging is not a snapshot. Inserts and deletes that land while you are traversing shift rows between pages, so a concurrently changing dataset can yield a row twice or skip one entirely. Deduplicating on id — as the loop above does — removes the duplicates; a row that shifts out of an already-read page is missed regardless, so traverse while the dataset is idle when you need completeness.

Because offsets stop at 1,000,000, a dataset larger than 1,001,000 items cannot be fully walked through this endpoint, even though /data/count still reports its exact size.

Data count

GET /api/v1/datasets/{dataset_id}/data/count — Get the total number of data items in a dataset without listing them.

The length of a /data response is a page size, not a total, so this is the endpoint to read when you only need “how many documents” — a count badge, or polling an upload until every file has landed.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e/data/count" \
  -H "X-Api-Key: your-key"

{ "count": 128 }

The count is uncapped — it is exact even for datasets past the offset limit above. A 404 is returned when the dataset does not exist or you do not have read permission on it, the same as GET /api/v1/datasets/{dataset_id}/data.

Download a raw file

GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw — Download the original file for a specific data item. Requires read permission on the containing dataset; ownership of the data item itself is not required, so a dataset shared with you is fully downloadable.

Export a dataset

Cognee Cloud has no one-click export of a whole dataset. The HTTP API has no COGX export either, because cognee.export() is part of the Python library. To take your data out, combine these endpoints:

What you want	How
The original files	List the items with GET /api/v1/datasets/{dataset_id}/data, then download each one with GET .../data/{data_id}/raw.
The knowledge graph (nodes and edges)	GET /api/v1/datasets/{dataset_id}/graph returns the dataset’s nodes and edges as JSON.
Usage and activity logs	Export CSV on the Activity, Analytics, and Memory Coverage pages.

To move data between Cognee Cloud and a local instance, see Local mode & sync.

Delete

DELETE /api/v1/datasets/{dataset_id} — Delete a dataset and all its contents.

DELETE /api/v1/datasets/{dataset_id}/data/{data_id} — Delete a specific data item from a dataset.

Deleting a dataset removes all associated documents, knowledge graph data, and embeddings. This cannot be undone.