Dataset Management

Endpoints for creating, listing, and managing datasets

Datasets are the organizational unit for all data in Cognee Cloud. Each dataset maintains its own knowledge graph and vector store. See Datasets for the underlying concept.

List datasets

GET /api/v1/datasets/ — List all datasets accessible to the authenticated user.

curl https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
  -H "X-Api-Key: your-key"

Create a dataset

POST /api/v1/datasets/ — Create a new dataset or return the existing one if the name already exists.

curl -X POST https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
  -H "X-Api-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"name": "my_dataset"}'

Datasets are also created implicitly when you call add or remember with a dataset_name that does not yet exist.

Dataset status

GET /api/v1/datasets/status — Get the processing status of all datasets.

Returns the pipeline state for each dataset: whether cognify is pending, running, or completed.

In-flight progress

GET /api/v1/datasets/status/progress — Get the same statuses, plus how far each running pipeline has got.

Takes the same selection parameters as /status: repeat dataset for each dataset UUID (omit it to cover every dataset you can read) and repeat pipeline to pick pipelines (omit it to default to cognify_pipeline).

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/status/progress?dataset=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
  -H "X-Api-Key: your-key"

Every value is an object {status, progress} rather than a bare status. The flat-versus-nested rule matches /status — flat for zero or one pipeline, nested per dataset and pipeline for more than one:

{
  "b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
    "status": "DATASET_PROCESSING_STARTED",
    "progress": {
      "completed_items": 3,
      "total_items": 10,
      "current_stage": "add_data_points"
    }
  }
}

{
  "b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
    "add_pipeline": { "status": "DATASET_PROCESSING_COMPLETED", "progress": null },
    "cognify_pipeline": {
      "status": "DATASET_PROCESSING_STARTED",
      "progress": { "completed_items": 3, "total_items": 10, "current_stage": "add_data_points" }
    }
  }
}

• completed_items / total_items: files finished out of files in the run. An item that errored still counts as finished, so completed_items always reaches total_items.
• current_stage: the task most recently entered in an item’s task chain. Results stream through the whole chain before they surface, so in practice this names the chain’s final task (add_data_points on the default cognify pipeline) once the first result lands, and is null before that. Treat it as “the run is moving”, not as a stage-by-stage position — completed_items/total_items are the fields to drive a progress bar from.

progress is null when there is nothing in flight to report — before the run’s first progress tick, and once the run reaches a terminal state, since the completed or errored record carries no progress snapshot. Treat null as “no progress information”, not as an error.

Progress ticks are throttled to roughly 20 database writes per run (the first and last file always persist), so on a large batch the numbers advance in steps rather than one file at a time.

/status is unchanged — it still returns bare status values in the same shape. Use /status/progress when you want the N-of-M numbers, and the cognify WebSocket when you want live in-flight updates while a run is executing; polling this endpoint is how a client recovers granular progress after a page refresh or a dropped subscription.

A 409 is returned if the progress cannot be retrieved — including when you ask for a dataset you do not have read permission on.

Graph summary

GET /api/v1/datasets/graph-summary — Get node and edge counts for each dataset.

Counts are computed once per dataset’s latest cognify run and cached, so this is much cheaper than the full graph endpoint when polling dataset sizes repeatedly.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary" \
  -H "X-Api-Key: your-key"

Pass one or more dataset_ids query parameters to summarize specific datasets. Omit it to summarize every dataset you have read access to.

curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary?dataset_ids=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
  -H "X-Api-Key: your-key"

Returns a list of summaries, one per dataset, each containing:

• datasetId: The dataset’s UUID
• pipelineRunId: The dataset’s latest cognify run, or null if it has never been cognified
• numNodes / numEdges: Graph size for that run
• computedAt: When the counts were cached, or null if they have not been computed. If pipelineRunId is set but computedAt is null, the last count attempt failed (for example, the graph store was unavailable) and the zero counts are placeholders — the next call retries the computation.

Dataset data

GET /api/v1/datasets/{dataset_id}/data — List all data items in a dataset.

Each item also carries the label and metadata attached at upload time:

• label: The label given to this file via the labels field on add or remember, or null if none was set
• externalMetadata: The stored metadata object — the external_metadata entry you sent merged over loader-derived keys, plus a node_set key when one was passed at ingest. A file uploaded without metadata stores an empty object rather than null.

GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw — Download the original file for a specific data item. Requires read permission on the containing dataset; ownership of the data item itself is not required, so a dataset shared with you is fully downloadable.

Delete

DELETE /api/v1/datasets/{dataset_id} — Delete a dataset and all its contents.

DELETE /api/v1/datasets/{dataset_id}/data/{data_id} — Delete a specific data item from a dataset.

Deleting a dataset removes all associated documents, knowledge graph data, and embeddings. This cannot be undone.