> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Dataset Management

> Endpoints for creating, listing, and managing datasets

Datasets are the organizational unit for all data in Cognee Cloud. Each dataset maintains its own knowledge graph and vector store. See [Datasets](/core-concepts/further-concepts/datasets) for the underlying concept.

## List datasets

**`GET /api/v1/datasets/`** — List all datasets accessible to the authenticated user.

```bash theme={null}
curl https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
  -H "X-Api-Key: your-key"
```

## Create a dataset

**`POST /api/v1/datasets/`** — Create a new dataset or return the existing one if the name already exists.

```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
  -H "X-Api-Key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"name": "my_dataset"}'
```

<Note>
  Datasets are also created implicitly when you call `add` or `remember` with a `dataset_name` that does not yet exist.
</Note>

## Dataset status

**`GET /api/v1/datasets/status`** — Get the processing status of all datasets.

Returns the pipeline state for each dataset: whether cognify is pending, running, or completed.

### In-flight progress

**`GET /api/v1/datasets/status/progress`** — Get the same statuses, plus how far each running pipeline has got.

Takes the same selection parameters as `/status`: repeat `dataset` for each dataset UUID (omit it to cover every dataset you can read) and repeat `pipeline` to pick pipelines (omit it to default to `cognify_pipeline`).

```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/status/progress?dataset=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
  -H "X-Api-Key: your-key"
```

Every value is an object `{status, progress}` rather than a bare status. The flat-versus-nested rule matches `/status` — flat for zero or one `pipeline`, nested per dataset and pipeline for more than one:

```json theme={null}
{
  "b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
    "status": "DATASET_PROCESSING_STARTED",
    "progress": {
      "completed_items": 3,
      "total_items": 10,
      "current_stage": "add_data_points"
    }
  }
}
```

```json theme={null}
{
  "b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
    "add_pipeline": { "status": "DATASET_PROCESSING_COMPLETED", "progress": null },
    "cognify_pipeline": {
      "status": "DATASET_PROCESSING_STARTED",
      "progress": { "completed_items": 3, "total_items": 10, "current_stage": "add_data_points" }
    }
  }
}
```

* **completed\_items** / **total\_items**: files finished out of files in the run. An item that errored still counts as finished, so `completed_items` always reaches `total_items`.
* **current\_stage**: the task most recently entered in an item's task chain. Results stream through the whole chain before they surface, so in practice this names the chain's **final** task (`add_data_points` on the default cognify pipeline) once the first result lands, and is `null` before that. Treat it as "the run is moving", not as a stage-by-stage position — `completed_items`/`total_items` are the fields to drive a progress bar from.

`progress` is `null` when there is nothing in flight to report — before the run's first progress tick, and once the run reaches a terminal state, since the completed or errored record carries no progress snapshot. Treat `null` as "no progress information", not as an error.

Progress ticks are throttled to roughly 20 database writes per run (the first and last file always persist), so on a large batch the numbers advance in steps rather than one file at a time.

<Note>
  `/status` is unchanged — it still returns bare status values in the same shape. Use `/status/progress` when you want the N-of-M numbers, and the [cognify WebSocket](/python-api/cognify) when you want live in-flight updates while a run is executing; polling this endpoint is how a client recovers granular progress after a page refresh or a dropped subscription.
</Note>

A `409` is returned if the progress cannot be retrieved — including when you ask for a dataset you do not have read permission on.

## Graph summary

**`GET /api/v1/datasets/graph-summary`** — Get node and edge counts for each dataset.

Counts are computed once per dataset's latest cognify run and cached, so this is much cheaper than the full graph endpoint when polling dataset sizes repeatedly.

```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary" \
  -H "X-Api-Key: your-key"
```

Pass one or more `dataset_ids` query parameters to summarize specific datasets. Omit it to summarize every dataset you have read access to.

```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary?dataset_ids=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
  -H "X-Api-Key: your-key"
```

Returns a list of summaries, one per dataset, each containing:

* **datasetId**: The dataset's UUID
* **pipelineRunId**: The dataset's latest cognify run, or `null` if it has never been cognified
* **numNodes** / **numEdges**: Graph size for that run
* **computedAt**: When the counts were cached, or `null` if they have not been computed. If `pipelineRunId` is set but `computedAt` is `null`, the last count attempt failed (for example, the graph store was unavailable) and the zero counts are placeholders — the next call retries the computation.

## Dataset data

**`GET /api/v1/datasets/{dataset_id}/data`** — List all data items in a dataset.

Each item also carries the label and metadata attached at upload time:

* **label**: The label given to this file via the `labels` field on [add or remember](/cognee-cloud/functionality/data-ingestion#how-per-file-labels-and-metadata-work), or `null` if none was set
* **externalMetadata**: The stored metadata object — the `external_metadata` entry you sent merged over loader-derived keys, plus a `node_set` key when one was passed at ingest. A file uploaded without metadata stores an empty object rather than `null`.

**`GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw`** — Download the original file for a specific data item. Requires `read` permission on the containing dataset; ownership of the data item itself is not required, so a dataset shared with you is fully downloadable.

## Delete

**`DELETE /api/v1/datasets/{dataset_id}`** — Delete a dataset and all its contents.

**`DELETE /api/v1/datasets/{dataset_id}/data/{data_id}`** — Delete a specific data item from a dataset.

<Warning>
  Deleting a dataset removes all associated documents, knowledge graph data, and embeddings. This cannot be undone.
</Warning>