> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# S3 Storage
> Step-by-step guide to using S3 for data ingestion and storage
A minimal guide to using S3 (or S3-compatible, e.g., MinIO) to ingest data and/or store Cognee's internal files.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have S3 credentials and access to an S3 bucket
## What S3 Storage Does
* **Ingest from S3**: Pass `s3://...` paths to `cognee.add()` to load data directly from S3
* **Store Cognee data on S3**: Set your data/system roots to S3 URLs to keep all files on S3
* **S3-compatible**: Works with MinIO and other S3-compatible services
## Prerequisites
Install with AWS extra if needed (boto3/s3fs) and add credentials to `.env`:
```dotenv theme={null}
aws_access_key_id=your_access_key
aws_secret_access_key=your_secret_key
aws_region=us-east-1
# Optional for S3-compatible endpoints (e.g., MinIO):
aws_endpoint_url=http://localhost:9000
```
## Option A: Ingest from S3
Pass S3 URIs (files or prefixes) directly to `remember()`. Directories/prefixes expand to files when credentials are set.
```python theme={null}
import asyncio
import cognee
async def main():
# Single file: ingest and build the graph in one call
await cognee.remember(
"s3://cognee-s3-small-test/Natural_language_processing.txt",
dataset_name="s3_single_demo",
self_improvement=False,
)
# Folder/prefix (recursively expands)
await cognee.remember(
"s3://cognee-s3-small-test",
dataset_name="s3_prefix_demo",
self_improvement=False,
)
# Mixed list
await cognee.remember(
[
"s3://cognee-s3-small-test/Natural_language_processing.txt",
"Some inline text to ingest",
],
dataset_name="s3_mixed_demo",
self_improvement=False,
)
if __name__ == "__main__":
asyncio.run(main())
```
```python theme={null}
import asyncio
import cognee
async def main():
# Single file
await cognee.add("s3://my-bucket/docs/paper.pdf")
# Folder/prefix (recursively expands)
await cognee.add("s3://my-bucket/datasets/reports/")
# Mixed list
await cognee.add([
"s3://my-bucket/docs/paper.pdf",
"Some inline text to ingest",
])
# Process the data
await cognee.cognify()
if __name__ == "__main__":
asyncio.run(main())
```
This loads data directly from S3 using the `s3://` URI. `remember()` expands prefixes, reads the S3 objects, and builds retrieval-ready memory for each target dataset.
This simple example uses S3 paths for demonstration. In practice, you can mix S3 files with local files, use dataset scoping, and apply custom loaders. The same `remember()` flow works with S3 paths.
## Option B: Store Cognee Data on S3
Keep Cognee's generated files (text copies, system files) on S3 by pointing roots to S3 URLs.
Add this to your `.env`:
```dotenv theme={null}
DATA_ROOT_DIRECTORY="s3://my-bucket/cognee/data"
SYSTEM_ROOT_DIRECTORY="s3://my-bucket/cognee/system"
# Optional: force S3 backend detection
STORAGE_BACKEND="s3"
```
This configures Cognee to store all its internal files (processed data, system files) on S3 instead of locally.
Cognee chooses S3 storage when roots start with `s3://` (or when `STORAGE_BACKEND=s3` and both roots are S3 URLs). If `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` are not set in `.env`, Cognee falls back to boto3/s3fs's default credential chain for native AWS S3 deployments instead of erroring. See [Object Storage](/setup-configuration/object-storage) for provider-specific setup details.
## Performance notes
Ingesting a file over the S3 backend costs **4 S3 requests per file** (1 PUT, 2 HEAD, 1 GET), down from 13. Cognee now uploads the payload once instead of twice and hashes it from the bytes it already holds, rather than downloading the object back several times to recompute a hash of content it just wrote.
Uploads are written in 4 MB chunks rather than read into memory whole, so the peak memory one upload adds is bounded by the chunk size instead of the file size — large files no longer scale RAM with their length.
Inside `DATA_ROOT_DIRECTORY`, an uploaded source file is stored under a content-addressed key, `/`. Expect one hash-named prefix per distinct payload rather than a flat listing. Derived text keeps its existing flat `text_.txt` name. See [Hash-based file storage](/core-concepts/main-operations/legacy-operations/add) for what this means for deduplication.
Cognee raises botocore's per-client connection pool to 32, above the default per-dataset item concurrency of 20, so concurrent items are limited by the network rather than by the pool. This is a fixed internal value — there is no environment variable for it.
Understand knowledge graph fundamentals
Configure providers and databases
Explore API endpoints