> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt > Use this file to discover all available pages before exploring further. # Organizing Your Data > Ingest the same mixed corpus three ways — one pile, node sets, separate datasets — and watch how each layout changes what recall answers Your assistant's memory holds both your product documentation and your sales-call transcripts, and its answers have started blending the two — a question about API rate limits comes back quoting whatever a rep promised a customer last quarter. The memory is doing its job by linking related facts; the problem is that everything lives in one undifferentiated pile. ## What You'll Build Four small files — two technical documents (an API reference and an architecture guide) and two sales-call transcripts (where a rep promises "unlimited requests" and "instant sync") — are ingested three ways: everything in one dataset, one dataset with node-set tags, and two walled-off datasets with node sets inside each. The same questions run under every layout, and each recall prints as a comparable `QUERY` / `SCOPE` / `ANSWER` block, so you can see exactly what each layout changes: the unfiltered pile blends promise with spec, node sets scope a query to one group while still answering cross-domain questions, and datasets make leakage impossible. The run ends with the rule of thumb the demo exists to teach: node sets are tags, datasets are walls. The complete runnable demo — the script plus the four data files it reads — is [`examples/demos/organizing_your_data/`](https://github.com/topoteretes/cognee/tree/dev/examples/demos/organizing_your_data) — this page walks through its key moments rather than reproducing it. The three sections below cover one layout each: the calls that build it, the output those calls produce in a real run, and what the layout is good and bad at. Output excerpts are verbatim from one run, trimmed with `...`. Every `ANSWER` is a live LLM call, so the wording varies from run to run; each `ANSWER` line is prefixed with the dataset it came from, so the same question can be compared across layouts by eye. ## Features in Play * [NodeSets](/core-concepts/further-concepts/node-sets) — tags each source with soft, overlappable groups at write time; `node_name` scopes retrieval to a tag at read time * [Datasets](/core-concepts/further-concepts/datasets) — the hard boundary with its own permissions and storage that keeps sales transcripts out of technical answers entirely * [Remember](/core-concepts/main-operations/remember) — ingests the four bundled files under each layout's labels * [Recall](/core-concepts/main-operations/recall) — runs the same questions under every layout, scoped by `datasets` and `node_name` ## Everything in One Dataset The baseline, and the layout that produces the complaint the page opens with. All four files land in the default dataset with no labels, and one recall runs with no filter. ```python theme={null} await cognee.remember( TECH_DOCS + [SALES_CALL_INITECH, SALES_CALL_HOOLI], self_improvement=False ) answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION) ``` Docs and sales calls share one retrieval pool, so the rate-limit question retrieves from the API reference and the transcripts alike. In this run the model resolved the conflict in favor of the documentation — but nothing in the query kept the rep's promise out of the pool. The `NOTE` line names the competition to watch for. ```text theme={null} ============================================================================== (1) ALL DATA IN ONE DATASET — the setup that confuses the LLM ============================================================================== QUERY : What rate limits does the Acme API enforce? SCOPE : everything (no filter) NOTE : docs and sales calls share one retrieval pool — the documented limit (100/min) competes with the rep's 'unlimited requests' promise ANSWER: [main_dataset] - Default: 100 requests per minute per API key. - Enterprise: can be raised to 500 requests per minute per key. ... - No plan offers unlimited requests. ``` **Pros** * The simplest layout: one `remember()` call, no labeling decisions, nothing to keep consistent later. * Every fact can link to every other, so any question can draw on the whole corpus. **Cons** * There is no way to scope a query. Every recall sees everything, and sources that disagree compete in one retrieval pool. * Whether the answer favors the spec or the promise is left to the LLM, so a correct answer in one run is not a guarantee for the next. * One set of permissions for the whole pile: you cannot let one audience read the transcripts and another only the docs. **Use it for** * A corpus that covers one domain for one audience, where every question should see everything. * Prototypes and small personal memories, before you know which groupings matter. ## One Dataset with Node Sets The same dataset and the same graph, but every item now carries node-set tags. Tags are soft and overlappable: everything is also tagged `acme_api`, so a shared grouping cuts across the docs/calls split — something no folder-style hierarchy could express. ```python theme={null} # An item can carry several tags at once: everything here is also tagged # "acme_api", so a shared grouping cuts across the docs/calls split. await cognee.remember(TECH_DOCS, node_set=["tech_docs", "acme_api"], self_improvement=False) await cognee.remember( [SALES_CALL_INITECH, SALES_CALL_HOOLI], node_set=["sales_calls", "acme_api"], self_improvement=False, ) ``` Tagging's effect shows the moment a recall is scoped to one tag. `node_name` restricts retrieval to the tagged subgraph, so the same question now answers from the documentation alone. The sales calls are still in the graph — they just don't participate in this query. ```python theme={null} docs_answer = await cognee.recall( RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION, node_name=["tech_docs"], ) ``` Scoped to `tech_docs`, the answer is the documented limit. Scoped to `sales_calls` with the same call shape, it is what the rep said, with no mention of the spec. ```text theme={null} QUERY : What rate limits does the Acme API enforce? SCOPE : node_name=['tech_docs'] NOTE : the documented truth only ANSWER: [main_dataset] - Default: 100 requests per minute per API key. - Enterprise: can raise to 500 requests per minute per key. ... - No plan offers unlimited requests; provision multiple keys to shard traffic. QUERY : What did we promise customers about rate limits? SCOPE : node_name=['sales_calls'] NOTE : what the reps actually said ANSWER: [main_dataset] We promised to remove the 100 requests/minute cap for enterprise deals and to allow unlimited (no throttling) requests during the pilot — the AE said they'd include this in the proposal deck. ``` The payoff of tags over walls is the question you ask with no filter at all. Because docs and calls share one graph, entities from both link to each other, and an unfiltered recall can draw on both sides. ```python theme={null} # The graph is still ONE graph — cross-domain questions work when you # want them to, by simply not filtering. cross_query = "Where do our sales promises contradict the technical documentation?" cross_answer = await cognee.recall(cross_query, query_type=SearchType.GRAPH_COMPLETION) ``` The deliberately unscoped recall lines up the sales promises against the API reference and cites both sources. This is the question a dataset wall would make impossible. ```text theme={null} QUERY : Where do our sales promises contradict the technical documentation? SCOPE : everything (no filter, on purpose) NOTE : a cross-domain question spanning both groups ANSWER: [main_dataset] Short answer — two direct contradictions: - Enterprise “remove rate limits entirely” (sales AE told Initech this) vs. the API docs stating Enterprise can raise the per-key limit only to 500 req/min and “No plan, including Enterprise, offers unlimited requests.” ... - “Unlimited requests during the pilot / we won’t throttle you” (AE promised, and agreed to put it in the proposal deck) vs. the API docs stating rate limits are enforced per API key ... ``` **Pros** * Scoped and cross-domain questions both work: `node_name` narrows a recall to one group, and leaving it out lets the graph connect the groups. * Tags overlap. One item can belong to several groups at once, like `tech_docs` and `acme_api` here. * Cheap to adopt: the same `remember()` call with a `node_set` argument, no new datasets or permissions to manage. **Cons** * The boundary is soft. An unscoped recall still sees everything, so keeping the promise out of the spec is the caller's job on every query. * Tags are assigned at write time, so a new grouping means re-ingesting the affected items. * No separate permissions, storage, or forget per tag — everything still lives in one dataset. **Use it for** * Mixed content that shares entities you still want linked, like documentation and call transcripts about the same product. * Memory that has to answer both narrow questions ("what does the spec say?") and cross-cutting ones ("where do the two disagree?"). * Slicing one team's memory by topic, source, or account without splitting it up. ## Separate Datasets The third layout gives each domain its own dataset. Graphs are built independently, permissions and storage are separate, and either side can be forgotten without touching the other. Node sets still work inside each dataset — here the sales dataset tags each call by account. ```python theme={null} await cognee.remember( TECH_DOCS, dataset_name="tech_docs", node_set=["api_reference"], self_improvement=False, ) await cognee.remember( SALES_CALL_INITECH, dataset_name="sales_calls", node_set=["initech"], self_improvement=False, ) await cognee.remember( SALES_CALL_HOOLI, dataset_name="sales_calls", node_set=["hooli"], self_improvement=False, ) ``` The two mechanisms compose: `datasets` picks the wall, `node_name` picks the tag inside it, and the answer covers one account's calls only. ```python theme={null} initech_query = "What was discussed with Initech?" initech_answer = await cognee.recall( initech_query, query_type=SearchType.HYBRID_COMPLETION, datasets=["sales_calls"], node_name=["initech"], ) ``` The `[tech_docs]` prefix on the first answer shows it came from that dataset alone — the sales calls cannot leak in. The combined `datasets` plus `node_name` filter narrows the second answer to the Initech call. ```text theme={null} ============================================================================== (3) SEPARATE DATASETS (tech vs sales) — hard walls, node sets inside ============================================================================== QUERY : What rate limits does the Acme API enforce? SCOPE : datasets=['tech_docs'] NOTE : sales calls cannot leak in ANSWER: [tech_docs] - Default: 100 requests per minute per API key. - Enterprise: up to 500 requests per minute per API key. ... - No plan offers unlimited requests. QUERY : What was discussed with Initech? SCOPE : datasets=['sales_calls'] + node_name=['initech'] NOTE : one account's calls only ANSWER: [sales_calls] They discussed Initech’s bursty batch jobs and that the 100 requests/min cap would be a blocker; Acme offered to remove rate limits for an enterprise deal and provide unlimited requests during the pilot, and Initech asked that the unlimited pilot offer be included in the proposal deck. ``` The layout has one footgun, and the demo closes on it. `recall()` without `datasets` spans every dataset you can read, so separating data at write time is not enough — queries must opt into the scope they want. ```python theme={null} # Caution: recall() without `datasets` spans ALL datasets you can read — # separating data at write time is not enough, queries must opt into the # scope they want. unscoped_answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION) ``` The same rate-limit question without `datasets` returns one `ANSWER` per readable dataset, `[tech_docs]` and `[sales_calls]`, so the rep's promise is back in the result. The run then closes with the `TAKEAWAY` banner. ```text theme={null} QUERY : What rate limits does the Acme API enforce? SCOPE : everything (no dataset filter) NOTE : spans both datasets again — scope must be chosen per query ANSWER: [tech_docs] - Default: 100 requests per minute per API key. - Enterprise: up to 500 requests per minute per API key. ... - No plan (including Enterprise) offers unlimited requests. ANSWER: [sales_calls] By default a 100 requests per minute cap is mentioned; Acme said they can remove rate limits for an enterprise deal and offered unlimited (no throttling) requests during the pilot. ``` **Pros** * Hard isolation: a recall scoped to one dataset can never surface content from another. * Each dataset has its own permissions and storage, and can be forgotten independently. * Node sets still slice inside a dataset, so you lose nothing from the previous layout within each wall. **Cons** * Graphs are built independently, so there is no cross-dataset reasoning. An unscoped recall returns one answer per dataset rather than one answer that connects them. * The wall only holds when the query names it: `recall()` without `datasets` spans every dataset you can read. * Dataset scoping walls datasets off only in [multi-user mode](/core-concepts/multi-user-mode/data-isolation-and-multi-tenancy), which the default Ladybug and LanceDB stack turns on automatically — keep `ENABLE_BACKEND_ACCESS_CONTROL` at its default or set it to `true`. **Use it for** * Domains that must never contaminate each other's answers, like the spec and the sales promises here. * Content with different access control or retention, where one audience or one lifecycle applies per dataset. * Multi-tenant memory, with one dataset per customer or team. ## Run It ```bash theme={null} uv run python examples/demos/organizing_your_data/organizing_your_data_demo.py ``` Run it from a checkout of the cognee repo with an [LLM provider](/setup-configuration/llm-providers) configured: the script reads its two documents and two transcripts from the sibling `data/` folder, and every answer is a live LLM call. Ladybug (the default) and Neo4j both support node-set filtering out of the box. The script calls `cognee.forget(everything=True)` before each of its three sections, so point it at a scratch instance rather than memory you want to keep — see [Forget](/core-concepts/main-operations/forget). ## Choosing a Layout Start with the simplest layout that answers your queries cleanly. One dataset with no labels is fine while the corpus covers a single domain for a single audience. Reach for node sets the moment distinct content types share entities you still want linked — scoped questions stay clean, and cross-domain questions keep working. Move a domain into its own dataset when it must never contaminate the other's answers, or when it needs different access control or retention — and remember the two compose, so datasets can carry node sets inside them. How node-set labels are written and how `node_name` filters retrieval by them. What a dataset isolates — documents, permissions, and processing. The minimal single-feature guide to tagging memories with node sets. A sibling demo where node-set scoping powers an agent's research phase. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.