Organizing Your Data

Ingest the same mixed corpus three ways — one pile, node sets, separate datasets — and watch how each layout changes what recall answers

Your assistant’s memory holds both your product documentation and your sales-call transcripts, and its answers have started blending the two — a question about API rate limits comes back quoting whatever a rep promised a customer last quarter. The memory is doing its job by linking related facts; the problem is that everything lives in one undifferentiated pile.

What You’ll Build

Four small files — two technical documents (an API reference and an architecture guide) and two sales-call transcripts (where a rep promises “unlimited requests” and “instant sync”) — are ingested three ways: everything in one dataset, one dataset with node-set tags, and two walled-off datasets with node sets inside each. The same questions run under every layout, and each recall prints as a comparable QUERY / SCOPE / ANSWER block, so you can see exactly what each layout changes: the unfiltered pile blends promise with spec, node sets scope a query to one group while still answering cross-domain questions, and datasets make leakage impossible. The run ends with the rule of thumb the demo exists to teach: node sets are tags, datasets are walls.

The complete runnable demo — the script plus the four data files it reads — is examples/demos/organizing_your_data/ — this page walks through its key moments rather than reproducing it.

The three sections below cover one layout each: the calls that build it, the output those calls produce in a real run, and what the layout is good and bad at. Output excerpts are verbatim from one run, trimmed with .... Every ANSWER is a live LLM call, so the wording varies from run to run; each ANSWER line is prefixed with the dataset it came from, so the same question can be compared across layouts by eye.

Features in Play

• NodeSets — tags each source with soft, overlappable groups at write time; node_name scopes retrieval to a tag at read time
• Datasets — the hard boundary with its own permissions and storage that keeps sales transcripts out of technical answers entirely
• Remember — ingests the four bundled files under each layout’s labels
• Recall — runs the same questions under every layout, scoped by datasets and node_name

Everything in One Dataset

The baseline, and the layout that produces the complaint the page opens with. All four files land in the default dataset with no labels, and one recall runs with no filter.

    await cognee.remember(
        TECH_DOCS + [SALES_CALL_INITECH, SALES_CALL_HOOLI], self_improvement=False
    )

    answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION)

Docs and sales calls share one retrieval pool, so the rate-limit question retrieves from the API reference and the transcripts alike. In this run the model resolved the conflict in favor of the documentation — but nothing in the query kept the rep’s promise out of the pool. The NOTE line names the competition to watch for.

==============================================================================
(1) ALL DATA IN ONE DATASET — the setup that confuses the LLM
==============================================================================

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : everything (no filter)
  NOTE  : docs and sales calls share one retrieval pool — the documented limit (100/min) competes with the rep's 'unlimited requests' promise
  ANSWER: [main_dataset] - Default: 100 requests per minute per API key.
          - Enterprise: can be raised to 500 requests per minute per key.
...
          - No plan offers unlimited requests.

Pros

• The simplest layout: one remember() call, no labeling decisions, nothing to keep consistent later.
• Every fact can link to every other, so any question can draw on the whole corpus.

Cons

• There is no way to scope a query. Every recall sees everything, and sources that disagree compete in one retrieval pool.
• Whether the answer favors the spec or the promise is left to the LLM, so a correct answer in one run is not a guarantee for the next.
• One set of permissions for the whole pile: you cannot let one audience read the transcripts and another only the docs.

Use it for

• A corpus that covers one domain for one audience, where every question should see everything.
• Prototypes and small personal memories, before you know which groupings matter.

One Dataset with Node Sets

The same dataset and the same graph, but every item now carries node-set tags. Tags are soft and overlappable: everything is also tagged acme_api, so a shared grouping cuts across the docs/calls split — something no folder-style hierarchy could express.

    # An item can carry several tags at once: everything here is also tagged
    # "acme_api", so a shared grouping cuts across the docs/calls split.
    await cognee.remember(TECH_DOCS, node_set=["tech_docs", "acme_api"], self_improvement=False)
    await cognee.remember(
        [SALES_CALL_INITECH, SALES_CALL_HOOLI],
        node_set=["sales_calls", "acme_api"],
        self_improvement=False,
    )

Tagging’s effect shows the moment a recall is scoped to one tag. node_name restricts retrieval to the tagged subgraph, so the same question now answers from the documentation alone. The sales calls are still in the graph — they just don’t participate in this query.

    docs_answer = await cognee.recall(
        RATE_LIMIT_QUERY,
        query_type=SearchType.HYBRID_COMPLETION,
        node_name=["tech_docs"],
    )

Scoped to tech_docs, the answer is the documented limit. Scoped to sales_calls with the same call shape, it is what the rep said, with no mention of the spec.

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : node_name=['tech_docs']
  NOTE  : the documented truth only
  ANSWER: [main_dataset] - Default: 100 requests per minute per API key.
          - Enterprise: can raise to 500 requests per minute per key.
...
          - No plan offers unlimited requests; provision multiple keys to shard traffic.

  QUERY : What did we promise customers about rate limits?
  SCOPE : node_name=['sales_calls']
  NOTE  : what the reps actually said
  ANSWER: [main_dataset] We promised to remove the 100 requests/minute cap for enterprise deals and to allow unlimited (no throttling) requests during the pilot — the AE said they'd include this in the proposal deck.

The payoff of tags over walls is the question you ask with no filter at all. Because docs and calls share one graph, entities from both link to each other, and an unfiltered recall can draw on both sides.

    # The graph is still ONE graph — cross-domain questions work when you
    # want them to, by simply not filtering.
    cross_query = "Where do our sales promises contradict the technical documentation?"
    cross_answer = await cognee.recall(cross_query, query_type=SearchType.GRAPH_COMPLETION)

The deliberately unscoped recall lines up the sales promises against the API reference and cites both sources. This is the question a dataset wall would make impossible.

  QUERY : Where do our sales promises contradict the technical documentation?
  SCOPE : everything (no filter, on purpose)
  NOTE  : a cross-domain question spanning both groups
  ANSWER: [main_dataset] Short answer — two direct contradictions:

          - Enterprise “remove rate limits entirely” (sales AE told Initech this) vs. the API docs stating Enterprise can raise the per-key limit only to 500 req/min and “No plan, including Enterprise, offers unlimited requests.” ...

          - “Unlimited requests during the pilot / we won’t throttle you” (AE promised, and agreed to put it in the proposal deck) vs. the API docs stating rate limits are enforced per API key ...

Pros

• Scoped and cross-domain questions both work: node_name narrows a recall to one group, and leaving it out lets the graph connect the groups.
• Tags overlap. One item can belong to several groups at once, like tech_docs and acme_api here.
• Cheap to adopt: the same remember() call with a node_set argument, no new datasets or permissions to manage.

Cons

• The boundary is soft. An unscoped recall still sees everything, so keeping the promise out of the spec is the caller’s job on every query.
• Tags are assigned at write time, so a new grouping means re-ingesting the affected items.
• No separate permissions, storage, or forget per tag — everything still lives in one dataset.

Use it for

• Mixed content that shares entities you still want linked, like documentation and call transcripts about the same product.
• Memory that has to answer both narrow questions (“what does the spec say?”) and cross-cutting ones (“where do the two disagree?”).
• Slicing one team’s memory by topic, source, or account without splitting it up.

Separate Datasets

The third layout gives each domain its own dataset. Graphs are built independently, permissions and storage are separate, and either side can be forgotten without touching the other. Node sets still work inside each dataset — here the sales dataset tags each call by account.

    await cognee.remember(
        TECH_DOCS,
        dataset_name="tech_docs",
        node_set=["api_reference"],
        self_improvement=False,
    )
    await cognee.remember(
        SALES_CALL_INITECH,
        dataset_name="sales_calls",
        node_set=["initech"],
        self_improvement=False,
    )
    await cognee.remember(
        SALES_CALL_HOOLI,
        dataset_name="sales_calls",
        node_set=["hooli"],
        self_improvement=False,
    )

The two mechanisms compose: datasets picks the wall, node_name picks the tag inside it, and the answer covers one account’s calls only.

    initech_query = "What was discussed with Initech?"
    initech_answer = await cognee.recall(
        initech_query,
        query_type=SearchType.HYBRID_COMPLETION,
        datasets=["sales_calls"],
        node_name=["initech"],
    )

The [tech_docs] prefix on the first answer shows it came from that dataset alone — the sales calls cannot leak in. The combined datasets plus node_name filter narrows the second answer to the Initech call.

==============================================================================
(3) SEPARATE DATASETS (tech vs sales) — hard walls, node sets inside
==============================================================================

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : datasets=['tech_docs']
  NOTE  : sales calls cannot leak in
  ANSWER: [tech_docs] - Default: 100 requests per minute per API key.
          - Enterprise: up to 500 requests per minute per API key.
...
          - No plan offers unlimited requests.

  QUERY : What was discussed with Initech?
  SCOPE : datasets=['sales_calls'] + node_name=['initech']
  NOTE  : one account's calls only
  ANSWER: [sales_calls] They discussed Initech’s bursty batch jobs and that the 100 requests/min cap would be a blocker; Acme offered to remove rate limits for an enterprise deal and provide unlimited requests during the pilot, and Initech asked that the unlimited pilot offer be included in the proposal deck.

The layout has one footgun, and the demo closes on it. recall() without datasets spans every dataset you can read, so separating data at write time is not enough — queries must opt into the scope they want.

    # Caution: recall() without `datasets` spans ALL datasets you can read —
    # separating data at write time is not enough, queries must opt into the
    # scope they want.
    unscoped_answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION)

The same rate-limit question without datasets returns one ANSWER per readable dataset, [tech_docs] and [sales_calls], so the rep’s promise is back in the result. The run then closes with the TAKEAWAY banner.

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : everything (no dataset filter)
  NOTE  : spans both datasets again — scope must be chosen per query
  ANSWER: [tech_docs] - Default: 100 requests per minute per API key.
          - Enterprise: up to 500 requests per minute per API key.
...
          - No plan (including Enterprise) offers unlimited requests.
  ANSWER: [sales_calls] By default a 100 requests per minute cap is mentioned; Acme said they can remove rate limits for an enterprise deal and offered unlimited (no throttling) requests during the pilot.

Pros

• Hard isolation: a recall scoped to one dataset can never surface content from another.
• Each dataset has its own permissions and storage, and can be forgotten independently.
• Node sets still slice inside a dataset, so you lose nothing from the previous layout within each wall.

Cons

• Graphs are built independently, so there is no cross-dataset reasoning. An unscoped recall returns one answer per dataset rather than one answer that connects them.
• The wall only holds when the query names it: recall() without datasets spans every dataset you can read.
• Dataset scoping walls datasets off only in multi-user mode, which the default Ladybug and LanceDB stack turns on automatically — keep ENABLE_BACKEND_ACCESS_CONTROL at its default or set it to true.

Use it for

• Domains that must never contaminate each other’s answers, like the spec and the sales promises here.
• Content with different access control or retention, where one audience or one lifecycle applies per dataset.
• Multi-tenant memory, with one dataset per customer or team.

Run It

uv run python examples/demos/organizing_your_data/organizing_your_data_demo.py

Run it from a checkout of the cognee repo with an LLM provider configured: the script reads its two documents and two transcripts from the sibling data/ folder, and every answer is a live LLM call. Ladybug (the default) and Neo4j both support node-set filtering out of the box. The script calls cognee.forget(everything=True) before each of its three sections, so point it at a scratch instance rather than memory you want to keep — see Forget.

Choosing a Layout

Start with the simplest layout that answers your queries cleanly. One dataset with no labels is fine while the corpus covers a single domain for a single audience. Reach for node sets the moment distinct content types share entities you still want linked — scoped questions stay clean, and cross-domain questions keep working. Move a domain into its own dataset when it must never contaminate the other’s answers, or when it needs different access control or retention — and remember the two compose, so datasets can carry node sets inside them.

NodeSets

How node-set labels are written and how node_name filters retrieval by them.

Datasets

What a dataset isolates — documents, permissions, and processing.

NodeSet Grouping

The minimal single-feature guide to tagging memories with node sets.

Agentic Procurement Decisions

A sibling demo where node-set scoping powers an agent’s research phase.