Gmail Ingestion

Sync Gmail messages into Cognee memory incrementally, with forget-on-delete

A minimal guide to turning your Gmail inbox into Cognee memory. gmail_source() returns a dlt resource you pass straight to remember(): the first sync loads the messages in a label, later syncs fetch only what changed, and messages you delete in Gmail are forgotten on the next sync.

This reads the content of your email. Nothing is fetched until you run the script. Scope what you ingest with label_ids, keep token.json private, and use a dedicated dataset so you can wipe it with a single cognee.forget.

Before You Start

• Complete Quickstart to understand basic operations
• Ensure you have LLM Providers configured (LLM_API_KEY in .env)
• Read dlt Integration for how structured sources flow through remember()
• Install the Gmail extra: pip install "cognee[gmail]"
• In the Google Cloud Console, enable the Gmail API, configure an OAuth consent screen (add yourself as a test user), and create an OAuth 2.0 Client ID of type Desktop app
• Save the downloaded client-secret JSON as credentials.json next to the script, or point GMAIL_CREDENTIALS_PATH at it (GMAIL_TOKEN_PATH sets where the token is cached, default token.json). The first run opens a browser to consent

Code in Action

import asyncio
import os

import cognee
from cognee.tasks.ingestion.connectors import gmail_source

# Keep the inbox in its own dataset so it is easy to inspect and forget.
DATASET_NAME = "gmail_inbox"

# Routing kwargs for remember(). Pass the same ones on every later sync.
#   write_disposition="merge" is REQUIRED: the add pipeline defaults to
#     "replace", which would wipe the whole synced inbox on the next sync.
#   max_rows_per_table=0 guarantees no per-table read cap applies (even if
#     DLT_MAX_ROWS_PER_TABLE is set), so orphan-cleanup (forget-on-delete)
#     compares against the *entire* synced corpus.
#   incremental_loading is left at its default (True) so a re-sync only builds
#     the graph for new or changed messages instead of the whole inbox again.
GMAIL_REMEMBER_KWARGS = {
    "primary_key": "id",
    "write_disposition": "merge",
    "max_rows_per_table": 0,
    "self_improvement": False,
}


async def main():
    credentials_path = os.environ.get("GMAIL_CREDENTIALS_PATH", "credentials.json")
    token_path = os.environ.get("GMAIL_TOKEN_PATH", "token.json")

    if not os.path.exists(credentials_path):
        print(
            f"Gmail OAuth client secrets not found at '{credentials_path}'.\n"
            "See the setup steps in this file's docstring, then re-run."
        )
        return

    # Start from a clean slate so the demo is reproducible.
    await cognee.prune.prune_data()
    await cognee.prune.prune_system(metadata=True)

    # Build the source. Scope it to INBOX and, for the demo, load only the 25
    # newest messages so the first run is quick. Drop ``max_results`` to load
    # the whole label; Gmail's quota allows roughly 250 messages a minute.
    source = gmail_source(
        credentials_path=credentials_path,
        token_path=token_path,
        label_ids=["INBOX"],
        max_results=25,
    )

    # ── Sync: load the newest messages ────────────────────────────────────
    print("\n=== Gmail sync ===")
    result = await cognee.remember(
        source,
        dataset_name=DATASET_NAME,
        **GMAIL_REMEMBER_KWARGS,
    )
    print(result)
    print("Sync stats:", source.cognee_sync_stats)

    answer = await cognee.recall(
        query_text="Summarize the most important emails in my inbox.",
        query_type=cognee.SearchType.GRAPH_COMPLETION,
        datasets=[DATASET_NAME],
    )
    print(answer[0].text)


if __name__ == "__main__":
    asyncio.run(main())

What Just Happened

Step 1: Configure the Sync

GMAIL_REMEMBER_KWARGS = {
    "primary_key": "id",
    "write_disposition": "merge",
    "max_rows_per_table": 0,
    "self_improvement": False,
}

These are the routing options for remember(); pass the same ones on every later sync. write_disposition="merge" upserts messages by their Gmail id instead of replacing the whole dataset on each sync. max_rows_per_table=0 only makes the intent explicit: Gmail is a document source, and Cognee always reads a document source’s whole staging table, so forget-on-delete compares against the entire synced inbox whatever this or DLT_MAX_ROWS_PER_TABLE says. incremental_loading stays at its default (True), so a later sync only builds the graph for new or changed messages, and self_improvement=False skips the automatic Improve step after each sync.

Step 2: Build the Gmail Source

    source = gmail_source(
        credentials_path=credentials_path,
        token_path=token_path,
        label_ids=["INBOX"],
        max_results=25,
    )

gmail_source() authenticates with your OAuth client secrets and returns a dlt resource scoped to the INBOX label. If credentials.json is missing, the script prints a message and exits; the setup steps it points to are the ones in Before You Start. max_results=25 loads only the 25 newest messages so the demo runs quickly. To load everything, see Loading Your Whole Inbox.

Step 3: Sync and Ask Your Inbox

    result = await cognee.remember(
        source,
        dataset_name=DATASET_NAME,
        **GMAIL_REMEMBER_KWARGS,
    )
    print(result)
    print("Sync stats:", source.cognee_sync_stats)

    answer = await cognee.recall(
        query_text="Summarize the most important emails in my inbox.",
        query_type=cognee.SearchType.GRAPH_COMPLETION,
        datasets=[DATASET_NAME],
    )
    print(answer[0].text)

The sync pulls the messages into the gmail_inbox dataset and builds the graph, which you then query with recall() using GRAPH_COMPLETION, restricted to that dataset. answer[0].text is the generated answer. source.cognee_sync_stats counts what this sync did: scanned messages fetched from Gmail, deleted messages forgotten, and skipped / failed fetches.

What Gets Ingested

Each message becomes one document. The subject is its title, the headers worth searching are folded into the text, and the body follows after a blank line:

# Q3 planning notes

From: Ada <[e-mail]>
To: [e-mail]
Cc: [e-mail]
Date: Tue, 15 Jul 2025 09:12:03 +0200
Received: 2025-07-15T07:12:03+00:00
Labels: INBOX, IMPORTANT
Thread: 1980b0f2c4a1d3e7

Here are the notes from this morning...

• Received is Gmail’s internalDate as a UTC ISO timestamp, alongside the sender’s own Date header.
• Empty fields are left out. A message with no Cc has no Cc: line; a message with no subject gets no # heading.
• The body is the message’s first text/plain part, found anywhere in the MIME tree. HTML-only mail has none, so those messages fall back to Gmail’s snippet — a short preview rather than the full text. That keeps raw markup out of entity extraction, but expect HTML newsletters to be represented thinly.

This is the text recall() searches and graph extraction runs on, so anything not in it — attachments, HTML bodies, other headers — can’t be asked about.

Syncing Again

To pick up new mail, run the same gmail_source() and remember() again later, for example once a day. You don’t choose how to sync; the connector decides from what it saved last time:

• No saved cursor (the first sync): it lists the label from the newest message down and fetches each one. If the run wasn’t capped with max_results and finished, it saves Gmail’s historyId as a cursor.
• Saved cursor: it asks Gmail’s History API what changed since the cursor. Only new or changed messages are fetched, and messages you deleted or trashed are forgotten. If nothing changed, no messages are downloaded.

Cognee then builds the graph only for the messages that are new or changed. Messages it already processed are skipped, because incremental_loading is on by default. If you pass incremental_loading=False, every sync re-runs graph extraction on every message in the dataset.

source = gmail_source(label_ids=["INBOX"])
await cognee.remember(source, dataset_name="gmail_inbox", **GMAIL_REMEMBER_KWARGS)
print(source.cognee_sync_stats)  # e.g. {'scanned': 3, 'skipped': 0, 'failed': 0, 'deleted': 1, ...}

The cursor only carries over when:

• the previous sync ran without max_results and finished. An interrupted sync saves nothing, so the next one starts over.
• later syncs use the same dataset_name and label_ids. Changing the labels loads the new selection from scratch.
• you don’t delete the connector’s saved state. The cursor lives in dlt’s pipeline state under ~/.dlt/pipelines/.
• you sync at least about once a week. Gmail expires history after roughly a week; with an expired cursor the connector loads every message in the label again instead of stalling.

prune doesn’t reset a sync. cognee.prune wipes Cognee’s memory, but not the saved cursor or dlt’s staged copy of your mail (the dlt_database_<dataset> staging database). The next sync reads that copy back, so every previously synced message returns to memory and goes through graph extraction again. Leave the script’s prune calls out of real syncs.

Capped syncs never switch to incremental. A run with max_results always starts from the newest message and stops after that many. It doesn’t remember where it stopped, so the next capped run starts from the top again instead of continuing to older mail. Messages you already have are requested from Gmail again and count against your quota, but they aren’t processed into the graph again. A capped run also saves no cursor and never forgets deleted mail.

Loading Your Whole Inbox

Drop max_results and let one sync run to completion:

source = gmail_source(label_ids=["INBOX"])
await cognee.remember(source, dataset_name="gmail_inbox", **GMAIL_REMEMBER_KWARGS)

• It takes a while. Gmail allows 6,000 quota units per user per minute, and fetching one message costs 20. The connector paces itself to 5,000 units a minute by default, about 250 messages, so 10,000 messages take around 40 minutes to download. Lower the budget with gmail_source(quota_units_per_minute=...) if other apps use the same account.
• Every message goes through graph extraction, so expect at least one LLM call per message. Try a smaller label first to estimate time and cost.
• Let it finish. An interrupted sync saves no cursor, so the next run starts from the beginning.

Once it finishes, every later sync fetches only what changed, as described in Syncing Again.

Sync Behavior

• Read-only access: the connector requests only the gmail.readonly scope and never modifies your mailbox. The OAuth token is cached at token.json with owner-only (0600) permissions.
• Changing labels: passing a different label_ids selection loads the new scope from scratch, including messages older than the last sync, and forgets mail that no longer matches. Reordering the same labels keeps the cursor. A capped or interrupted sync forgets nothing.
• Foreground syncs only: deletions propagate only when remember() runs in the foreground (the default). A run_in_background=True run skips orphan cleanup.
• Messages that fail to fetch: only a 404/410 from Gmail means the message is genuinely gone. Those count under skipped in cognee_sync_stats and are forgotten like any other deletion (a capped sync forgets nothing). Anything else — a rate limit, a 5xx, a dropped connection — is retried up to six times by the Google client and then raised, which ends the sync before the cursor advances. The next run picks up from the same point instead of mistaking a transient error for a deleted message.
• History API calls that fail: a similar rule applies to the call that asks what changed since the cursor. Only a real 404 (not 410) means the cursor expired, which triggers the full reload described in Syncing Again. Anything else — again a rate limit, a 5xx, a dropped connection, after the Google client’s retries — ends the sync with the saved cursor untouched, so the next run resumes from the same point instead of reloading the whole label.

Gmail Integration

Let users connect their mailboxes through Cognee’s API instead of a local token

dlt (Data Load Tool)

How dlt sources, merge syncs, and orphan cleanup work

Remember

All remember() parameters, including the dlt options

Forget

Wipe the Gmail dataset when you are done