` and `<h1>`. Any `<`, `&`, or tag-like content in that text corrupted the rendered page (a `</td>` or `<script>` in a golden answer escaped its cell and broke the table), and made the generated report an HTML-injection sink when opened in a browser, since the text originates from benchmark data and model output rather than from Cognee. Both modules now apply `html.escape` at every interpolation point: the details-table cell values, the derived column headers (`metrics_dashboard.py`, which titles them from the item keys), the per-metric `<h3>` section heading, and the `benchmark` name in the page template. The fields this covers are the ones each module already rendered — `question`, `answer`, `golden_answer`, `reason`, and `score` in `dashboard_generator.py`, and whatever keys the metric items carry in `metrics_dashboard.py` (`question`, `answer`, `golden_answer` by default, `question` and `retrieval_context` for `contextual_relevancy`, and those two plus `golden_context` for `context_coverage` — each with `reason` and `score`). Plotly figure HTML from `create_distribution_plots()` / `create_ci_plot()` and the already-assembled `details_html` list are deliberately left unescaped as trusted, pre-rendered markup, so charts render exactly as before. **Impact:** benchmark fields that contained raw HTML previously rendered as markup and now display as literal text — there is no flag to restore the old behavior. No public API signature, configuration option, or environment variable changed, and no migration is required; regenerate a dashboard to pick up the fix (PR #4429). * **Breaking: `cognee.validate()` now rejects a dataset it cannot read instead of falling back to the default stores.** `validate()` resolved its `dataset` argument through `get_authorized_existing_datasets(..., "read", user)` and read the first result, but when *none* of the requested names resolved it left the resolved dataset as `None` and entered the database context unscoped — so with backend access control disabled a typo, an unknown name, or a dataset the caller had no `read` permission on produced a `ValidationReport` about the single shared graph and vector stores, indistinguishable from a real report about the requested dataset. Every requested name must now resolve to a dataset the caller can read; otherwise `DatasetNotFoundError` (`"Dataset not found or not readable."`, from `cognee.modules.data.exceptions`, `status_code=404`) is raised *before* any graph or vector adapter is opened, so no store is touched on the rejected path. The check compares counts, so a partially authorized list is rejected as a whole — `dataset=["mine", "not-mine"]` raises rather than silently validating `mine` — and it does not branch on `ENABLE_BACKEND_ACCESS_CONTROL`, so the rejection is identical with access control on or off. Selection is otherwise unchanged: when all names are authorized, the first still determines which graph and vector store are read, and one graph is still checked per call. **Caller impact:** the default `dataset="main_dataset"` is subject to the same rule, so a bare `await cognee.validate()` against an installation where `main_dataset` does not exist yet now raises `DatasetNotFoundError` — where previously it reported on the shared stores with access control disabled, and failed with a generic `CogneeValidationError` ("A dataset must be provided...") with it enabled; and because dataset names resolve only within datasets the caller *owns*, a dataset merely shared with the caller is rejected when requested by name. Passing `dataset=None` or an empty list still skips resolution entirely, which remains the only path that enters no dataset context — with access control disabled that reads the unscoped shared stores, while with it enabled the call still fails, since a dataset is required to resolve the per-dataset databases. **Over HTTP the rejection surfaces as a `500`, not a `404`:** `GET /api/v1/validate` wraps every exception from the route in `{"status": "error", "reason": "validation failed: ..."}` with status `500`, so the `DatasetNotFoundError` never reaches the global `CogneeApiError` handler that would map it to `404` — expect additional `500`s whose `reason` is `validation failed: DatasetNotFoundError: Dataset not found or not readable. (Status code: 404)` — the `404` in the string is the exception's own string form, the HTTP status stays `500` — rather than a rise in `4xx`, and match the `reason` for `Dataset not found or not readable.` if you alert on this. Remediation is to request a dataset the caller is authorized to read, or to catch `DatasetNotFoundError`. No configuration option, environment variable, or migration ships with the fix, and `validate()`'s signature is unchanged (fixes #4414, PR #4455, mirroring contributor PR #4416). * Adds `TELEMETRY_ORIGIN`, an environment variable that labels where a telemetry event comes from, so events can be segmented by origin. `send_telemetry` reads it with `os.getenv("TELEMETRY_ORIGIN", "sdk")` and stamps the value onto every event's `properties` payload as `telemetry_origin`; the Cognee-managed cloud sets `TELEMETRY_ORIGIN=cloud`, and everything else — local SDK usage, self-hosted servers — reports the default `sdk`. It is read directly from the environment on each event, matching how `send_telemetry` already reads `TELEMETRY_DISABLED` and `ENV`, so there is no config-schema change and it cannot be set through `cognee.config.set(...)`. The property is written before the `**additional_properties` spread, so a caller that passes its own `telemetry_origin` in `additional_properties` still overrides it per call. **Impact:** non-breaking and no action is required — telemetry payloads gain one property, and `TELEMETRY_DISABLED=true` still suppresses events entirely. No public API signature or migration changed (PR #4433). * Fixes the Redis session-cache adapter (`CACHE_BACKEND="redis"`) returning `None` instead of `[]` from `get_latest_qa_entries` for an empty or unknown session. `RedisAdapter.get_latest_qa_entries` takes a `lindex` fast path when `last_n == 1` — the hot path, since loading the previous turn of a session reads with `last_n=1` — and that branch ended in `if data else None`, contradicting the method's declared `-> list[SessionQAEntry]` return type and diverging from the SQL and FS adapters; the general `lrange` path used for every other `last_n` already returned `[]`. It now returns `[]` on the fast path as well, and the `lindex` optimization is kept. **Impact:** `cognee.session.get_session()` was not affected, because `SessionManager.get_session` carries an `if entries is None` guard that returns `""` or `[]`; the `TypeError: 'NoneType' object is not iterable` this could raise on Redis (while the same call worked on SQLite and FS) was reachable only from code calling the cache adapter directly with `last_n=1`, including through the backward-compatible `get_latest_qa` shim. No public API signature, configuration option, environment variable, or migration changed, and no action is required on upgrade (fixes #3930, PR #3931). * Fixes the Ladybug and Amazon Neptune graph adapters reporting a *failed* batch edge-existence check as an *empty* one, which could let a cognify run finish reporting success while persisting nothing. `has_edges(edges)` takes a list of `(source_id, target_id, relationship_name)` triples and returns the subset that already exists in the graph; `LadybugAdapter.has_edges` (`GRAPH_DATABASE_PROVIDER="ladybug"`, the default backend) and `NeptuneGraphDB.has_edges` (`GRAPH_DATABASE_PROVIDER="neptune"`) both wrapped the query in `except Exception`, logged the error, and returned `[]`. That value is indistinguishable from a genuine "none of these edges exist yet" answer, and the adapter method's only production caller is the cognify dedup step — `find_existing_edge_identities` in `cognee/modules/graph/utils/retrieve_existing_edges.py`, called from `extract_graph_from_data` to subtract already-stored edges before writing the rest. So when the store was unavailable or corrupt — a corrupt write-ahead log after an unclean shutdown, for example — the check answered "nothing exists", every extracted edge was treated as new and written, those writes failed against the same broken store, and the run completed without raising while the graph stayed empty. Both adapters now log and then re-raise, which is what the other backends already did: `neo4j` logs and re-raises `Neo4jError`, and `postgres` and `turso` never caught the error in the first place. Ladybug's empty-**input** short-circuit is unchanged — `has_edges([])` still returns `[]` without touching the store — and Neptune, which has no such short-circuit, is unchanged in that respect too. Only the batch `has_edges` path was touched; the single-edge `has_edge` and every other adapter method behave as before. **Impact:** no public API signature, configuration option, environment variable, or migration changed, and a run against a healthy graph store behaves exactly as it did. What changes is the failure mode: an ingest that previously "succeeded" against a broken store now fails with the backend error propagated to the caller, so a job that was silently a no-op may begin surfacing as a failure in logs and CI after upgrading. That failure reports a problem that was already present rather than introducing a new one, so the remedy is to fix or restore the graph store, not to suppress the error. This covers the silent-data-loss existence check in the linked issue; the startup write-ahead-log recovery and `SIGTERM` handling also requested there are a separate change and are not included. Unit tests in `cognee/tests/unit/infrastructure/databases/test_has_edges_error_propagation.py` cover the three cases — a successful check returning the existing-edge tuples, a query failure raising instead of returning `[]`, and empty input short-circuiting without reaching the store (fixes #4348, PR #4430). * Gives each relational [dlt](/integrations/dlt-integration) source a stable identity, so re-ingesting one updates it in place instead of piling up copies. A source is now stored as a single record keyed on `dlt_source:{dataset_name}:{source_name}` — an identity that does not include the data — with change detection carried separately by a content hash over the source's tables and rows. A plain re-run of `add()` / `remember()` on an already-ingested source is idempotent and skips it without reprocessing, whether or not the data changed; to pick up upstream changes, re-ingest explicitly with `add(..., incremental_loading=False, data_cache=False)` (the completed-skip runs whenever either flag is on) or `update()` with the record's UUID. The record then updates in place under the same identity (so the source is never missing from the store between runs), and the next `cognify()` purges that source's previously derived graph nodes, edges, and vectors before re-emitting the current rows, so upstream deletions and edits no longer leave stale rows behind. Two further changes ship alongside it: dlt row embeddings move out of the shared `DocumentChunk_text` collection into their own `DltRow_text` collection — graph-completion retrieval reads both, but chunk search is now documents-only — and the `DLT_MAX_ROWS_PER_TABLE` default changes from `50` to `0`, which means no cap, so a source that was previously truncated to 50 rows per table is now ingested in full unless you set a positive value (via the env var or the `max_rows_per_table` kwarg). **Impact:** three things to plan for on upgrade. First, reprocessing a *changed* dlt source now requires `delete` permission on the dataset, because the purge is re-authorized as a delete; a run without it fails loudly rather than quietly serving stale rows, so check the permissions of any automation that re-syncs dlt sources. Second, dlt data ingested before this change was stored as pre-manifest per-row records, which are no longer supported — `cognify()` raises on them with a message naming the record, and re-adding the source ingests it once as a manifest (and sweeps the legacy records away), so expect a one-off re-ingest of existing dlt datasets. Third, renaming a dlt source or its dataset changes the identity and is therefore a remove + add: the new name ingests fresh and the old name's records remain until you delete them. No public API signature changed, and `primary_key`, `write_disposition`, `query`, and `max_rows_per_table` are accepted exactly as before (COG-2222, PR #4278). * Fixes background `remember()` work being destroyed by Python's garbage collector mid-flight, so the call reported success while the knowledge graph was never updated. Both fire-and-forget paths were affected: the `run_in_background=True` run that performs `add()` followed by `cognify()`, and the session-to-graph bridge that `remember(session_id=...)` starts when `self_improvement` is on — it defaults to `True`, so every `session_id` call took that path. Each created its task with a bare `asyncio.create_task(...)` and kept the only application-level reference on the returned `RememberResult`. The event loop holds just a weak reference to a running task, so once the caller dropped that result — as `POST /api/v1/remember` does, serializing the result and discarding the object — the surviving task/closure/result references formed a closed cycle with nothing outside it, and the cycle collector could collect the pending task, leaving `Task was destroyed but it is pending!` in the logs as the only trace. Both tasks are now held in a module-level anchor set for their whole lifetime and remove themselves on completion via `add_done_callback`, the same pattern already used for background sync (`cognee/api/v1/sync/sync.py`) and for background pipeline runs (`cognee/modules/pipelines/layers/pipeline_execution_mode.py`). **Who is affected:** callers that start background remember work and do not keep the returned `RememberResult` alive — the HTTP endpoint, and SDK code that calls `remember(..., run_in_background=True)` or `remember(session_id=...)` without holding the result. A caller that kept the result and awaited it was never at risk, and blocking `remember()` calls were never affected. The loss depended on a collection cycle landing at the wrong moment, so an affected deployment saw it intermittently rather than on every request; runs that vanished this way were never recorded as failures, so re-run any background ingestion whose data is missing from the graph. The documented behavior of `remember()` is unchanged — background mode still returns immediately and the result can still be awaited later — it is simply now reliable, and no public API signature, configuration option, environment variable, or migration changed, so deploying the fix is the whole action (fixes #4312, PR #4456). * Fixes `GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw` rejecting a caller who holds a dataset-level `read` grant but does not own the data item. The handler already authorized the dataset with `get_authorized_existing_datasets([dataset_id], "read", user)`, then resolved the item with `get_data(user.id, data_id, dataset[0].id)` — and `get_data` re-checks `data.owner_id == user_id`, raising `UnauthorizedDataAccessError` (HTTP **401**) when they differ. Since a document's `owner_id` is the principal that ingested it, anyone reading a dataset shared with them through an ACL could list its items but got a 401 on every raw download, even though the dataset permission check had already passed. The handler now resolves the item entirely within the authorized dataset: `resolve_data_id(dataset[0].id, data_id)` maps the caller-supplied id to the canonical row id (falling back to the recorded pre-fork `legacy_id`, so ids issued before the dataset-scoping upgrade keep resolving, exactly as before), and the row is then taken from `get_dataset_data(dataset[0].id)`, which is scoped by dataset membership and does not consult `owner_id`. **Impact:** dataset `read` is now sufficient to download raw files — a request that previously returned 401 for a non-owning reader returns the file. Nothing else about the endpoint changed: it is still gated on `read` for the containing dataset, an id that is unknown in that dataset still returns 404, and the owner's own downloads, the S3 streaming path, the local-file path, and the 501 response for unsupported storage schemes all behave as before. One adjacent change: when the dataset itself is not found or not readable, the 404 body's key is now `message` instead of `detail`, so a client reading `detail` off that specific response needs updating (fixes #4162, COG-5923, PR #4468, mirroring contributor PR #4200). * Fixes file-extension detection being case-sensitive, so files named `data.CSV`, `NOTES.MD`, `config.YAML`, `payload.JSON`, or `feed.XML` were ingested as plain text. `guess_file_type` (`cognee/infrastructure/files/utils/guess_file_type.py`) took the extension from the file name and compared it against literal lowercase lists — `.txt`/`.text`, `.csv`, `.md`/`.markdown`, `.json`, `.xml`, `.yaml`/`.yml` — but `Path("data.CSV").suffix` is `".CSV"`, which matches none of them. Those are exactly the formats that carry no magic-number signature, so an unmatched uppercase extension fell through to `filetype.guess`, which returned `None`, and the file was labeled `text/plain` with extension `txt`. The extension is now lowercased before the lookup, so case no longer affects the result. **Impact:** two things downstream change for files whose extension was not already lowercase. `get_file_metadata` stores the guessed mime type and extension, and `classify_documents` selects the document class from that extension — so a `.CSV` file is now a `CsvDocument`, whose reader expands each row into `key: value` text, instead of a plain `TextDocument` chunked as prose; `.MD`, `.JSON`, `.XML`, and `.YAML` already mapped to `TextDocument` and keep that class, but the `mime_type` and `extension` recorded for them are now the real ones (`text/markdown`/`md` rather than `text/plain`/`txt`). Loader selection also shifts: `LoaderEngine.get_loader` tries the raw path extension first and does *not* lowercase it, so the corrected content-detected extension is what takes effect on the fallback branch, and `data.CSV` now reaches the CSV loader path — `CsvLoader`, or the DLT CSV loader when the `dlt` extra is installed — rather than `TextLoader`. Magic-number formats (PDF, images, audio, video) were already case-independent and are unaffected, and an uppercase `.TXT` still resolves to `text/plain` exactly as before. No public API signature, configuration option, environment variable, or migration changed. Data already ingested keeps the type it was stored with — re-ingest affected files to pick up the corrected type. Unit tests in `cognee/tests/unit/infrastructure/files/utils/test_guess_file_type.py` assert correct detection for lower- and uppercase csv/md/json extensions and uppercase XML/YAML, plus the unchanged `.TXT` case (PR #4453). * Adds an opt-in, audit-grade provenance ledger: an append-only, tamper-evident record of the document → chunk → entity → relationship lineage each ingestion produced. This is the migration `b8c1d3e5f7a9` listed above — it creates the single `provenance_entries` table (SQLite and Postgres) that backs the feature, and it is idempotent, so it is a no-op where the table already exists. **Default off** — with `PROVENANCE_TRACKING` unset, the cognify task list and behavior are identical to before, nothing is written to the new table, and no public API signature changed. Enabling it (`PROVENANCE_TRACKING=true` on `CognifyConfig`, `cognee/modules/cognify/config.py`) splices a `record_provenance` task (`cognee/tasks/provenance/record_provenance.py`) into the default pipeline right after `add_data_points` — where node ids are persisted and stable — and before the contradiction-detection spread, committing all of one invocation's entries as a single chained transaction. The task returns its input unchanged and swallows all of its own errors, logging a warning rather than failing the run, so provenance can never break ingestion; DataPoints from custom `graph_model` schemas that lack the default `made_from`/`is_part_of`/`contains` shape are covered by a generic walk using the same traversal `add_data_points` uses. Every entry carries a SHA-256 checksum over its canonical JSON plus the previous entry's checksum, linked by a unique `sequence_id`, so tampering — deletion, reordering, or a single-field edit — is detectable. Ledger keys are prefixed with the dataset id, because cognee entity ids are deterministic and the ledger lives in the shared relational database, so two datasets mentioning the same entity name keep separate version chains. A new `ProvenanceManager` (`cognee.modules.provenance.get_provenance_manager()`) is the programmatic surface: `track_entity`, `track_chunk`, `track_relationship`, `get_provenance`, `get_lineage`, `trace_lineage`, `revision_history`, `invalidate`, `verify_chain`, `check`, and `get_statistics`. All entries share one `sequence_id` sequence and each commit takes a ledger-wide write lock (a Postgres advisory lock, so it serializes across processes), which is why one task invocation commits as one transaction — concurrent runs queue behind each other for their ledger writes. A batch that fails rolls back entirely, so its entries never claim sequence numbers and the chain still verifies as intact: `verify_chain()` proves the stored entries were not tampered with, not that everything an ingestion produced was recorded. **Operator note:** enabling this adds relational-DB writes to every `cognify()` run, so plan for disk and backup sizing and for the cost of periodic verification; `verify_chain()` streams the ledger by keyset pagination instead of materializing it client-side, and `get_statistics()` aggregates DB-side, so both are safe to run against a large table — `check()`, the referential-integrity pass, also streams its rows but first loads every entry id and activity id into memory, so its footprint grows with the ledger. See [Provenance ledger](/python-api/cognify#provenance-ledger) (COG-6172, PR #4476). *** ## v1.4.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.2)** Release that bumps the package version from `1.4.1` to `1.4.2`. No new Alembic revision ships in this cut, so no migration is required on upgrade. The entries below are the work promoted from the development branch into this release; work logged under the v1.4.1.dev0 and v1.4.1.dev1 pre-release sections below — including the JSON visualization endpoints (PR #4331) and the cancelled-request session cleanup (PR #4250) — also ships in this release. ### Highlights * Names raw-text uploads made over a `serve()` connection by content hash. The remote client sent every raw string passed to `remember()` or `add()` — including each string inside a list — under the fixed filename `data.txt`, so all text uploads for a tenant collided on a single remote object and concurrent adds raced the server's content-hash read-back, failing with `FileContentHashingError` 409s. The client now derives the name from the text itself as `text_<md5_hash>.txt`, reusing the same `TextData` namer local ingestion applies to nameless text, so a given string gets the same object name whether it is ingested locally or remotely. File-like uploads are unchanged and still use their own `name` (falling back to `upload`), and the uploaded text content is unchanged. **Operator impact:** tooling or tests that assert the uploaded basename is `data.txt` need updating to expect `text_<md5_hash>.txt`; the client exposes no way to supply your own filename. No public API signature, configuration option, or environment variable changed (PR #4366). * Clarifies that the Postgres graph store is a demo feature and states where to reach us about production use. The `postgres` graph backend's module and adapter docstrings, and the `graph_database_provider == "postgres"` branch in `get_graph_engine.py`, now say explicitly that it is not production-ready and that production workloads should use a graph-native backend such as Kuzu or Neo4j; the production-ready adapter is a licensed product, and the contact route is now `social@cognee.ai` (or a call with the sales team via [cognee.ai](https://www.cognee.ai)) rather than the previously documented `social@topoteretes.com`. This is a documentation and positioning change only — the adapter's behavior, its `GRAPH_DATABASE_*` configuration and fallback to the relational `DB_*` settings, and its existing limitation that `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` raise `SearchTypeNotSupported` are all unchanged (PR #4366). * Fixes a connection-pool deadlock in the API-key authentication path that could take the whole API down under concurrent load — surfacing first as intermittent `401`s and gateway timeouts, then as a full outage, with Postgres backends stranded `idle in transaction`. `UserManager.get_by_token()` in `cognee/modules/users/get_user_manager.py` looked up the `UserApiKey` row in its own session and then resolved the user with `self.get()` **while that session was still open**; `self.get()` borrows the request-scoped session behind the FastAPI `get_user_db` dependency, which is held until the response is finished, so it checked out a *second* pooled connection and kept it open for the entire request — across the slow completion call. Every in-flight authenticated request therefore pinned two pooled connections instead of one: a pool of N slots could safely serve only about N/2 concurrent authenticated requests, and at N concurrent requests every slot was held by an API-key lookup waiting for a second connection that could never free — a circular wait that deadlocked the pool (with the relational defaults of `pool_size=5` plus `max_overflow=35`, that ceiling is 40). `get_by_token()` now resolves both rows with two `select()` calls inside a single short-lived session and returns from inside it, never calling `self.get()`, so authentication holds one connection and releases it as soon as the lookup finishes. This is a **separate cause from the cancelled-request cleanup fix logged under v1.4.1.dev0** (PR #4250): the deadlock here reproduces with plain concurrency and no cancellation involved, which is why that earlier change did not resolve #4197. Two workarounds operators may have reached for did not address it and can be reconsidered — enlarging `POOL_ARGS` only raised the concurrency at which the deadlock hit, and `idle_in_transaction_session_timeout` never reclaimed these slots, because they belong to live, in-flight requests rather than abandoned ones. No public API signature, configuration option, or environment variable changed and no migration is required; redeploy to apply (fixes #4197, PR #4354). * Keeps idle subprocess-backed database engines (embedded Ladybug/Kuzu graph, LanceDB vector) alive for a configurable idle TTL instead of closing them every time a dataset context exits. Previously, the last holder of a dataset's queue slot evicted and force-closed the cached engine on release, so a follow-up request for the same dataset paid for a worker close plus a fresh spawn. The release path now refreshes the engine's idle timestamp, and a background daemon thread (`subprocess-idle-reaper`, started lazily on the first kept-alive release) sweeps the engine caches every `max(5, min(60, TTL / 4))` seconds and force-closes only the engines that have gone a full TTL without use. The sweep skips datasets holding an active queue slot — the same pin that protects them from capacity eviction — so an operation running longer than the TTL cannot lose its engine mid-flight, and it skips engines that are not subprocess-backed, so remote stores (Neo4j, Postgres, PGVector) are unaffected and keep plain LRU behavior. The TTL is set with the new `SUBPROCESS_IDLE_TTL_SECONDS` environment variable, default `600` seconds; negative values are clamped to `0` and fractional values are accepted. **Operator impact:** a kept-warm worker holds its PID, its memory, and — for the file-based graph store — its database file lock for up to the TTL after its last use, so worker PIDs are now stable across requests and a data directory can stay locked longer than before (relevant when another tool or backup job needs to open the same files). The number of retained idle workers is still bounded by the engine cache capacity (`DATABASE_MAX_LRU_CACHE_SIZE`). Set `SUBPROCESS_IDLE_TTL_SECONDS=0` to restore the previous close-at-release behavior; the keep-alive also has no effect when the dataset queue is disabled (`DATASET_QUEUE_ENABLED=false`), because the release path never runs (PR #4384). * Fixes the remaining code paths that held one pooled relational connection while acquiring a second, which could deadlock a bounded Postgres or PGVector pool. This is a different mechanism from the cancellation leak fixed in #4250: nothing was abandoned mid-transaction, the connections simply *overlapped* — a method opened a session (or an `engine.begin()` connection) and then, inside it, called a helper that checks out a connection of its own, so each such call pinned two at once. Once concurrency reached the pool's ceiling, every waiting task held one connection and waited for another that could not come free, and the stuck backends showed up as `idle in transaction`. The default embedded SQLite store has no bounded shared pool, so this only affected Postgres/PGVector deployments. All affected paths were reordered to resolve the lookup first and open the working session second: in `SQLAlchemyAdapter` (`cognee/infrastructure/databases/relational/sqlalchemy/SqlAlchemyAdapter.py`), `insert_data`, `delete_entity_by_id`, `get_all_data_from_table`, `get_table_names`, and `extract_schema` now resolve `get_table()` / `get_table_names()` / `get_schema_list()` before opening their own connection, since each of those helpers checks one out; `create_role`, `create_tenant`, `add_user_to_tenant`, `select_tenant`, `get_default_user`, `get_deletion_counts`, and `get_document_ids_for_user` resolve their user, tenant, and per-dataset lookups outside the session that does the work; and `get_pipeline_run_metrics` computes its token count on the session it already has open instead of calling `fetch_token_count`, which would have opened a second one. `delete_role`, `remove_user_from_role`, and `get_users_in_role` — where the permission check depends on data read from the first session — split into two sequential sessions with the check running on its own connection in between; the entity lookup still precedes the permission check, so `EntityNotFoundError` / `UserNotFoundError` / `RoleNotFoundError` still take precedence over a permission failure exactly as before, and `get_default_user`'s deferred `create_default_user()` call stays inside the same error handling, so a missing schema still surfaces as `DatabaseNotCreatedError`. Every method keeps its arguments, return value, and exceptions. No public API signature, configuration option, environment variable, or migration changed, and no pool resizing is needed — deploying the release is the whole action (PR #4392, follow-up to #4197). * Fixes the SQL session-cache engine crashing at startup when `POOL_ARGS` sets `"poolclass": "nullpool"`. `POOL_ARGS` has two consumers: the relational adapter already normalized the string `"nullpool"` to SQLAlchemy's `NullPool` class before building its engine, while `SqlCacheAdapter` read the same relational pool arguments and passed them to `create_async_engine` unchanged — SQLAlchemy inspects `poolclass` as a class, so the string raised `CacheConnectionError: Failed to initialize SQL cache engine for …: 'str' object has no attribute '__dict__'` for both `CACHE_BACKEND="sqlite"` and `CACHE_BACKEND="postgres"`. The cache adapter now applies the same normalization, so one `POOL_ARGS` value works for both consumers — which matters behind an external pooler (for example Neon's `-pooler` pgbouncer endpoints), where disabling client-side pooling is the deliberate choice and the cache engine connects to that same endpoint. **Operator impact:** none beyond upgrading — no environment variable was added or renamed, no configuration change is required, and no migration ships with the fix; deployments that leave `POOL_ARGS` unset or omit `poolclass` are unaffected, and a deployment that had to drop `poolclass` to work around the crash can restore it (PR #4376). * Fixes an explicitly configured relational `POOL_ARGS` being ignored for per-dataset PGVector engines under `ENABLE_BACKEND_ACCESS_CONTROL="true"`. `PGVectorAdapter` resolved pool arguments as `VECTOR_POOL_ARGS` → built-in access-control default → relational `POOL_ARGS`, so the built-in default (`{"pool_size": 2, "max_overflow": 20}`, which exists to curb connection fan-out when every dataset gets its own engine) beat the operator's own sizing and `POOL_ARGS` silently did nothing in multi-user mode. Precedence is now `VECTOR_POOL_ARGS` → relational `POOL_ARGS` → the access-control default (used only when neither is set, and only while access control is on; an empty pool config otherwise), so explicit configuration outranks the built-in default. **Operator impact:** a deployment that sets `POOL_ARGS` while running with backend access control enabled will see its per-dataset PGVector pools resized to that value after upgrading — larger or smaller than the previous 2/20 depending on what is configured, and multiplied by the number of active datasets — so check the total against the server's `max_connections` and set `VECTOR_POOL_ARGS` to keep PGVector on its own sizing if you want the two to differ. No environment variable was renamed or added and no migration is required; restart or redeploy to pick up the fix (PR #4351). * Scopes role visibility to membership, so a tenant member can see the roles they belong to and who else is in them without holding tenant-wide user-management permission. `GET /api/v1/permissions/tenants/{tenant_id}/roles` no longer raises `PermissionDeniedError` for callers without that permission: when the tenant exists it now returns `200` (a nonexistent `tenant_id` still returns `404`), with owners and user-management holders (for example, an Admin role) seeing every role in the tenant and everyone else seeing only the roles they are a member of. **Client impact:** the `403` is gone from this endpoint — a caller who previously got `403` now gets a filtered list, and a caller who belongs to no role in the requested tenant (including a `tenant_id` for a tenant they are not part of) gets an empty list, so clients that branched on `403` to detect "not allowed" need to branch on the returned list instead. `GET /api/v1/permissions/tenants/{tenant_id}/roles/{role_id}/users` now returns `200` to members of the role itself; non-members without user-management permission still receive `403`. That endpoint also fixes a cross-tenant scoping hole: the role is now resolved by `(role_id, tenant_id)` rather than by id alone, so a role id from another tenant returns `404` instead of that tenant's member list. No new endpoint, request or response field, permission type, environment variable, or migration ships with this change (COG-6064, PR #4336). * Fixes two code paths that held one pooled relational connection open while checking out a second, which could deadlock a bounded Postgres/PGVector pool under concurrency. Each in-flight call pinned two connections at once, so once concurrent calls reached the pool's ceiling the waiters formed a circular wait — requests hung rather than erroring, and the stranded backends showed up as `idle in transaction` and then as pool exhaustion. Two sites are fixed: `get_or_create_dataset_database` — the function that creates per-dataset databases under `ENABLE_BACKEND_ACCESS_CONTROL="true"` — wrapped `create_authorized_dataset(...)` in an `async with db_engine.get_async_session()` block that never used `session`, holding that connection idle while the callee opened its own session plus another for the permission grant; the dead wrapper is removed, so the call now runs with no outer session. That branch fires only when a dataset is passed to `get_or_create_dataset_database` by name rather than id — a live path from `add()` and `cognify()` in v1.4.0 and earlier, while since v1.4.1 the database context resolves dataset names to ids before entry, making the removed wrapper a latent hazard rather than a reachable deadlock on current code. And `PGVectorAdapter.delete_data_points` called `await self.get_table(collection_name)` *inside* its write session, but `get_table()` opens its own `engine.begin()` connection, so the table lookup is now resolved before `get_async_session()` — exactly as the sibling `retrieve()` and `search()` methods already did. When access control is off, PGVector borrows the relational engine, so the `delete_data_points` overlap contended the pool shared with every relational query; under access control it runs on the per-dataset PGVector engines, which are smaller (`pool_size=2`, `max_overflow=20` by default) and reach the two-connections-per-call ceiling soonest. This produces the same `idle in transaction` signature as the cancellation leak fixed in #4250 and the auth-path deadlock of #4197 (fixed in #4354) but by a different mechanism; the defect class is the same one swept across the relational adapter in #4392. No public API signature, configuration option, environment variable, or database migration changed, and no pool resizing is needed — restart or redeploy to pick up the fix (PR #4389). * Fixes extraction schemas derived from a custom `graph_model` dropping domain fields that the model inherits from your own `DataPoint` subclasses. `datapoint_model_to_basemodel` — the conversion behind `graph_model_to_graph_schema` and the structured-output schema Cognee hands the LLM — selected fields by reading each class's own `__annotations__`, so only the fields annotated on the leaf class survived: given `Animal(DataPoint)` with `species: str` and `Dog(Animal)` adding `breed: str`, the schema for `Dog` contained `breed` alone and the LLM was never asked to extract `species`. Fields are now taken from the model's merged `model_fields` minus the names defined on `DataPoint` itself, so inherited domain fields are preserved — including required fields, fields with defaults or default factories, nested `DataPoint` fields (converted through the same shared cache, which also terminates cyclic `A → B → A` model graphs), and fields inherited through several levels of subclassing. `DataPoint`'s own infrastructure fields (`id`, `version`, `type`, `created_at`, `metadata`, and the rest) stay excluded, and because the exclusion is by field name a subclass that overrides `metadata` keeps it out of the schema too. **Impact:** the JSON schema returned by `graph_model_to_graph_schema` gains properties for inheritance-based custom models, and extraction for those models now populates the inherited fields, so downstream validators or snapshot tests that pinned the previous leaf-only schema need updating. Flat custom models — every field annotated on the class you pass, as in the guide examples — produce the same extraction schema as before; the one visible difference is in `graph_model_to_graph_schema` output, where a `metadata` override annotated on the class previously leaked into the schema and is now excluded like the rest of the infrastructure fields. No public API signature, configuration option, or environment variable changed, and no migration is required (SDK-161, PR #4373). * Takes subprocess-engine teardown off the response path for datasets served by subprocess-mode graph or vector databases with the dataset queue enabled. Previously, the last task to release a dataset's queue slot fetched the cached engine and awaited its `close()` inline, so an interactive search or recall waited for the worker process to shut down and drop its file lock before returning. Teardown now routes through the engine cache: the eviction still runs synchronously while the slot is held, so no caller can fetch a dying engine, but the adapter's `close()` runs on the cache's dedicated close threads. File-lock safety is preserved by the cache's pending-close latch instead of by making the caller wait — the next creation of the same engine waits until the previous worker has exited and released its lock, bounded, after which it falls back to the existing `SUBPROCESS_OPEN_LOCK_RETRIES` / `SUBPROCESS_OPEN_LOCK_BACKOFF` open retries. **Operator impact:** lower interactive search and recall latency for subprocess-mode datasets, with unchanged lock safety; a failing engine `close()` now surfaces in the logs with its traceback rather than propagating out of the dataset-context exit, so monitor logs for teardown warnings instead of relying on request errors to reveal them. No configuration option or environment variable changed and no migration is required (PR #4358). *** ## v1.4.1.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.1.dev1)** Development pre-release that bumps the package version from `1.4.1.dev0` to `1.4.1.dev1` and updates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself. No new Alembic revision ships in this cut; the entries below are the work merged to the development branch since v1.4.1.dev0. ### Highlights * Adds JSON siblings to the visualization endpoints, so an external dashboard can consume the same graph the built-in HTML page renders. `GET /api/v1/visualize/json` returns the full preprocessed payload — nodes, links, color maps, schema graph and schema data, pipeline stages, edge classes, bundles, provenance index, and memory map — plus `search_events`. `GET /api/v1/visualize/semantic` returns semantic positions and clusters for the same subgraph as a separate call, so a client that never opens the semantic view never pays for the embedding fetch and PCA behind it. `GET /api/v1/visualize/brains` returns every dataset the caller may read as a small `{dataset_id: {"name", "nodes", "links", "node_set_colors"}}` preview, with `max_nodes` (default `500`, maximum `5000`) applied independently to each dataset rather than as one larger shared cap. `GET /api/v1/visualize/live-events` returns search and improve events newer than a `since` cursor, for polling a timeline without re-fetching the whole payload; the filter is strict, so passing the previous response's `cursor` straight back never delivers an event twice. `GET /api/v1/schema/provenance/json` does the same for the memory-provenance graph. The matching Python entry points — `visualize_graph_json`, `visualize_semantic_json`, `build_brains_payload`, `get_live_events`, and `get_memory_provenance_payload` — are exported from `cognee.api.v1.visualize`. Authorization is unchanged and shared: every JSON route runs the same read-permission check as its HTML counterpart, and the HTML and JSON paths were refactored onto one `fetch_visualization_data` / `fetch_dataset_graph_data` pair so the two cannot drift on which subgraph they return. One scoping caveat on `live-events`: `dataset_id` gates who may call the endpoint, but the events themselves are collected per user rather than filtered to that dataset — the same events `/visualize/json` already embeds for that dataset. The existing `GET /api/v1/visualize` and `POST /api/v1/visualize/multi` endpoints are unchanged, and no configuration option or environment variable was added (CLO-401, PR #4331). * Fixes an explicit relational `POOL_ARGS` being silently ignored by PGVector when backend access control is enabled. `PGVectorAdapter` resolved pool arguments as `VECTOR_POOL_ARGS` → built-in access-control default → relational `POOL_ARGS`, so in multi-user mode the built-in `{"pool_size": 2, "max_overflow": 20}` — a deliberately small default, because each dataset gets its own engine and a large pool fans out as N datasets × `pool_size` — outranked an operator's explicit sizing, which therefore had no effect. Precedence is now `VECTOR_POOL_ARGS` → relational `POOL_ARGS` → the access-control default, with that default applying only when neither is set and multi-user mode is on (and an empty dict otherwise), so explicit configuration beats the built-in. If you set `POOL_ARGS` while running with access control enabled, PGVector now honors it — check that the resulting size is what you want before deploying, since per-dataset connection use can rise above the previous fixed default. No public API signature, configuration option, or environment variable changed (PR #4351). *** ## v1.4.1.dev0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.1.dev0)** Development pre-release that bumps the package version from `1.4.0.dev4` to `1.4.1.dev0` and updates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself. The release cut introduces no functional code, public API, configuration, or environment-variable changes; the entries below are the work merged to the development branch since v1.4.0.dev4. Most of them require no action, but two entries add Alembic revisions that existing deployments must run when upgrading — `c3d5e7f9a1b2` from the session-context deduplication fix and `d4e6f8a0b2c3` from the connector-credential `workspace_id` addition — see those entries for details. ### Highlights * Fixes batched triplet reads skipping triplets on the Neo4j and Ladybug graph backends, which could leave Memify's triplet-embedding pass with incomplete coverage. `get_triplets_batch(offset, limit)` is the paginated read behind `create_triplet_embeddings`, and `get_triplet_datapoints` walks the entire graph with one offset loop, advancing the offset by each batch's size until a batch comes back short or empty — which is only exhaustive if every call slices the same ordering. Both Cypher adapters placed `SKIP $offset LIMIT $limit` after `RETURN` on a `MATCH` with no `ORDER BY`, so consecutive pages were cut from an undefined result order and a multi-batch run could miss triplets (and index others twice), leaving the `Triplet_text` collection incomplete while still reporting success — most likely on large graphs, where the default `triplets_batch_size=100` means many pages. Both adapters now sort before paginating, with `ORDER BY` moved into a `WITH` clause ahead of `SKIP`/`LIMIT` so the skip applies to an already-ordered stream: `start_node.id, end_node.id, type(relationship)` on Neo4j and `start_node.id, end_node.id, relationship.relationship_name` on Ladybug. This aligns them with the SQL-backed Postgres and Turso adapters, which already ordered by `(source_id, target_id, relationship_name)` and are unchanged. No migration or user action is required beyond re-running `create_triplet_embeddings` to pick up triplets a previous run missed; the contents of any single batch are unchanged, but which rows land in which page now differs, so callers that paginate `get_triplets_batch` directly and relied on the previous (undefined) order should not assume the old grouping. No public API signature, configuration option, or environment variable changed (PR #4334). * Adds `opencode` as an agent connection type, accepted by the Python `register()` API and the `POST /api/v1/agents/register` endpoint's `type` field alongside the existing `sdk`/`api`/`mcp`/`claude_code`/`workflow`/`unknown` values, with idempotent registration, listing, and unregistration. The Cognee Cloud dashboard's **Get started** section gains an **OpenCode** connection card that walks through installing the Cognee plugin with `npx @cognee/cognee-opencode setup`; like the Claude Code and Codex cards, it auto-detects a new session — OpenCode sessions are recognized by the `opencode_` prefix the plugin puts on the session id. No existing connection type, API signature, or configuration option changed (PR #4333). * Extends the opt-in OpenTelemetry layer from spans-only to spans, metrics, and logs for the core memory operations, aligned to memory-semconv v0.1.0. `COGNEE_TRACING_ENABLED=true` (or `enable_tracing()`) now also configures a `MeterProvider` and attaches an OTel log bridge, reusing an externally configured provider when one exists; failures setting up either are swallowed, so a partial OpenTelemetry install degrades to traces-only. `add()`, `cognify()`, `search()`, and `forget()` emit spans named `memory.store`, `memory.process`, `memory.retrieve`, and `memory.delete` carrying `memory.system`, `memory.operation`, and — depending on the operation — `memory.collection`, `memory.query.text` (first 500 characters only, to cap cardinality and bound how much user input reaches your backend), `memory.query.type`, and `memory.result.count`; `add()` is instrumented for the first time. **This renames the existing `cognee.api.search`, `cognee.api.cognify`, and `cognee.api.forget` spans, so dashboards, saved queries, and alerts matching the old names must be updated** — span attributes stay backward compatible, since the `cognee.*` keys are still set alongside the new `memory.*` keys. Six metric instruments are recorded: `memory.operation.duration` (histogram, `ms`; on `forget()` only the delete-everything path records metrics), `memory.items.stored`, `memory.items.retrieved`, `memory.items.deleted`, `memory.query.result.count`, and `memory.vector.searches`; four more (`memory.data.bytes.stored`, `memory.graph.nodes.added`, `memory.graph.edges.added`, `memory.operation.errors`) are registered for custom instrumentation but not yet recorded by Cognee. Metrics and logs are only collected when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (or `console_output=True`) — unlike spans, they have no in-memory buffer — and their endpoints are derived from the traces endpoint by rewriting `/v1/traces` to `/v1/metrics` and `/v1/logs`, overridable with the standard `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. Separately, OTLP span export moves from `SimpleSpanProcessor` to `BatchSpanProcessor` (call `disable_tracing()` before exit to flush), and the HTTP-only exporter special case — previously just Langfuse's `/api/public/otel` — now also covers Dynatrace (`dynatrace.com`, `/api/v2/otlp`), port `:4318`, and endpoints with an explicit `:443` port followed by a path (`:443/`), which fixes Dynatrace export silently dropping traces under the gRPC exporter; note that an endpoint spelling out `:443` and including a path now exports over HTTP where it previously used gRPC. Everything remains fully opt-in and a no-op when disabled or when OpenTelemetry is absent, and no new environment variable was introduced (PR #4323). * Lets Cognee's own errors reach HTTP callers on `POST /api/v1/search`, `POST /api/v1/recall`, `POST /api/v1/remember` (including its COGX archive-import and `/entry` variants), and `POST /api/v1/improve` with their real status code and message. The four routers previously caught Cognee errors per type and rewrapped them into ad-hoc bodies such as `{"error": "...", "detail": "..."}` or `{"error": "...", "hint": "..."}`, sometimes replacing the actual message with unrelated advice — a recall permission failure, for example, returned `403` with a "Recall prerequisites not met" hint telling the caller to ingest and cognify first. The routers now re-raise every `CogneeApiError` subclass so the global handler in `cognee/api/client.py` answers with the error's own status code and the body `{"detail": "<message> [<ErrorName>]"}`. **Status codes are unchanged** (`402`/`403`/`404`/`409`/`422` all come from the exceptions themselves); only the response body shape changed, so clients parsing the legacy `error`/`hint` fields on these endpoints should switch to reading the HTTP status code and `detail` — see [Error Handling](/api-reference/introduction#error-handling). Two adjacent gaps were fixed in the same pass: recall's unreachable `except PermissionDeniedError: return []` arm was removed as dead code (it was always shadowed by an earlier catch, so the `200`-with-empty-list behavior it suggested never shipped), and a permission denial during a remember COGX archive import no longer collapses into a generic `409` "error occurred during COGX archive import". The generic fallbacks for unexpected, non-Cognee errors are unchanged (`500` with `{"error": "Internal server error", "detail": "..."}` for search; `409` with `{"error": "..."}` for recall, remember, and improve), and `POST /api/v1/cognify` and the LLM endpoints still return the legacy `{"error": "Token budget exhausted", "detail": "..."}` body on `402`. No public API signature, configuration option, or environment variable changed (SDK-255, PR #4285). * Shrinks the `cognee-mcp` server's advertised tool surface so agents no longer load the whole catalog into context. By default, `tools/list` now advertises 8 of the 10 registered tools — the memory API (`remember`, `recall`, `forget`), the three workspace UI entry points (`visualize_graph_ui`, `upload_file_ui`, `open_cognee_workspace`), plus FastMCP's synthetic `search_tools` and `call_tool` — while the structured-JSON workspace helpers are found by calling `search_tools(query=...)` (BM25 ranking, up to 10 results) and invoked by name or through the `call_tool` proxy. Hiding a tool never makes it unreachable: unadvertised tools stay directly callable, which is what keeps the workspace UI (which calls its internals by name) working. The new `COGNEE_MCP_TOOL_MODE` environment variable — also settable per-process with `--tool-mode` — selects `default` (the surface above), `minimal` (only the three memory tools plus the search pair), or `all` (the previous flat catalog with no search transform); unknown values log a warning and fall back to `default`. One sharp edge: `search_tools` matching is purely lexical with no stemming, and zero-scoring tools are dropped, so a terse query like `dataset` misses `list_datasets_json` (token `datasets`) — multi-word natural-language queries are the reliable form. In the same rework, the `cognify_file` tool was removed and folded into `remember`, which now accepts `filename` + `content_base64` (up to 10 MB) to ingest an uploaded file alongside the existing `data` text form; session-cache writes (with `session_id`) remain text-only (PRs #4283 and #4258). * Makes `cognee-cli config` a working, persistent interface. `cognee-cli config get <key>` and bare `config get` previously printed "not implemented" — they were gated on `hasattr` checks for methods that never existed — and now return real values through the new `cognee.config.get(key)` and `cognee.config.get_all()` APIs, masking secret values (`llm_api_key`, `embedding_api_key`, `vector_db_key`) by default; pass `--show-secrets` on the CLI or `reveal_secrets=True` in Python to print them in plaintext. `config set` and `config unset` previously mutated only the in-process `@lru_cache`d config singletons, so a value vanished the moment the CLI process exited; the CLI now calls `cognee.config.set(key, value, persist=True)`, which additionally writes the value under its resolved environment-variable name to the `.env` file in the current working directory (creating the file if needed) — the same file every config class already reads — so values survive across CLI invocations and are picked up by the next process started from that directory. `unset` resets a key to its default through the same persisted path. The `persist` parameter is new and defaults to `False`, so SDK callers of `config.set()` keep the previous in-memory-only behavior; unknown keys now raise `InvalidConfigAttributeError` on both paths (COG-5970, PR #4287). * Gives graph nodes a notion of fact validity over time. Every `DataPoint` gains a `valid_to` field (milliseconds-epoch integer, default `None` = still current) — distinct from the temporal stack's Event/Interval `time_to`, which records when an event happened, not whether the fact still holds. The new `close_node(node_id, at_ms=None)` helper (`cognee/tasks/storage/close_node.py`) stamps `valid_to` on an existing node when a fact is superseded — mark "Alice works at X" closed and write the replacement, instead of deleting the old node — and `is_valid(node, at_ms=None)` reports whether a node is still current (`valid_to` is `None` or in the future), accepting both `DataPoint` instances and plain graph-node dicts. Persistence goes through a new optional `update_node(node_id, values)` extension on `GraphDBInterface`, currently implemented only on the default Ladybug (Kuzu) adapter with a read-modify-write that leaves fields the caller did not name untouched; on backends without it, `close_node` logs a warning and returns `False` rather than failing silently (`True` means the node existed and was updated). Closing is last-write-wins — re-closing overwrites the earlier stamp, so guard with `is_valid` first where idempotence matters. No existing API signature, configuration option, or environment variable changed (SDK-200, PR #4105). * Turns the client-side LLM RPM limiter on automatically when the provider shows evidence it cannot keep up. Previously the limiter was purely opt-in: unless you set `LLM_RATE_LIMIT_ENABLED=true`, Cognee dispatched unbounded and a busy `cognify` run could keep hammering a provider that was already rejecting requests. A new overload policy (`cognee/infrastructure/llm/overload_policy.py`) watches the dispatch seam in `cognee/shared/rate_limiting.py`; when a failed dispatch carries overload evidence anywhere in its exception cause chain — a rate-limit error, a timeout (how overwhelmed local servers surface, since they never send rate limits), or HTTP `429`, `503`, or `529` — it logs one warning naming the cause and the budget, and paces every subsequent dispatch with `LLM_RATE_LIMIT_REQUESTS` / `LLM_RATE_LIMIT_INTERVAL` for a 900-second cooldown. Further evidence inside that window extends it silently; once the window lapses quietly, behavior returns to the configured state, and a fresh episode warns again. Because adapters enter the limiter inside their retry loop, retried attempts are paced too. The new `AUTO_RATE_LIMIT` environment variable (default `true`) controls this; set it to `false` to keep the old fully unbounded behavior, or keep using `LLM_RATE_LIMIT_ENABLED=true` to pace from the first request regardless. Response latency alone is not treated as overload evidence. Relatedly, when `LLM_RATE_LIMIT_REQUESTS` is not set explicitly, serial local inference servers (Ollama, llama.cpp by provider; LM Studio by the `lm_studio/` model prefix) now default to `10` requests per interval instead of `60`, since the cloud default would still flood them; vLLM is deliberately classed as a regular provider and keeps `60`. An explicitly configured `LLM_RATE_LIMIT_REQUESTS` always wins. User impact: runs against a strained provider now slow down instead of failing, at the cost of taking longer; requests already dispatched when the limiter engages cannot be un-queued (PR #4240). * Raises the default `chunks_per_batch` for the standard Cognify pipeline from `100` to `2000`. `get_default_tasks` in `cognee/api/v1/cognify/cognify.py` uses this value as the `batch_size` for the graph extraction/summarization task and for `add_data_points`, so larger batches mean fewer, bigger chunk-level task batches per run. An explicit `chunks_per_batch` argument and the `chunks_per_batch` value from `CognifyConfig` both still take precedence, and the temporal Cognify pipeline default is unchanged at `10` (PR #4240). * Scopes the default session to the dataset, so omitting `session_id` no longer funnels every dataset's turns into one shared conversation. `SessionManager.resolve_session_id` (now public, pure, and synchronous — it performs no database lookups) returns an explicit `session_id` unchanged, but when none is given and the manager knows its dataset — from the `dataset_id` constructor argument or the `current_dataset_id` context variable that dataset-scoped `recall()`/`search()` calls enter — it derives `default_session_<dataset_id>` instead of the single global `"default_session"`. With no dataset known the plain global default is still used, exactly as before. Both the read and write sides route through the same function, so an omitted `session_id` resolves identically in either direction within a dataset context. **Behavior change:** history that previously accumulated under the shared `default_session` is now written per dataset; existing `default_session` entries are not migrated and stay readable by passing the literal `session_id="default_session"`. `cognee.session.get_session`'s `session_id` default changed from `"default_session"` to `None` to defer resolution to the manager; a bare call outside any dataset context rebuilds its manager scoped to the caller's existing `main_dataset` (a read-only lookup — no dataset is created) so dataset-scoped writes are readable back without stating a dataset, and raises a `SessionPreconditionError` (a `CogneeValidationError`) when no `main_dataset` exists rather than silently returning the unscoped global session. Relatedly, `set_database_global_context_variables` now accepts exactly `Optional[UUID]` and rejects dataset names and UUID *strings* with a `CogneeValidationError` (`SessionManager` likewise rejects a non-UUID `dataset_id` with `SessionParameterValidationError` instead of degrading to an unscoped session); every production call site already passed a UUID object, but automation or tests that entered the context by dataset name must now resolve the dataset and pass its `.id`. In the CLI, the per-user `scoped_session_id` scheme (`"<user_id>:default"`) is removed from `search`, `recall`, `feedback`, and `sessions` — the dataset-scoped default replaces it, so `cognee search` now passes `session_id=None` and the explicit `--session-id` values you pass to the other commands are used verbatim rather than being prefixed with your user id. No configuration option or environment variable changed (SDK-255, PR #4286). * Fixes duplicate session-context rows in the SQL session-cache backends (`CACHE_BACKEND=sqlite`, the default, or `postgres`) permanently breaking a session with a 503. `create_session_context_entry` appended a row on every call, so racing update-then-create writers (for example the session persist watermark) accumulated several rows for the same `(user_id, session_id, entry_id)`; `update_session_context_entry` then resolved that key with `scalar_one_or_none()` and raised `MultipleResultsFound`, which surfaced as a per-session 503 that recurred on every subsequent turn of that session. Creates now issue a dialect-specific `ON CONFLICT (user_id, session_id, entry_id) DO UPDATE` upsert (last writer wins, refreshing `payload` and `expires_at`), so those flows converge on a single row, and the update path resolves to the newest row (`ORDER BY seq DESC LIMIT 1`) instead of raising while its `UPDATE` still rewrites every straggler duplicate with the merged payload. The `cache_session_context` table gains a `uq_cache_session_context_entry` unique index on that key — the index the upsert targets. Cache tables are created on init rather than managed by Alembic, so **fresh** databases get the index from the table definition and need no action, but an **existing** deployment must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) before the upsert has an index to conflict on: new Alembic revision `c3d5e7f9a1b2` deletes all but the newest (`MAX(seq)`) row per key and creates the index, covering both the Alembic-connected database and the standalone SQLite `cache.db` that the default backend keeps next to the relational database. Because the migration mutates data and builds a unique index, it can take time and hold locks on a large `cache_session_context` table — back up the database and run it in a maintenance window. The migration has deliberately no runtime fallback: if `CACHE_DB_URL` points at a separate non-SQLite database it cannot reach, it raises `RuntimeError` naming that URL and printing the `DELETE` / `CREATE UNIQUE INDEX` statements to apply there manually before re-running, rather than letting a deployment proceed against a cache database the upsert would break on. No public API signature, configuration option, or environment variable changed (fixes #4226, PR #4232). * Fixes cancelled requests leaking Postgres connections stuck in `idle in transaction`, which could exhaust the connection pool until every query failed. `SQLAlchemyAdapter.get_async_session()` in `cognee/infrastructure/databases/relational/sqlalchemy/SqlAlchemyAdapter.py` ended in `finally: await session.close()`, which runs before the `async_sessionmaker` context manager's own exit; when the surrounding task was being cancelled — a client disconnect, a timeout, or any `asyncio` cancellation — that `await` was interrupted before the `ROLLBACK` reached Postgres, so the connection went back to the pool still inside an open transaction. Enough of those piled up and the pool hit its ceiling, at which point requests began failing across the board (authentication included, since the API-key lookup is itself a DB query). The explicit `close()` is removed and replaced with an `except asyncio.CancelledError` branch that calls `await asyncio.shield(self._discard_cancelled_session(session))` and then re-raises; the new `_discard_cancelled_session()` static helper calls `await session.invalidate()`, which drops the DBAPI connection from the pool without needing the `ROLLBACK` round-trip that cancellation keeps interrupting, and logs (rather than raises) if the invalidation itself fails. The `asyncio.shield` keeps the cleanup from being cut short by the same cancellation. Successful requests and ordinary exceptions are unaffected — the sessionmaker's own context manager already closes and rolls back on those paths. No public API signature, configuration option, environment variable, or database migration changed; no pool-sizing changes are needed beyond deploying the fix (fixes #4197, PR #4250). * Fixes Zep/Graphiti imports silently dropping scope from every entity node and every fact when the export spells the scope key `session_id` rather than `group_id`. In `ZepSource` (and its `GraphitiSource` alias), the episode branch already resolved the record's scope as `group_id or session_id`, but the entity and fact branches built `COGXScope(session_id=...)` from `group_id` alone — so a single export using the `session_id` spelling imported with scope intact on its episodes and empty on all of its nodes and edges, with nothing raised. Both branches now apply the same fallback (`node.get("group_id") or node.get("session_id")`, `edge.get("group_id") or edge.get("session_id")`), and `group_id` keeps precedence where a record carries both, matching the episode branch. User impact: exports already keyed on `group_id` are unaffected and import exactly as before; if you imported a `session_id`-keyed Zep or Graphiti export — most consequentially into a multi-tenant graph, where entities and facts lost their tenant attribution — re-run that import to restore scope on the affected nodes and edges. No public API signature, configuration option, environment variable, or migration changed (PR #4293). * Fixes Mem0 and LangMem imports landing zero memories when the payload carries an accepted wrapper alias that is present but empty ahead of a populated one. `Mem0Source._load_raw` and `LangMemSource._load_raw` unwrapped on the *first* accepted alias that happened to be a list — `results`, `memories`, `items` for Mem0, and `memories`, `results`, `items`, `data` for LangMem — so a payload like `{"results": [], "memories": [...]}` unwrapped to the empty `results` list and never read the populated alias, and the import still reported success with nothing imported. Both sources now scan their alias order and return the records under the first alias that actually carries any, filtering to dict items before deciding an alias is populated — the same rule `ZepSource`'s `_first_list` helper already applied. Error behavior is preserved: a dict where none of the accepted aliases maps to a list, and a payload that is not a list, both still raise `ValueError`, while a recognized wrapper whose aliases are all empty still yields zero records without raising. User impact: payloads whose first accepted alias was already the populated one are unaffected and import exactly as before; if a Mem0 or LangMem import reported success but landed zero memories, re-run that import. No public API signature, configuration option, environment variable, or migration changed (PR #4314). * Fixes `improve()` silently retargeting the caller's default dataset when the `dataset` argument could not be resolved. Previously an internal `_resolve_dataset_name` helper returned `"main_dataset"` for any unresolved reference, so a mistyped, nonexistent, or unauthorized dataset UUID quietly enriched the caller's own default dataset instead of failing, and individual stages re-resolved the dataset independently. The target dataset is now resolved and authorized once, up front, through the same `resolve_authorized_user_datasets` layer that `remember()` and `memify()` use: a UUID the caller does not hold `write` permission on raises `PermissionDeniedError` — read permission is not enough, since every improvement stage writes — and a UUID that does not exist raises the identical error, so dataset existence is never confirmed to an unauthorized caller. Dataset **names** keep their owner-scoped semantics: a name is resolved — or created — in the caller's own scope, so improving a name another user happens to own creates the caller's own dataset with that name rather than touching theirs; a dataset owned by someone else must be targeted by UUID. The session-bridging stages now receive the resolved dataset UUID instead of re-resolving a name. User impact: if your integration passed `improve()` a bad or unauthorized dataset reference and relied on the silent `main_dataset` fallback, the call now raises `PermissionDeniedError` — catch it and pass a UUID you hold `write` permission on. The `improve()` signature, configuration options, environment variables, and migrations are unchanged (SDK-255, PR #4294). * Adds an optional `workspace_id` owner dimension to the third-party connector credential store, so several users can share one connection instead of each connection belonging to exactly one user. `upsert_credential` in `cognee/modules/integrations/credentials.py` gains a keyword-only `workspace_id: Optional[UUID] = None`; because the `integration_credentials` table is keyed on the external account (`UNIQUE(provider, provider_account_id)`), a reconnect by a *different* owner while the current connection is still active raises `CrossUserConflictError` rather than silently taking the connection over, and `workspace_id` now decides who that owner is. When it is passed, the conflict comparison runs on `workspace_id` — two different users in the same workspace can reconnect the same external account, while a different workspace is still refused — and `user_id` continues to record which user actually connected it. Omitted (the default), the ownership and conflict rules are exactly the previous single-user contract, compared on `user_id`. A companion `get_active_credential_for_workspace(workspace_id, provider)` mirrors the existing `get_active_credential_for_user(user_id, provider)` for the new dimension, filtering on `status == "active"` and ordering newest-first; use it for connections upserted with a `workspace_id`, since `user_id` on those rows is only who connected them, not the owner. The `IntegrationCredential` model gains a matching `workspace_id` column — `nullable`, indexed, and deliberately **not** a foreign key (a plain opaque id, same as `user_id`, so there are no cascade semantics) — backed by new Alembic revision `d4e6f8a0b2c3` (on top of `c3d5e7f9a1b2`), which adds the column and the `ix_integration_credentials_workspace_id` index. An **existing** deployment must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) to pick up the column; the revision is additive and both its `upgrade()` and `downgrade()` inspect the live schema first, so re-running either is a no-op. Existing rows keep `workspace_id` as `NULL` and behave as before. This module is internal plumbing rather than part of the public SDK: no public API signature, configuration option, or environment variable changed (CLO-406, PR #4318). *** ## v1.4.0.dev4 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev4)** Development pre-release that bumps the package version from `1.4.0.dev3` to `1.4.0.dev4`; the accompanying `uv.lock` change records the new version only. No new Alembic revision ships in this cut. Relative to v1.4.0.dev3 this build also restores the v1.4.0.dev2 changes (including the OAuth integrations framework and its `b2c4d6e8f0a1` migration) that the dev3 release branch did not carry. ### Highlights * Adds an `embedding_max_concurrent_data_points` setting (default `150`, env var `EMBEDDING_MAX_CONCURRENT_DATA_POINTS`, also settable via `cognee.config.set_embedding_config({"embedding_max_concurrent_data_points": ...})`) controlling how many data points may be in flight to the embedding engine during indexing. `index_data_points` in `cognee/tasks/storage/index_data_points.py` previously capped concurrency with a hardcoded `asyncio.Semaphore(4)` — four concurrent batches regardless of batch size; the semaphore is now sized `max(1, embedding_max_concurrent_data_points // batch_size)`. The default changes effective throughput: with the LiteLLM engine's default `batch_size=100`, indexing now runs one batch at a time (about 100 points in flight) instead of four (about 400) — raise `EMBEDDING_MAX_CONCURRENT_DATA_POINTS` to restore or exceed the old parallelism, or lower it to stay under provider rate limits. `embedding_batch_size` and the embedding rate-limit settings are unchanged, and no public API signature changed (SDK-285, PR #4144). * Fixes the `dry_run=True` cost estimator applying the `ACCEPT_LOCAL_FILE_PATH` gate inconsistently across platforms. In `cognee/modules/cognify/estimator.py`, `_path_candidate` decided whether an existing file path was *absolute* — and therefore rejected when the flag is disabled — with a bare `value.startswith("/")` check. That check does not describe absolute paths on Windows, so a genuine Windows path like `C:\data\notes.txt` was not recognized as one: with `ACCEPT_LOCAL_FILE_PATH=False` it slipped past the gate and the path string itself was priced as raw text instead of raising, while conversely a `/`-prefixed string — which Windows resolves to a *drive-relative*, unusable path — was rejected even though a real run ingests it as text. Detection now matches `save_data_item_to_storage` exactly: the string must look absolute for the current platform (`/`-prefixed, or drive-lettered when `os.name == "nt"`) **and** satisfy `Path(os.path.normpath(value)).is_absolute()`. On Windows, `C:\...` paths that exist now raise `Local files are not accepted` when the flag is disabled, and drive-relative `/`-prefixed strings fall through to raw text; POSIX behavior is unchanged. Existing relative paths and strings that are not existing files remain raw text on every platform, and enabling the flag (the default) is unaffected. The rest of the PR is unit-test repair with no runtime effect — a missing `SAMPLE_ARGUMENTS` entry for `ProviderNotDeducibleError` in the error-contract test, and an autouse fixture that snapshots and restores patched module attributes so mocks stop leaking between test modules. No public API signature, configuration option, environment variable, or migration changed (PR #4257). * Fixes Letta imports silently dropping every message — and the whole conversation episode — when a message serializes `content` as `null` and carries its text under the `text` alias. In `LettaSource`, `_message_text` resolved the alias with `message.get("content", message.get("text"))`, but `dict.get`'s default only fires on a *missing* key: a Letta serializer that writes unset fields as `null` instead of omitting them produced `content: None`, so the `text` fallback never ran and the message resolved to `""`. The caller skips any message with no text, and because every message in one agent file shares the same serialization, the agent's turn list ended up empty and the `if turns:` gate emitted no `COGXEpisode` at all — the import still reported success. The fallback is now explicit (`content = message.get("content")`, then `if content is None: content = message.get("text")`), so those messages and their episode are imported. Files that were already importing correctly are unaffected, and an explicitly empty `content` (`""`) is still treated as a message with no text rather than falling through to `text`. User impact: if you imported Letta agent files and found conversation history missing, re-run those imports to recover the episodes. No public API signature, configuration option, environment variable, or migration changed (PR #4261). * Fixes `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` failing at query time on installations that do not have the optional `postgres` extra. Both retrievers began by checking whether the active graph engine is one of the Postgres graph backends, which they reject with `SearchTypeNotSupported` because those backends cannot run Cypher. That check imported the Postgres adapters (`PostgresAdapter`, `PostgresHybridAdapter`), and those adapter modules import `asyncpg` at module scope — a dependency that ships only with `cognee[postgres]` — so on an installation without the extra the check itself raised `ImportError`/`ModuleNotFoundError` rather than the backend it was trying to detect. In `NaturalLanguageRetriever.get_retrieved_objects` the import was unguarded, so the error propagated to the caller; in `CypherSearchRetriever.get_retrieved_objects` it sat inside the block whose handler wraps unexpected exceptions, so it surfaced as a misleading `CypherSearchError`. The check no longer imports backend adapters at all: Cypher capability is now declared on the adapter class itself — `GraphDBInterface` defines `supports_cypher_queries = True`, the Postgres, Postgres hybrid, and Turso adapters override it to `False` — and both retrievers read the flag off the active engine, raising `SearchTypeNotSupported` (naming the rejected adapter class) when it is false. User impact: on a Cypher-capable backend such as Kuzu or Neo4j, both search types now work without installing `cognee[postgres]`; installations that do have the extra are unaffected, and Postgres graph backends still raise `SearchTypeNotSupported` as before. The Turso graph backend — whose `query()` also runs SQL rather than Cypher — is newly covered by the same check, so it now raises `SearchTypeNotSupported` up front instead of failing when the generated Cypher reaches it. No public API signature, configuration option, environment variable, or migration changed (fixes #4123, PRs #4124 and #4274). *** ## v1.4.0.dev3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev3)** Development pre-release that bumps the package version from `1.4.0.dev2` to `1.4.0.dev3` and re-resolves `uv.lock`. Unlike the other pre-releases in this line it was cut from a release branch rather than the tip of the development branch, so it carries the deployment and Ollama-validation work below while omitting some changes already shipped in v1.4.0.dev2 — most notably the OAuth integrations framework and its `b2c4d6e8f0a1` migration are absent from this build; both are included again in v1.4.0.dev4. ### Highlights * Adds an advisory Ollama model-support warning and ships S3 support in the default Docker image. When `LLM_PROVIDER="ollama"`, a new `cognee/infrastructure/llm/ollama_support.py` helper classifies the configured `LLM_MODEL` against a built-in support matrix as the LLM configuration is validated and logs a warning for models Cognee has not validated for structured graph extraction: `llama3` through `llama3.3` and `qwen2.5` tags of 14B or larger are recommended (nothing logged); `mistral`, `phi3`, `phi3.5`, and `qwen2.5` below 14B or with no parseable size warn about known limitations (schema validation errors, silent drops); every other model warns that it is unvalidated and extraction quality may vary. The check is advisory only — nothing is blocked, no exception is raised, and each distinct `LLM_MODEL` value warns at most once per process. Matching ignores an `ollama/` prefix and everything after the `:` in a tag (except `qwen2.5`, whose tag is parsed for a parameter count), and the message points at the new `docs/ollama_models.md` in the repo; see [Model Support Warning](/setup-configuration/llm-providers#model-support-warning) on the LLM Providers page for the full matrix. Separately, the root `Dockerfile` now includes the `aws` extra (`s3fs[boto3]`) in both `uv sync` steps, so the source-built API image supports [S3 file storage](/guides/s3-storage) out of the box without passing `COGNEE_EXTRAS="aws"`. The access-control handler compatibility check also now accepts the `ladybug`/`kuzu` provider aliases interchangeably — both handler names register the same embedded Ladybug (formerly Kuzu) handler and both provider names satisfy its check — so mixing the current and legacy names no longer raises `EnvironmentError` at startup under backend access control. The branch also adds runnable examples: `examples/demos/local_ollama_example.py` (fully local add → cognify → search with Ollama; since moved to `examples/guides/local_ollama_example.py`) and `examples/tutorials/migrate_from_mem0_tutorial.py` with a bundled sample export (since moved to `examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py`). No public API signature, configuration option, or environment variable changed (PR #4272). *** ## v1.4.0.dev2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev2)** Development pre-release that bumps the package version from `1.4.0.dev1` to `1.4.0.dev2`; the `uv.lock` change records the new version only. One entry requires action: the new OAuth integrations framework ships Alembic revision `b2c4d6e8f0a1`, which existing deployments must run when upgrading (`cognee.run_migrations()`, or `alembic upgrade head`) — see that entry for details. ### Highlights * Adds an opt-in `memify` pipeline, `consolidate_entities`, that merges near-duplicate `Entity` nodes left behind by repeated `cognify` runs or multi-source ingestion (e.g. "New York City" vs "NYC", whose name-derived ids never collapse on their own). `consolidate_entities_pipeline` (in `cognee/memify_pipelines/consolidate_entities.py`) runs two new tasks exported from `cognee.tasks.memify`: `detect_entity_duplicates` embeds each entity name and clusters candidates by cosine similarity (`similarity_threshold=0.85` default, `top_k=10` neighbors) plus normalized-name equality (`name_match=True`), and `merge_entity_duplicates` picks the canonical node (most connected, ties broken by oldest then name), re-points every edge onto it with direction preserved, unions descriptions, records a `merged_from` report, then deletes the duplicate nodes and their `Entity_name` vector embeddings. Merging stays within one `EntityType` unless `allow_cross_type=True`; `protect_node_types` excludes types entirely, and `dry_run=True` logs the plan without mutating anything. The pipeline only runs when invoked — existing pipelines, public APIs, configuration options, and environment variables are unchanged (PR #3534). * Fixes `cache_root_directory` in `BaseConfig` (`cognee/base_config.py`) silently accepting relative paths. The sibling root directories — `data_root_directory`, `system_root_directory`, and `logs_root_directory` — were already normalized through `ensure_absolute_path()`, which resolves absolute paths, passes `s3://` URLs through untouched, and raises `ValueError` on a relative path; `cache_root_directory` alone skipped that check, so a relative cache path would resolve differently depending on the current working directory, a hard-to-debug misconfiguration. The one-line change adds the same `ensure_absolute_path()` call for `cache_root_directory`, placed after the S3 auto-config logic so S3 cache paths keep working. Action needed only if you currently set a relative `cache_root_directory`: it now raises `ValueError` at configuration time and must be an absolute path (or an `s3://` URL). No new configuration option, environment variable, public API signature, or migration is involved (PR #3581). * Adds a `COGNEE_PROVENANCE_MODE` environment variable (also readable from `.env`) controlling provenance stamping on DataPoints during pipeline runs, backed by the new `cognee/modules/pipelines/provenance_config.py` (`ProvenanceConfig`, cached via `get_provenance_config()`). Valid values are `lightweight` (default), `deep`, and `disabled`; values are lowercased, and an unknown value logs a warning and falls back to `lightweight`. In the shipped gating, `disabled` skips the `_stamp_provenance` call in `run_tasks_base.py` entirely as a zero-overhead escape hatch, while `lightweight` and `deep` both run the existing stamping (the config exposes `is_lightweight()`/`is_deep()` helpers, but no code path yet branches on them). PR #3775 as merged left `run_tasks_base.py` half-edited — a broken two-argument `_stamp_provenance(result_data, pipe_name)` call raised `TypeError: missing 1 required positional argument: 'task_name'` on every pipeline run alongside the surviving ungated call — and PR #4215 (SDK-317) repaired this before release with a single full-argument call gated on `get_provenance_config().is_disabled()`. Default behavior is unchanged; no action needed unless you want `disabled` (PRs #3775 and #4215). * Fixes `LangchainChunker` (`cognee/modules/chunking/LangchainChunker.py`), which could not be instantiated at all after the chunker API unification even though it is user-facing — documented in the `cognify()` docstring and offered among the CLI chunker choices — so selecting it crashed mid-pipeline. Three stacked breaks are repaired, re-landing the alignment from PR #2966 that was reverted in #3167: the constructor parameter is renamed from `max_chunk_tokens` to `max_chunk_size` (every `Document.read()` call passes `max_chunk_size=`, which previously raised `TypeError`), `super().__init__` now passes the three arguments the `Chunker` base actually accepts, and `read()` checks `self.max_chunk_size` and emits `DocumentChunk` with `chunk_size=token_count` plus `importance_weight` propagation — matching `TextChunker` — instead of the undefined `word_count`/`token_count` fields while omitting the required `chunk_size`. New regression tests are guarded with `pytest.importorskip("langchain_text_splitters")` so environments without the `langchain` extra skip cleanly. `TextChunker` behavior, configuration options, and environment variables are unchanged (fixes #3888, PR #3893). * Changes LLM configuration to reduce required setup: `LLM_PROVIDER` is now optional and inferred from the `llm_model` prefix (e.g. `anthropic/claude-...` implies `anthropic`) via a new validator in `cognee/infrastructure/llm/config.py`; an explicit provider — kwarg or env var — always wins, and a prefix outside `KNOWN_LLM_PROVIDERS` raises `ProviderNotDeducibleError` with a message naming the supported providers and pointing litellm-routed prefixes (e.g. `openrouter/`, `groq/`, `deepseek/`) at `LLM_PROVIDER="custom"`. Behavioral change requiring action: such prefixes previously defaulted to `openai` passthrough and now fail fast until you set `LLM_PROVIDER="custom"`. Additionally, a central `instructor_modes.py` table replaces the `default_instructor_mode` values duplicated across nine adapters; the `embedding_rate_limit_*` fields move from `LLMConfig` to `EmbeddingConfig` (field and env-var names unchanged); quote-stripping now covers all declared string fields instead of a 14-field allow-list; and the Ollama validator no longer checks embedding env vars, leaving that to `EmbeddingConfig` (SDK-142, part of #3382, PR #3994). * Adds the catalog contract underpinning the planned Integrations Hub and Use-Case Gallery: a new top-level `catalog/` directory in the cognee repo with a draft-07 JSON Schema (`catalog/schema.json`), a validating loader (`catalog/loader.py`, runnable as `python -m catalog.loader`) that applies three passes — schema validation, naming rules, local path resolution — with aggregated errors, ten seed YAML entries under `catalog/entries/` spanning all three kinds (integrations such as `claude-code` and `langgraph`, packages such as `qdrant` and `weaviate`, and use-cases such as `agent-memory` and `temporal-reasoning`), an advisory cross-repo drift check (`catalog/inventory_sync.py`) against the `cognee-integrations` repository's `inventory.yml`, a `Catalog` CI workflow (`.github/workflows/catalog.yml`), and a contributor guide (`docs/contributing/add-catalog-entry.md`). This is chunk 1 of issue #3603; Hub/Gallery rendering and full cross-repo aggregation are deferred to follow-ups. The catalog tooling is not shipped in the wheel, so the installed `cognee` package, public APIs, configuration options, and environment variables are unchanged (SDK-151, PR #4023). * Adds opt-in temporal contradiction resolution for functional (single-valued) relationships to the cognify pipeline. When a relationship like `ceo_of` should hold only one target per subject but the graph accumulates several — e.g. one document names Alice as CEO and a later one names Bob — cognee previously kept both as current facts. Passing the new keyword `cognify(functional_relationships={"ceo_of"})` appends a `resolve_temporal_contradictions` task (in `cognee/tasks/graph/resolve_temporal_contradictions.py`) that runs last, after the graph is written: for each declared relationship it groups stored edges by subject, keeps the most recent assertion (by `updated_at`) as current, and tags the older ones with `superseded=True`, `superseded_by` (the winner's `edge_object_id`), and `supersession_reason` — nothing is deleted and provenance is preserved. The core, `tag_superseded_edges` in `cognee/modules/graph/utils/temporal_conflict_resolver.py`, is deterministic with no LLM call, resolution runs against the stored graph so a fact ingested today supersedes one from last month, and re-runs are idempotent. The parameter defaults to `None`, so the default pipeline, `add_data_points`, and all existing behavior are unchanged; no environment variable or migration is involved (SDK-184, PR #4084). * Adds a Graph Insight Report that describes a built knowledge graph instead of only drawing it: a Markdown report covering hub nodes ("god nodes", ranked by degree plus `networkx.pagerank` over the entity/entity-type layer, with a degree-only fallback), surprising cross-set connections (entity pairs whose endpoints belong to different `node_set`s, resolved through `belongs_to_set` edges), edge provenance (EXTRACTED entity-to-entity relationships vs DERIVED chunk/document scaffolding), and LLM-suggested follow-up questions ready to pipe into `search()` — the one section that costs an LLM call. Exposed three ways: a new public `cognee.report(datasets="main_dataset", output_path="graph_report.md", top_n=10)` API in `cognee/api/v1/report/report.py`, a new `SearchType.GRAPH_REPORT` backed by `GraphReportRetriever` in `cognee/modules/retrieval/graph_report_retriever.py`, and a `cognee-cli report` command. The feature is additive-only: it reads solely via `get_graph_data()`, so there are no schema, storage, or migration changes, and no existing API signature, configuration option, or environment variable changed (SDK-188, PR #4101). * Adds opt-in contradiction detection between newly ingested facts and facts already stored in the graph. When enabled, cognify appends a `detect_contradictions` task (in `cognee/tasks/graph/detect_contradictions.py`) that runs last: it collects the entities the current ingestion touched, fetches their 1-hop stored neighbourhood via `get_neighborhood`, asks the LLM which fact pairs conflict, and records each contradiction as a queryable `contradicts` edge carrying both facts, the reason, and a confidence score, alongside a logged warning — nothing is deleted or overwritten, and the task swallows its own errors so detection can never break ingestion. It is enabled through configuration rather than a new function argument: `CONTRADICTION_DETECTION=true` on `CognifyConfig` (`cognee/modules/cognify/config.py`), tuned by `CONTRADICTION_CONFIDENCE_THRESHOLD` (default 0.5) and `CONTRADICTION_MAX_FACTS` (default 500); it also applies to `remember()`, which builds its graph through `cognify()`. Default off — with the flag unset the pipeline task list and behavior are identical to before, and no public API signature changed (SDK-199, PR #4104). * Fixes three `CogneeError` subclasses that overrode `__init__` without calling `super().__init__()`, leaving `Exception.args` empty — which broke `repr()`, exception chaining (`raise ... from cause`), and the centralized logging in `CogneeApiError.__init__`. The offenders were `EntityNotFoundError` and `NodesetFilterNotSupportedError` in `cognee/infrastructure/databases/exceptions/exceptions.py` and `WrongTaskTypeError` in `cognee/modules/pipelines/exceptions/tasks.py`; an AST sweep confirmed these were the only three in the repo. All now call `super().__init__(message, name, status_code)`; the two database exceptions pass `log=False` because they are raised in routine control flow (user resolution, migrations, pruning, graph projection), so `args` and chaining are restored without flooding logs with ERROR lines for expected conditions, while `WrongTaskTypeError` — a genuine programming error — keeps default logging. A new regression guard, `cognee/tests/unit/exceptions/test_cognee_error_contract.py`, enforces the contract via both a static AST sweep and a runtime check of every importable subclass. Constructor signatures, exception types, and raising sites are unchanged, so no caller needs action (SDK-240, fixes #3749, PR #4151). * Fixes ingestion turning a `/`-prefixed string that is not an actual file — e.g. a text note like `/remember to call Bob about the meeting` — into a broken `file://` URI pointing at a non-existent path, which the loader later failed on. The absolute-path branch of `save_data_item_to_storage` (in `cognee/tasks/ingestion/save_data_item_to_storage.py`) converted unconditionally; it now also requires `abs_path.is_file()`, mirroring the relative-path branch, so an absolute-looking string becomes a `file://` URI only when it points at an existing file and otherwise falls through to text ingestion on every platform. Existing absolute file paths still convert as before, and `ACCEPT_LOCAL_FILE_PATH=false` still rejects existing local files with `IngestionError`. The dry-run estimator's `_path_candidate` (`cognee/modules/cognify/estimator.py`), which mirrors this routing, is updated to price a missing absolute path as text instead of raising "file does not exist". Side effect of keying on "existing file": directory paths, broken symlinks, and pathological paths also route to text rather than a broken URI. Follow-up to #3892; no public API signature, configuration default, or migration changed (SDK-234, fixes #3887, PR #4155). * Adds a reusable OAuth integrations framework to the cognee server, with Slack as the first provider. The framework (`OAuthIntegration` ABC and registry in `cognee/modules/integrations/`, a generic per-provider connect/callback/disconnect router in `cognee/api/v1/integrations/routers/get_integrations_router.py`) stores third-party credentials AES-256-GCM-encrypted in a new `integration_credentials` table, with key-rotation support via the `INTEGRATION_CREDENTIALS_KEYS` / `INTEGRATION_CREDENTIALS_ACTIVE_KEY_ID` env vars (legacy single-key `INTEGRATION_CREDENTIALS_KEY` still works). The Slack integration adds workspace connect/disconnect with token refresh and revoke, a `/cognee-ask` slash command answering via `HYBRID_COMPLETION` search through Slack's `response_url`, a per-channel allowlist, an App Home tab, a "Remember this" message shortcut that saves messages via `cognee.remember()`, request verification via `SLACK_SIGNING_SECRET` (plus `SLACK_`-prefixed app credentials), and a matching frontend Integrations page. The table is created by new Alembic revision `b2c4d6e8f0a1`, so existing deployments must run the migration (`cognee.run_migrations()`, or `alembic upgrade head`) to create it; the revision is guarded and skips if the table already exists. The core SDK pipeline (`add`/`cognify`/`search`) is unchanged (PR #4175). * Fixes `cognee-cli --api-url <cloud-url>` commands failing against reachable cloud tenants with a bogus "Cannot connect to Cognee API… Is the server running?" error. `api_dispatch.dispatch()` ran a fatal pre-flight `client.health()` probe before every command, using an unauthenticated one-off `httpx` client against the DB-backed `/health` endpoint and mapping any exception — including 503, 404, and 401 responses — to that message. The probe is removed: the real command runs immediately, and only genuine transport failures (classified by the new `is_connection_error()` helper, matching `httpx.TransportError`) are translated into a friendly message that includes the attempted URL; real HTTP errors such as 401 now surface their actual detail. Dataset collection calls (`datasets_list`, `datasets_create`, `datasets_delete_all`) now use the canonical trailing slash on `/api/v1/datasets/` and the shared client sets `follow_redirects=True`, so both cloud and local OSS deployments work; `health()` remains available as an explicit check, reuses the pooled authed client, and returns the body on 503 instead of raising. No CLI flags, config options, or environment variables changed (CLO-321, PR #4189). * Adds an opt-in, server-side default synthesis prompt for the MCP `recall` tool. Previously the per-call `system_prompt` argument (added in PR #4122) was the only way to override the terse `RecallPayloadDTO` default, so a shared cognee deployment could not define a default synthesis policy — every MCP client had to resend the full prompt on each call. `CogneeClient.recall` in `cognee-mcp/src/cognee_client.py` now resolves a default when the caller omits `system_prompt`: `COGNEE_MCP_RECALL_SYSTEM_PROMPT` supplies inline prompt text, and `COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE` points to a file holding it (inline takes priority; an unreadable file logs a warning and is skipped). Precedence is explicit caller argument, then env default, then backend default, and the resolution happens before the API/SDK branches so both modes are covered. With neither variable set, behavior is unchanged — no `system_prompt` is added to the payload — and the `search` tool is deliberately unaffected (PR #4205). * Fixes `forget(everything=True)` appearing to hang for 15+ minutes when deleting thousands of vector rows from LanceDB, the default vector backend. `LanceDBAdapter.delete_data_points` issued one `collection.delete` per id; each delete is a LanceDB commit that appends a table version, and manifest listing degrades as versions accumulate, so a 13,207-id wipe became 13k increasingly slow commits (\~7 deletes/sec and falling) that no per-call timeout or worker-death check could catch. Deletes now run as `id IN (...)` predicates in batches of 1,000, controlled by the new `DELETE_PREDICATE_BATCH_SIZE` class attribute; per-id single-quote escaping is preserved, sequential batches from one caller cannot commit-conflict, and missing ids or collections remain no-ops. The measured 13,207-id delete drops from 30+ minutes to about 0.1 seconds. Other providers (PGVector, Turso, hybrid Postgres, Neptune) already batched and are unchanged; no public API signature, configuration option, or environment variable changed (PR #4235). * Fixes Ollama model name handling that persisted after PR #3994's provider inference: when `LLM_MODEL` is set to a litellm-style prefixed name such as `ollama/llama3.1:8b`, the provider was correctly inferred as Ollama, but `OllamaAPIAdapter` stored the model name verbatim and sent the full prefixed string to Ollama's OpenAI-compatible endpoint, where no model by that name exists. The adapter's `__init__` (in `cognee/infrastructure/llm/structured_output_framework/litellm_instructor/llm/ollama/adapter.py`) now strips the prefix with `model.removeprefix("ollama/")` when the name starts with `ollama/`, so both `llama3.1:8b` and `ollama/llama3.1:8b` resolve to the bare model name `llama3.1:8b`. Unprefixed model names behave exactly as before, and no public API signature, configuration option, or environment variable changed; new unit tests cover both spellings and the inferred-provider path end to end (PR #4243). * Fixes a `TypeError` when the shared storage `JSONEncoder` encountered a plain `datetime.date` value. JSON Schema fields declared with `format: "date"` are materialized as `datetime.date` objects, but `JSONEncoder.default` in `cognee/modules/storage/utils/__init__.py` only handled `datetime`, `UUID`, and `Decimal`, so dates fell through to the standard library encoder and raised. A new `isinstance(obj, date)` branch now serializes plain dates with `date.isoformat()` as ISO-8601 strings; it is placed after the existing `datetime` check (which matches first, since `datetime` is a `date` subclass), so datetime serialization is byte-for-byte unchanged, as are UUID and Decimal handling, and unsupported objects still raise `TypeError`. No public API signature, configuration option, or environment variable changed (fixes #4239, PR #4244). * Fixes `parse_timestamp` in `cognee/modules/migration/cogx.py` — the helper feeding all five migration source adapters (Mem0, Zep, Graphiti, Letta, LangMem) and `export.py` — returning a mix of timezone-aware and naive datetimes: epoch values and ISO strings with a `Z` or offset came back UTC-aware, while offset-less ISO strings, bare dates, and naive `datetime` passthrough stayed naive. `LettaSource` can see both shapes in one conversation (Letta serializes `created_at` with or without an offset depending on version), which made `render_episode`'s `.timestamp()` sort apply the importing machine's local timezone — reordering transcript turns on any machine not set to UTC — and made direct naive/aware comparisons raise `TypeError`. `parse_timestamp` now always returns timezone-aware UTC datetimes (or `None`): offset-less inputs get `tzinfo=timezone.utc`, matching what the exporting systems store, while values with an explicit offset are preserved. `external_created_at`/`external_updated_at` metadata and rendered episode timestamps now carry `+00:00` where they were previously naive; no function signature or configuration changed (PR #4248). * Fixes the Letta and Zep migration adapters silently importing nothing when an export carries an empty key alias ahead of the populated one. The `_first_list(container, *keys)` helper, duplicated in `cognee/modules/migration/sources/letta.py` and `zep.py`, resolves the different spellings exports use for the same collection (e.g. `messages`/`in_context_messages`/`message_history` in Letta, `facts`/`edges`/`entity_edges` in Zep), but it returned on the first key whose value was a list — and since `[]` is a list, an alias that was present but empty short-circuited the scan. A Letta file with `messages: []` and the real history under `in_context_messages` therefore imported nothing, with `records()` yielding an empty stream and the import reporting success; six call sites across the two adapters were affected. The helper now keeps scanning until an alias actually yields dict records. Behavior changes only for inputs that previously imported nothing: a populated alias still wins in the same priority order, and all-empty inputs still yield nothing (PR #4253). * Fixes `recall()` auto-routing sending natural-language questions to coding-rules retrieval because they happened to contain a code token. In the rule-based router (`cognee/api/v1/recall/query_router.py`), the incidental code-token pattern (`def `, `return `, `async `, `await `, `import `, `class X(`, `.py`, `function x(`) scored `3.0` toward `CODING_RULES` — enough to clear the router's `2.0` default threshold and win outright, so questions like "Describe the import process for customer records" or "What does the return policy say?" were routed to coding rules, and the token could also outrank stronger intent cues in the same query. That weight is now `1.0`, which sits below the default threshold: an incidental code token can no longer select a search mode by itself, and it loses to any stronger cue. Explicit coding vocabulary (`coding rules`, `code review`, `best practice`, `lint`, `refactor`) keeps its `5.0` weight and still routes to `CODING_RULES`. User impact: affected queries now resolve to `GRAPH_COMPLETION` or to whichever cue actually dominates, so the `search_type`/`kind` reported on their results can differ from before; pass `query_type` explicitly to bypass the router entirely. No public API signature, configuration option, environment variable, or migration changed (PR #4207). * Fixes edge-filtered neighborhood queries crashing on the default Ladybug (Kuzu) graph backend. `LadybugAdapter.get_neighborhood(edge_types=[...])` built its Cypher with an `ALL(rel IN r WHERE rel.relationship_name IN $edge_types)` predicate over the variable-length relationship binding `r` from `-[r*1..depth]-`; in Kuzu `r` is a `RECURSIVE_REL` rather than a `LIST`, so `ALL(...)` is a binder-type mismatch that, combined with the parameter reference, drove the engine into a failed internal assertion. Because Ladybug is the default backend, any edge-filtered neighborhood query failed out of the box (Neo4j and Postgres were unaffected); the built-in search and visualization flows never pass `edge_types`, so the crash surfaced for programmatic callers of the public `get_neighborhood` primitive — custom retrievers and SDK code supplying an edge-type allow-list. The `ALL()` predicate is now dropped; when `edge_types` is supplied the adapter fetches the paths unfiltered and post-filters in Python, keeping a neighbor iff some path reaching it within `1..depth` hops has every edge type in the allowed set (undirected) — the same semantic Neo4j and Postgres already enforce. The behavioral tradeoff: the filtered branch enumerates every path up to `depth` before post-filtering, so its cost grows combinatorially with node degree × depth on dense graphs; it is best suited to shallow, targeted neighborhoods. The fast `edge_types=None`/`[]` path is unchanged. No public API signature, configuration option, or environment variable changed (fixes #3585, carries community fix #3591 by @ly-wang19, PR #4156). * Surfaces dataset and data-item **names** in the text channel of the `cognee-mcp` JSON tools `list_datasets_json` and `list_dataset_data_json`. Previously both tools put names only in `structuredContent` and emitted just a count in `content[0].text` (e.g. `8 dataset(s).`), so text-only MCP clients — such as agents in Cursor that never see `structuredContent` — could not tell what existed and fell back to raw HTTP. A new shared `_format_named_items` helper in `cognee-mcp/src/server.py` now renders one `- name (id)` line per item into the text content (falling back to `- name` when an id is absent and `(unnamed)` when a name is absent), prefixed by a header line (`8 datasets:`, or `1 dataset:` in the singular). The text is capped at 50 items, with a trailing `… and N more (see structuredContent).` note when the list is longer; an empty list reads `No datasets found.` / `No data items found.`. The change is backward-compatible and text-only: `structuredContent` (`{datasets: [...]}` / `{data: [...]}`), both tool schemas, and the Cognee workspace UI are unchanged, and no configuration option or environment variable was added (CLO-319, PR #4186). * Fixes the `cognee-mcp` client failing or hanging when listing datasets or checking status against Cognee Cloud, and makes API-mode `cognify` submit a background run instead of blocking. In API mode the client's `list_datasets` requested `/api/v1/datasets` (no trailing slash) and relied on the server's 307 redirect to the canonical `/api/v1/datasets/`; the client does not follow redirects, so the call failed with an HTTP error on the redirect response, and the redirect `Location` could additionally downgrade to `http://` against the HTTPS-only edge (see the server-side fix, CLO-320). `list_datasets` now calls the canonical trailing-slash route `/api/v1/datasets/` directly, so no redirect is involved. Separately, the client applied its single client-wide `timeout=300.0` to every request, so a hung or black-holed read-only GET froze the caller for a full five minutes; a per-request `READ_TIMEOUT_SECONDS = 30.0` is now applied to the dataset **list** and **status** GETs, while the 300s client-wide default remains for other requests. `READ_TIMEOUT_SECONDS` is a hardcoded module constant in `cognee-mcp/src/cognee_client.py`, not an environment variable or documented setting; the tradeoff is that a valid but very slow (>30s) GET will now time out. Finally, the client's `cognify` POST now sends `run_in_background: true`, so the request submits the pipeline run on the server and returns immediately instead of holding the HTTP request open for the whole run; the MCP `cognify` tool already returned immediately and directs callers to poll dataset status, which now reflects the server-side background run. No public MCP tool signature, configuration option, or environment variable changed (CLO-322, PR #4184). * Fixes OpenRouter embedding requests failing with a `400 invalid_value` error on `encoding_format`. Older LiteLLM releases serialize an *omitted* `encoding_format` as JSON `null`; OpenAI tolerates it, but OpenRouter rejects it (it accepts only `"float"` or `"base64"`), so an OpenRouter embedding config (`EMBEDDING_PROVIDER="custom"`, `EMBEDDING_MODEL="openrouter/openai/text-embedding-3-small"`) could 400 on every embedding call. `LiteLLMEmbeddingEngine.embed_text` now sets `encoding_format="float"` whenever it detects an OpenRouter route — a model id beginning with `openrouter/`, an explicit `openrouter` provider, or an `openrouter.ai` endpoint host (all matched case-insensitively). Cognee always consumes float vectors, so the value is safe to make explicit. The guard is scoped narrowly to OpenRouter on purpose: Cognee does not enable `litellm.drop_params`, and `encoding_format="float"` would raise `UnsupportedParamsError` for providers such as gemini/bedrock/vertex\_ai, so it is not applied unconditionally. On current LiteLLM versions the guard is a no-op for `openrouter/`-prefixed models (litellm's dedicated OpenRouter branch drops the omitted parameter), but endpoint-based configs — an unprefixed model pointed at an `openrouter.ai` endpoint — route through litellm's OpenAI handler, which injects the `null` even on current versions, so the guard is what fixes those. The change requires no user action, and no public API signature, configuration option, or environment variable changed (SDK-311, fixes #3660, PR #4195). * Adds a `COGNEE_EXTRAS` Docker build argument to the root `Dockerfile`, so optional-dependency groups can be baked into a source-built API image without editing the `Dockerfile`. The argument takes a space-separated list of extra names — `docker build --build-arg COGNEE_EXTRAS="docs langchain" -t cognee-custom .` — which the build expands into `--extra <name>` flags on top of the existing default set (which has since grown to include the `aws` extra, so `aws` no longer needs to be passed). It is applied to **both** `uv sync` steps: the second sync is exact and would otherwise drop extras installed only in the dependency-cache layer, so applying it twice is what makes the packages reach the final runtime stage. The argument defaults to an empty string, making it a no-op, so existing builds are unaffected. Both `uv sync` invocations keep `--frozen`, so the argument only selects extras already resolved in `uv.lock` rather than resolving new dependencies — builds stay deterministic and no lockfile change is required. Note that the `cognee` service's `build:` block in `docker-compose.yml` has no `args:` entry, so `docker compose up --build cognee` does not forward the value until you add one. The argument is declared only in the root API `Dockerfile`; the MCP and frontend images do not accept it. No public API signature, configuration option, or environment variable changed (SDK-314, PR #4211). * Stamps messages remembered through the Slack integration with a `slack` node set, so Slack-sourced data carries a structured origin marker into the graph. Slack's "Remember this" shortcut called `remember()` with only `dataset_name="slack"` and no `node_set`, so the resulting document reached the graph with no Slack marker at all — no `NodeSet` node, no `belongs_to_set` edge, no `source_node_set` property — and the only trace of where the content came from was the relational dataset name and the English prefix baked into the message text (`In #channel, <@user> said: …`). `remember_message` now passes `node_set=SLACK_NODE_SET` (a new module constant equal to `["slack"]` in `cognee/modules/integrations/slack/remember_message.py`), which materializes a `slack` [NodeSet](/core-concepts/further-concepts/node-sets) node with `belongs_to_set` edges that propagate from the document down to its chunks and extracted entities, and sets the `source_node_set` property the pipeline carries across task boundaries. The practical effect is that Slack items gain the same source dimension every other ingestion path already had: they are grouped and colored by origin in graph visualization, and retrieval can be scoped to them with `recall(..., node_name=["slack"])`. The change is non-breaking and requires no user action — Slack data already in the graph is untouched and is only marked if those messages are remembered again — and no public API signature, configuration option, or environment variable changed (SDK-318, PR #4216). * Reduces CPU spent on openai-python's per-response type introspection in Cognify and other LLM-heavy workloads. On every API response the OpenAI SDK rebuilds its `ChatCompletion` response tree, repeatedly calling `get_origin`, `get_args`, `is_annotated_type`, and `is_literal_type` against the same static types; at cognify scale that introspection dominated CPU. A new internal module `cognee/infrastructure/llm/openai_type_cache.py` wraps each of those four helpers in a `functools.lru_cache(maxsize=4096)` and rebinds the cached versions at every known openai import site — patching the source module alone is insufficient, because most call sites use `from ._compat import get_origin` and capture the name at import time. The install runs once and is idempotent, triggered when `cognee.infrastructure.llm` is imported, which any normal Cognee usage already does before an OpenAI SDK call is made; no user action or configuration is required. In an end-to-end 200-document Cognify run, CPU time dropped from \~93.85s to \~67.70s (≈28%). Failure modes degrade safely rather than break: the rebind targets are openai-python **private** modules (`openai._models`, `openai._utils._compat`, `openai._utils._typing`, `openai._response`, `openai._legacy_response`, `openai._base_client`) that are not covered by its compatibility guarantees, so if a future openai release moves them the import fails, `install()` becomes a no-op, and Cognee keeps running against the uncached originals; likewise an unhashable argument falls back to the original uncached helper, and any call site that is not rebound simply stays uncached. No public API signature, configuration option, or environment variable changed (COG-5963, PR #2876). * Adds `LangMemSource`, a memory-migration source for importing LangMem memories, so LangMem joins Mem0, Letta, Zep/Graphiti, and COGX archives as a system `cognee.remember()` can import from. Construct it with `LangMemSource(data, mode="re-derive")` — exported from `cognee.modules.migration.sources` — where `data` is a path to a LangMem JSON export, an already-parsed list (for live-API use, e.g. a response fetched with the LangMem client), or a dict wrapping the list under `memories`, `results`, `items`, or `data`; any other shape raises `ValueError`. Each item becomes a COGX **memory** record: content is the first string present among `content`, `text`, `memory`, `data`, and `message` (items with none of those are skipped), `user_id` falls back to `namespace` for the record's scope (which also carries `agent_id`, `session_id`, and `run_id` when present), `categories` accepts a single string or a list, `created_at`/`createdAt`/`timestamp` and `updated_at`/`updatedAt` supply timestamps, `id` falls back to a positional `langmem-<index>` identifier, and any `metadata` is carried over nested under `langmem_metadata`. The mode default is the base-class `re-derive`, which suits LangMem's free-form text (it carries no derived graph of its own to preserve). The change is additive: no existing source, `remember()` signature, configuration option, or environment variable changed. See [Migrate Memory Systems with COGX](/examples/migrate-memory-systems) (PR #4208). * Fixes ingested document names being recorded percent-encoded, and platform-foreign paths being recorded whole. `get_file_metadata` derived the document name (`FileMetadata["name"]`, persisted as `Data.name`) with `Path(file_path).stem`, where `file_path` is the opened stream's `.name`. In the ingestion pipeline that value is a percent-encoded `file://` URI — `LocalFileStorage` wraps every opened file in a `FileBufferedReader` named `Path(full_path).as_uri()` — so escapes leaked into the stored name on every platform: a file named `Annual Report.pdf` was recorded as `Annual%20Report`. `Path` is also OS-specific, so a raw backslash path from another caller (`C:\Users\me\report.pdf`) yielded the whole path minus its extension as the "stem" on POSIX (the reverse direction was never broken: Windows path handling accepts `/` as a separator). A new `_derive_basename` helper now percent-decodes `file://` URIs (`unquote(urlparse(...).path)`) and resolves the basename with `PureWindowsPath`, which treats both `/` and `\` as separators on every host OS, so those inputs are recorded as `Annual Report` and `report`; a degenerate input that yields an empty stem becomes `None`, letting the caller fall back to an explicitly supplied filename. The prior extension-less stem semantics are preserved: only the last suffix is stripped (`archive.tar.gz` → `archive.tar`), dotfiles such as `.gitignore` stay intact, and the extension continues to be stored separately in `FileMetadata["extension"]` — persisted for the original file as the `Data.original_extension` column, while `Data.extension` describes the stored text representation (typically `txt`). Separately, `classify()`'s fallback for `BufferedReader` / `SpooledTemporaryFile` inputs passed without an explicit `filename` now derives the basename with `str(data.name).replace("\\", "/").split("/")[-1]` instead of splitting on `/` alone (the same normalization as `_normalize_filename` in `cognee/tasks/ingestion/utils.py`), so a Windows-style stream name resolves to `report.pdf` rather than the entire path; this is a defensive path, because in the ingestion pipeline `classify()` only ever sees a `file://` URI. The change is backward-compatible and requires no action: POSIX paths and unencoded names resolve exactly as before, deduplication remains content-hash based so no record changes identity, and no public API signature, configuration option, environment variable, or database migration changed. Names already stored before the fix are not rewritten automatically, though re-adding a file refreshes its stored name through the re-ingest update path (SDK-237, PR #4157). * Fixes `cognify` failing on dlt-backed sources when the schema and foreign-key edges are registered in the relational rollback ledger. `extract_dlt_fk_edges` built its edge tuples with stringified node ids in the source/destination slots (`str(source_table_id)`, `str(relationship.id)`, and the `doc_id`/`target_data_id` strings it carries in its document map), but the same tuple list is handed both to `graph_engine.add_edges` and to the ledger's `upsert_edges`, which declares `List[Tuple[UUID, UUID, str, Dict]]` and binds slots 0/1 straight into the `edges` table's `source_node_id`/`destination_node_id` columns — both typed `UUID(as_uuid=True)`. SQLAlchemy's UUID bind processor reads `.hex` off the bound value, so the ledger insert raised `AttributeError: 'str' object has no attribute 'hex'` and failed the run. The failure was not limited to schemas that declare foreign keys: the per-row `is_row_of` edge linking each dlt row document to its `SchemaTable` node is emitted for every dlt row, so CSV, SQL connection-string, Gmail, and Slack-export ingestion were all affected. Slots 0/1 now carry `UUID` objects (including `UUID(doc_id)` and `UUID(target_data_id)` for the row-level edges), matching the tuple contract `upsert_edges` and the standard `get_graph_from_model` path already rely on, while each edge's JSON attribute dict keeps its string `source_node_id`/`target_node_id` copies, so stored edge properties are unchanged. Runs whose provenance is stamped in-graph skip the ledger write and were never affected. No public API signature, configuration option, environment variable, or database migration changed (SDK-183, PR #4081). * Doubles the remote request timeouts used by the `cognee.serve()` client, so long-running server-side work is no longer cut off mid-flight. `CloudClient` bounds ordinary remote operations (`remember`, `recall`, `improve`, `add`, `cognify`, `search`, `forget`) with a client-wide `aiohttp` total timeout, and applies a separate per-request timeout to archive uploads (`content_type="cogx-archive"`, the `cognee.push()` path) whose total is uncapped in favour of a per-read inactivity bound. Both limits were 300 seconds, which a blocking `cognify()` over a large dataset — or an archive upload plus its synchronous server-side import — could legitimately exceed, aborting the call client-side even though the server was still making progress. `DEFAULT_TIMEOUT` is now `total=600` and `UPLOAD_TIMEOUT` is now `sock_read=600`; the `sock_connect=30` connect bound and the uncapped upload total are unchanged, as is the two-tier structure itself (only the two 300s values moved). Both are hardcoded module constants in `cognee/api/v1/serve/cloud_client.py`, not environment variables or documented settings, so there is no way to tune them per deployment; the tradeoff is that a genuinely hung request now blocks for up to ten minutes rather than five. The change is backward-compatible, and no public API signature, configuration option, or environment variable changed (SDK-284, PR #4145). *** ## v1.4.0.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev1)** Development pre-release that bumps the package version from `1.4.0.dev0` to `1.4.0.dev1` (the tag sits on the SQL session-cache fix merge rather than a separate release-cut commit). No new Alembic revision ships in this cut. ### Highlights * Gates delete operations behind the same per-dataset lock that serializes pipeline runs. The in-process asyncio lock registry moves from `cognee/modules/pipelines/operations/pipeline.py` into the new `cognee/infrastructure/locks/dataset_lock.py`, exporting `dataset_lock`, `get_dataset_lock`, and `held_datasets`; pipeline runs and deletes now acquire from one shared registry. `datasets.delete_dataset` and `datasets.delete_data` in `cognee/api/v1/datasets/datasets.py`, plus `cognee.forget`'s dataset- and data-level memory clearing in `cognee/api/v1/forget/forget.py`, now run inside `async with dataset_lock(dataset_id)`, so a delete waits for any in-flight `add`/`cognify`/`memify` run on the same dataset (and vice versa) and two deletes on one dataset are serialized, while different datasets still proceed in parallel. The lock is re-entrant per execution context via the `held_datasets` `ContextVar` and remains process-local (asyncio) — it does not protect against multiple processes or workers touching the same dataset. No public API signature, configuration option, or environment variable changed (PR #4042). * Changes `cognee.improve()` so session-persistence failures are no longer swallowed. `_bridge_sessions` in `cognee/api/v1/improve/improve.py` wrapped its `persist_sessions_in_knowledge_graph_pipeline(...)` call in a blanket `try/except Exception` that logged `improve: session persistence failed (non-fatal)` as a warning and let the remaining improve stages (trace persistence, agent-context extraction, session distillation) run as if bridging had succeeded. That handler is removed: exceptions now propagate out of `improve()` (a `finally` still releases the per-session improve lock), and error handling is delegated to the pipeline system, which already records pipeline-run failures where suppression is appropriate. Callers that relied on `improve()` never raising during session bridging will now see those exceptions and can react to real failures instead of getting silent partial results. No public API signature, configuration option, or environment variable changed (PR #4056). * Adds a first-class `dataset_id` keyword parameter to `cognee.remember()` and routes it through improve operations. Previously `dataset_id` was only an untyped pass-through kwarg (listed in `RememberKwargs`/`_ADD_ONLY` and forwarded to `add()`), and the `self_improvement=True` paths always invoked `improve()` with `dataset_name` — so improve could not target a dataset addressed by UUID. `remember()` in `cognee/api/v1/remember/remember.py` now declares `dataset_id: Optional[UUID] = None` (takes precedence over `dataset_name`), resolves or creates the dataset before the session branch, and both the permanent and session self-improvement paths call `improve(dataset=dataset_id or dataset_name, ...)`. `RememberResult` fills `dataset_name` from the pipeline run when only an id is supplied, session-mode results now include `dataset_id`, and `CloudClient.remember` forwards the id as a `datasetId` form field. Passing `dataset_id` with `MemorySource` imports or typed `MemoryEntry` payloads raises `ValueError`; those stay dataset-name based. No env vars or migrations involved (PR #4158). * Fixes local-mode tenant context in the Cognee UI and batches related frontend/backend changes. `TenantContext`/`useTenant()` now expose `tenantReady`, `podUnreachable`, `isOwner`, `availableTenants`, and `releaseLoader`, and `LocalProvider` supplies the same shape, giving self-hosted (local mode) sessions the identical context contract as cloud tenants; tenant pod domain resolution is centralized in `getTenantApiDomain.ts` (explicit `NEXT_PUBLIC_TENANT_API_DOMAIN`, else parsed from `NEXT_PUBLIC_MANAGEMENT_API_URL`). Every `createHttpClient()` instance now attaches an `X-Request-Id` correlation header (the opt-in `setup.ts` registration is removed). Uploads in `rememberData.ts` pass `timeoutMs` to the shared HTTP client instead of racing a local `AbortController` against the client's 30-second default, restoring the intended 5-minute upload window, and `DatasetsPage` reports upload failures and knowledge-graph build failures separately while enforcing `MAX_FILES_PER_UPLOAD`. On the backend, a new `GET /api/v1/datasets/graph-summary` endpoint returns per-dataset node/edge counts cached in `GraphMetrics` per latest cognify `pipeline_run_id`, far cheaper for status polling than the full graph endpoint (PR #4179). * Fixes the Mistral transcription adapter sending an entire Windows path as the API `file_name`. `MistralAdapter.create_transcript` derived the file name with `input.split("/")[-1]`; on Windows the audio path uses backslash separators (e.g. `C:\audio\clip.mp3`) and contains no forward slashes, so the split left the value unchanged and the whole path — rather than the basename `clip.mp3` — was sent to the Mistral transcription API. The basename is now derived with `str(input).replace("\\", "/").split("/")[-1]`, normalizing both `\` and `/` separators (the same handling used by `_normalize_filename` in `cognee/tasks/ingestion/utils.py`), so Windows and POSIX paths both send just the file name. The change is backward-compatible: POSIX paths resolve to the same basename as before, and no public API signature, configuration option, or environment variable changed (fixes #3587, PR #3588). * Fixes writes failing on Postgres-backed graph and vector stores when node/edge fields or vector payloads contain NUL bytes (`\u0000`). Postgres text columns and JSONB reject the `\u0000` escape, and while the vector store's `json` column accepts it on insert, the `payload::jsonb` casts used by search/merge queries later reject it — so ingesting content with an embedded NUL byte could error. A shared `sanitize_relational_payload` helper now strips NUL bytes from strings and recurses through nested containers (dicts, lists, tuples), decoding `bytes`/`bytearray` values as UTF-8 with replacement so invalid byte sequences do not break persistence. It is applied in the Postgres graph adapter to the `id`, `name`, `type`, and `properties` of nodes and to the `source_id`, `target_id`, `relationship_name`, and `properties` of edges (ids sanitized identically on both sides so references stay consistent), and in `PGVectorAdapter` to each data point's serialized payload. This is an internal serialization fix for the Postgres graph and PGVector adapters; no public API signature, configuration option, environment variable, or database migration changed (PR #4153). * Fixes ingestion crashing on Windows when a string starts with `/` or is drive-relative. In `save_data_item_to_storage`, the absolute-path branch treated any string beginning with `/` (or, on Windows, one whose second character is `:`) as a local file path and called `Path(...).as_uri()` on it. On Windows, `os.path.normpath("/etc/hosts")` yields a *drive-relative* path and a drive-relative input like `C:notes.txt` stays drive-relative, so `as_uri()` raised `ValueError` (relative paths cannot be expressed as `file:` URIs) — any POSIX-style path string or plain text note starting with `/` crashed `add()`. The branch is now additionally guarded by `Path(os.path.normpath(data_item)).is_absolute()`, so on the current platform only genuinely absolute paths convert to a `file:` URI; non-absolute `/`-prefixed or drive-relative strings fall through to the existing relative-path/text handling and are ingested as text (saved to Cognee's data storage as a text file). POSIX behavior is unchanged (`/...` paths still convert to `file:` URIs) and genuine Windows absolute paths (`C:\...`) still convert as before. No public API signature, configuration option, or environment variable changed, and `accept_local_file_path` continues to govern acceptance of true absolute paths (fixes #3887, PR #3892). * Fixes S3 ingestion failing on Windows with `PermissionError` (WinError 32). In `data_item_to_text_file`, the S3 branch downloaded the object into a `tempfile.NamedTemporaryFile` created with the default `delete=True` and then passed `temp_file.name` to the loader, which reopens the file by name while Cognee's handle is still open. On Windows that reopen raises `PermissionError [WinError 32]`, so every S3 ingestion failed. The temp file is now created with `delete=False`, its handle is flushed and closed before the loader reopens it, and it is removed with `os.unlink` in a `finally` block (guarded against `OSError`) so no temp file is leaked — mirroring the `delete=False` pattern already used by the SQLAlchemy and ladybug S3 temp-file paths. POSIX behavior (Linux/macOS) is unchanged and temporary files are still cleaned up after use. No public API signature, configuration option, or environment variable changed (fixes #3339, PR #3340). * Fixes two failures in the SQL session-cache backend (`CACHE_BACKEND=postgres` / `sqlite`) when ids are UUID-like or when many writers target the same session concurrently. First, cache key columns (`user_id`, `session_id`, `qa_id`, `entry_id`, `log_key`, and the KV `key`) now use a `StringKey` `TypeDecorator` that coerces stringable ids such as `uuid.UUID` to `str` in the bind processor: the asyncpg dialect renders explicit bind casts, so an id bound as a `uuid.UUID` made Postgres parse `text = uuid` and raise `42883` ("operator does not exist"), while SQLite rejected the non-str bind outright — passing a `uuid.UUID` id (rather than a string) to the adapter could therefore fail to read or write. The decorator normalizes every read and write to the same string regardless of the caller's type; because DDL is delegated to the underlying `Text` impl, the emitted column stays plain TEXT and existing tables need no migration. Second, on Postgres each same-session write transaction now takes a transaction-scoped `pg_advisory_xact_lock` keyed by `(table, user_id, session_id)` before writing, so concurrent writers to one session queue instead of deadlocking on the sliding-TTL UPDATE (SQLSTATE `40P01`); the lock auto-releases at COMMIT/ROLLBACK, `delete_session` acquires the per-table locks in a fixed order so a multi-table writer can't cycle with single-table writers, and the whole mechanism is a no-op on SQLite (which serializes writers with its own single-writer lock). The tradeoff is that concurrent writes to the *same* session may serialize slightly; writes across different sessions are unaffected. No public API signature, configuration option, environment variable, or database migration changed (PR #4182). *** ## v1.4.0.dev0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev0)** First development pre-release after v1.4.0, bumping the package version from `1.4.0` to `1.4.0.dev0` and re-resolving `uv.lock`. Note that cognee cuts its `.devN` pre-releases after the corresponding stable release, so despite what PEP 440 version ordering suggests, `1.4.0.dev0` is newer than `1.4.0`. No new Alembic revision ships in this cut. ### Highlights * Fixes two crash/debuggability issues around telemetry and CLI user resolution. `send_telemetry()` in `cognee/shared/utils.py` called `asyncio.get_running_loop()` unconditionally, so in a sync context or after event-loop shutdown it raised `RuntimeError` and crashed the caller mid-pipeline; the `get_running_loop()`/`create_task()` preamble is now wrapped in `try/except RuntimeError`, making telemetry genuinely best-effort — the event is dropped instead of taking down the pipeline. Separately, `resolve_cli_user()` in `cognee/cli/user_resolution.py` caught bare `except Exception:`, so database connectivity or ORM failures were silently treated as "user not found"; the handler is narrowed to `EntityNotFoundError` (what `get_user()` raises for an unknown UUID), so genuine infrastructure errors now propagate with their original traceback. No public API signature, configuration option, or environment variable changed, and no action is needed (PRs #3276 and #3327). * Fixes three gaps in the recall API's result normalization that left agents and MCP clients with incomplete metadata. `AGENTIC_COMPLETION` results, previously normalized as `kind: "unknown"`, now map to `SearchResultKind.GRAPH_COMPLETION` in `cognee/modules/recall/methods/normalize_search_payload.py`. `FEELING_LUCKY` searches are resolved via `select_search_type()` in `cognee/modules/search/methods/get_retriever_output.py` before the retriever is instantiated, so the payload (and tracing span) carries the effective search type — e.g. `CHUNKS` — instead of the unresolved sentinel that also normalized to `"unknown"`. And when a completion entry is a Pydantic `BaseModel` (a `response_model` search), the item now gets `kind: "structured"` with the `structured` field populated from `model_dump()` while `text` stays renderable, where `structured` was previously always `null`. Normalization-layer only: no retriever ranking, public API signature, configuration option, or environment variable changed (fixes #3820, PR #3822). * Adds optional TLS to the Redis cache adapter, which previously had no SSL path and would hang against managed Redis endpoints with in-transit encryption (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis) until `socket_timeout` and then raise `CacheConnectionError`. Two new `CacheConfig` settings — `cache_ssl` (bool) and `cache_ssl_cert_reqs` (`"required"`/`"optional"`/`"none"`), settable via the `CACHE_SSL` and `CACHE_SSL_CERT_REQS` environment variables — are threaded through `get_cache_engine`/`create_cache_engine` into `RedisAdapter`, which forwards `ssl` and `ssl_cert_reqs` to both its sync and async `redis.Redis` clients. This brings the Redis cache to parity with the Postgres and pgvector adapters, which already read SSL settings from `DATABASE_CONNECT_ARGS`. Defaults are off and `"required"`, so existing deployments connect exactly as before; to enable TLS, set `CACHE_SSL=true` and adjust `CACHE_SSL_CERT_REQS` for self-signed certificates (fixes #3850, PR #3851). * Changes the four session-context methods on `SessionManager` (`create_session_context_entry`, `get_session_context_entries`, `update_session_context_entry`, `delete_session_context` in `cognee/infrastructure/session/session_manager.py`) to let `SessionParameterValidationError` propagate on an empty or whitespace `user_id`/`session_id`, matching `add_qa` and every other method in the class. Previously each wrapped `_validate_session_params` in `try/except Exception: return False` (or `[]`), so a caller bug like `user_id=""` was indistinguishable from an unavailable cache — and unlike the cache path, without even a `logger.warning`. The fail-open behavior for infrastructure errors is unchanged: when the cache is unavailable or the cache operation fails at runtime, the methods still return `False`/`[]` with a log message rather than raising, and behavior for valid inputs is identical. Callers passing invalid IDs will now see an exception instead of a silent `False`; no public API signature, configuration option, or environment variable changed (PRs #3881 and #3911). * Adds a pre-flight dry-run token/cost estimator and makes LLM quota/billing exhaustion fail fast instead of retrying. `remember(dry_run=True)`, `cognify(dry_run=True)`, and the CLI `--dry-run` flag on both commands return a stage-level estimate (structured graph extraction plus chunk summarization) of LLM tokens and rough USD cost with no LLM calls, no ingestion, and no graph writes; the estimator (`cognee/modules/cognify/estimator.py`) reuses the real pipeline's document classification, chunker, and prompt templates, resolves datasets read-only, and rejects unsupported inputs (remote URLs, directories, binary formats, session memory, remote `serve()` mode) loudly rather than mis-estimating. Separately, a shared `llm_retry_condition` in `cognee/infrastructure/llm/retry_config.py` now governs every structured-output adapter and BAML, treating quota/billing wordings like `insufficient_quota` as terminal — `LLMGateway.acreate_structured_output` converts them into an actionable `LLMQuotaExceededError` — while transient rate limits (including Gemini free tier's recoverable "exceeded your current quota") still retry. `dry_run` defaults to `False`, so existing calls are unaffected (SDK-136, fixes #3643, PR #3974). * Adds an optional `LLM_PROVIDER=mcp-sampling` backend that lets cognee, when running as an MCP server (`cognee-mcp`) inside a host such as Claude Code or Cursor, delegate all LLM completions to the host's own model via MCP's `sampling/createMessage` — so no `LLM_API_KEY` is needed (the provider is excluded from the API-key-required set; `LLM_MODEL` is only a hint, the host picks the model). The new `MCPSamplingAdapter` reads the host session from the MCP SDK's per-request context via `get_sampling_session()`, so `cognee-mcp/server.py` needed no changes; structured output is produced by embedding the response model's JSON Schema in the prompt with a bounded validate/repair loop, since sampling returns free text only. It fails closed with an actionable `MCPSamplingUnavailableError` when cognee is not running as an MCP server or the host did not grant the `sampling` capability. Completions only — embeddings, audio, and vision are not covered, so configure an embedding provider for vector search; non-MCP usage and all existing providers are unchanged, and `mcp` remains an optional dependency (SDK-139, closes #3644, PR #3982). * Adds video ingestion: a new `video_loader` (`cognee/infrastructure/loaders/core/video_loader.py`) transcribes a video's audio track through the existing `LLMGateway.create_transcript` path and feeds the text into the normal pipeline as a regular `TextDocument` — no new document type. Supported extensions are `mp4`, `m4v`, `mov`, `webm`, `mkv`, and `avi`; on OpenAI/Azure the transcript carries inline `[HH:MM:SS]` segment timestamps so timing survives chunking, while other providers fall back to a plain transcript. ffmpeg is optional: `.mp4`/`.webm` are sent straight to the transcription endpoint, other containers use a system ffmpeg (found via `shutil.which("ffmpeg")`) to extract the audio first and raise an actionable error when it is absent. The PR also fixes a real cross-provider bug: `create_transcript` forwards `**kwargs`, but the mistral/gemini/custom adapters rejected them with a `TypeError` inside the retry loop; the `**kwargs` contract is now uniform across `LLMInterface` and all adapters. No new configuration options, environment variables, or extras were added (SDK-141, related issue #3636, PR #3986). * Adds Turso (libSQL) as a selectable backend for all three cognee stores. Vector: `VECTOR_DB_PROVIDER=turso` routes to the new `TursoVectorAdapter` (collections as libSQL tables with `F32_BLOB` vector columns, cosine similarity via `vector_distance_cos`), working embedded (local file) or against Turso cloud via `VECTOR_DB_URL`/`VECTOR_DB_KEY`, and requires the `cognee[turso]` extra (`libsql-experimental`). Relational: `DB_PROVIDER=turso` selects a `TursoAdapter` that is a thin subclass of `SQLAlchemyAdapter` — a libSQL file is a SQLite file, so it reuses the same `aiosqlite` driver, sqlite dialect, and Alembic migrations; remote mode uses Turso's embedded-replica sync via `DB_TURSO_URL`/`DB_TURSO_AUTH_TOKEN`. Graph: `GRAPH_DATABASE_PROVIDER=turso` stores the knowledge graph as `graph_node`/`graph_edge` tables with recursive-CTE traversals, local/embedded only, needs no extra dependency, and optionally takes a `GRAPH_DATABASE_URL` file path. Vector and graph each ship a per-dataset dataset-database handler, so multi-user isolation under `ENABLE_BACKEND_ACCESS_CONTROL` works unchanged. All Turso backends are opt-in; defaults and existing backends are untouched (SDK-152, SDK-175 and SDK-176; PRs #4027, #4077 and #4080). * Adds deterministic, LLM-free code-graph extraction backed by the external `enola` binary (Apache-2.0, by Enola Labs). The new `cognee/tasks/code_graph` package runs `enola --generate` (discovered on `PATH` or via the `ENOLA_PATH` env var, with an `install_enola` helper pinned to a known version) and maps its `.enola/facts.jsonl` output — module/symbol/route/storage/dependency/service facts — to DataPoint models with deterministic `uuid5` ids plus typed edges (`calls`, `imports`, `implements`, `depends_on`, and others). User-facing entry points: `cognee.remember(repo_path_or_git_url, content_type="code")` indexes one or more local repos or git URLs (optional `index_vectors=True` enables embeddings; the default graph-only run needs no LLM key), a new `SearchType.CODE` queries the result through the new `CodeRetriever`, and `get_code_graph_tasks(repo_path)` plugs into `run_custom_pipeline`. Extraction covers Go, Python, TypeScript, Java, Kotlin, Swift, Ruby, C/C++, PHP, Vue, Svelte, OpenAPI, and gRPC. No enola code is vendored and no new Python dependencies are added; a missing binary raises an actionable `EnolaNotInstalledError` (COG-5837, PR #4037). * Adds Langfuse tracing support by wiring Langfuse into cognee's existing OpenTelemetry pipeline as just another OTLP destination — no separate Langfuse SDK. Setting `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` (optionally `LANGFUSE_HOST`, with `LANGFUSE_BASE_URL` accepted as an alias) makes `base_config.py` derive the OTLP endpoint (`{host}/api/public/otel/v1/traces`) and the `Authorization: Basic` header and turn tracing on; both keys must be set together or a `ValueError` is raised, and an explicit `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` always wins. Generation spans in `get_observe.py` now emit the vendor-neutral `gen_ai.request.model` and `gen_ai.system` attributes plus `langfuse.observation.type` and `SpanKind.CLIENT`, so Langfuse and any other OTLP backend render LLM calls as generations, and `tracing.py` forces the HTTP exporter for Langfuse endpoints (detected by the `/api/public/otel` path) since Langfuse does not support gRPC. Fully opt-in and off by default; requires the `cognee[tracing]` extra; no existing configuration or public API changed (SDK-167, PR #4055). * Adds an opt-in `litellm_native` structured-output framework that returns validated Pydantic objects without the `instructor` library, using LiteLLM's own `response_format`. Set `STRUCTURED_OUTPUT_FRAMEWORK="litellm_native"` and `LLMGateway` routes `acreate_structured_output` to a single universal `NativeLiteLLMAdapter` (`cognee/infrastructure/llm/structured_output_framework/litellm_native/`) covering every provider: when `litellm.supports_response_schema(model)` is true (OpenAI, Azure, Gemini, Mistral, Bedrock, …) the Pydantic model is passed straight through as `response_format` and validated with `model_validate_json`; otherwise (Ollama, llama.cpp, custom endpoints) it falls back to `response_format={"type": "json_object"}`, injects the JSON Schema into the prompt, and on validation failure feeds the error back for up to 3 self-correcting retries. Error handling matches the instructor adapters: auth and budget errors (`LLMPaymentRequiredError`) are terminal, rate limits retry with backoff, and content-policy violations fall back to the configured fallback model. The default remains `instructor` and the `baml` path is untouched; `create_transcript`/`transcribe_image` are unaffected (SDK-172, PR #4066). * Adds a one-command evaluation runner and makes the eval harness a clean optional addon. `run_eval(config) -> EvalResult` (`cognee/eval_framework/runner.py`) chains corpus building, answering, evaluation, and dashboard generation for a single deterministic config and returns artifact paths plus aggregate metrics; it is exposed as both `cognee eval …` and `python -m cognee.eval_framework …`, mapping the same flags onto `EvalConfig`. The evaluator registry now resolves engines lazily by import path, so importing the registry or running the DirectLLM engine no longer pulls in `deepeval`; selecting DeepEval without the dependency raises an actionable error pointing at `pip install "cognee[eval]"`, a new umbrella extra that installs the dashboard (`plotly`), the DeepEval engine, and dataset-download deps — `--engine direct_llm --no-dashboard` runs without it, and the runner preflights dashboard imports before any paid pipeline work. A `seed` is now actually threaded into the benchmark adapters, and artifacts are namespaced under `<output-dir>/<benchmark>_<engine>/` with the resolved config saved alongside (SDK-174, PR #4073). * Adds optional EXIF metadata extraction and perceptual-hash deduplication to `ImageLoader` (`cognee/infrastructure/loaders/core/image_loader.py`). With `IMAGE_EXIF_ENABLED=true`, `_extract_exif_metadata` pulls camera make/model, date taken, exposure, F-number, ISO, focal length, and GPS coordinates from the image's EXIF data and appends them as an `[EXIF Metadata]` block to the vision-LLM transcription. With `IMAGE_PERCEPTUAL_HASH_ENABLED=true`, a 64-bit difference hash (dHash) is computed per image and appended as a `[Perceptual Hash: …]` marker; an in-memory check against hashes seen in the same process flags visually similar re-ingestions with a duplicate note rather than skipping them. Both features are controlled by plain environment variables read at load time, are off by default, and use only PIL, which is already a cognee dependency — no new packages, and existing behavior is byte-identical unless a flag is enabled (partially addresses #3637, PR #4076). * Fixes BAML structured output failing on Pydantic models with PEP 604 optional fields. With `STRUCTURED_OUTPUT_FRAMEWORK=baml`, any response model containing an `X | None` field raised `ValueError: Unsupported type for BAML mapping: str | None` — in practice making `GRAPH_COMPLETION` recall unusable with BAML (common on local/Ollama setups), since the completion response model uses `str | None` fields. The cause: `map_type()` in `create_dynamic_baml_type.py` only matched `origin is Union` (`typing.Union`), while PEP 604 unions report `get_origin() == types.UnionType`, so they fell through to the unsupported-type error even though the equivalent `typing.Optional[str]` worked. The condition is now `if origin is Union or origin is types.UnionType`, handling both spellings on Python >= 3.10 (the project minimum). One-line fix plus a comment; no public API, configuration option, or environment variable changed, and non-BAML frameworks are unaffected (PR #4121). * Adds an optional `system_prompt` parameter to the MCP `recall` tool and forwards it end to end, so MCP clients can override the synthesis prompt for completion searches. Previously the tool signature had no such parameter, so custom recall system prompts were silently impossible over MCP. The parameter now flows through `CogneeClient.recall` (`cognee-mcp/src/cognee_client.py`) in both modes: in API mode it is included in the JSON payload POSTed to `/api/v1/recall`, and in direct/SDK mode it is passed as a kwarg to `cognee.recall`. When omitted, nothing is added to the payload or kwargs, so existing behavior is unchanged; regression tests cover both the API payload forwarding and the MCP tool forwarding. No other tool parameters, defaults, or server endpoints changed (fixes #4120, PR #4122). * Fixes `cognee remember --dry-run` and `cognee cognify --dry-run` silently executing real remote operations when `--api-url` is supplied. The API dispatch path bypassed the local command implementations that honor `dry_run` and simply ignored the flag, so a "dry run" performed an ordinary remote remember/cognify. `dispatch()` in `cognee/cli/api_dispatch.py` now checks `args.dry_run` up front and raises a `RuntimeError` ("--dry-run is not supported in --api-url mode. Run without --api-url to estimate locally without remote side effects.") before constructing the API client, so no health check or operation request is ever sent. Users who want a dry-run estimate must run without `--api-url`; ordinary API-mode behavior without the flag, local dry runs, and server behavior are unchanged, and no API endpoint, dependency, or configuration option changed (closes #4125, PR #4126). * Changes `visualize_graph()` and `GET /api/v1/visualize` to render a bounded, relevant subgraph by default instead of the whole graph. The renderer now selects a small set of seed nodes, expands their *k*-hop neighborhood, and caps the result at `max_nodes`, keeping renders fast and readable on large graphs. Seeds are resolved by priority — explicit `seed_node_ids` > a `recall()` or search result's graph provenance (`recall_result`, via `used_graph_element_ids`) > a `query` string's nearest (distance-ranked) vector hits > the graph's highest-degree nodes as a fallback — so a bare `visualize_graph()` call still shows a representative view, and "show me the subgraph behind this answer" and query-seeded views are deterministic and capped. New **keyword-only** parameters were added to `visualize_graph()`: `full`, `query`, `seed_node_ids`, `recall_result`, `neighborhood_depth` (default `2`), `neighborhood_seed_top_k` (default `10`), and `max_nodes` (default `500`); when a neighborhood exceeds `max_nodes`, nodes are kept by hop distance from the seeds and edges survive only when both endpoints do (no dangling edges). To restore the previous whole-graph render, pass `full=True` (or `?full=true` on the endpoint). `GET /api/v1/visualize` gains matching query params `full`, `query`, `seed_node_ids`, `neighborhood_depth`, `neighborhood_seed_top_k`, and `max_nodes` (`recall_result` is Python-only). The change is backward-compatible: the new parameters are keyword-only, so existing positional callers keep working; the underlying renderer and shared graph primitives are reused. See [Graph Visualization → Bounded subgraph by default](/guides/graph-visualization) (SDK-140, PR #3985). * Fixes the stored file size not refreshing when a file is re-ingested. On the re-ingest update path, `ingest_data` assigned the new size to `data_point.file_size`, but the `Data` model defines the column as `data_size` (the name already used correctly on the new-record branch). SQLAlchemy silently ignored the nonexistent attribute, so the persisted `data_size` stayed at its original value after a file was re-added with new or changed content. The assignment now targets `data_point.data_size`, so re-ingestion records the current size. No public API signature, configuration option, environment variable, or database migration changed (fixes #3160, PR #3578). * Fixes regex entity extraction config files failing to load on platforms whose default locale encoding is not UTF-8 (commonly Windows). `RegexEntityConfig._load_config` previously opened the config JSON with `open(path, "r")`, which relies on the platform's locale encoding; on a non-UTF-8 system a config containing non-ASCII characters (for example Unicode entity names, descriptions, or regex patterns) could raise a decode error and fail to load. The file is now opened with an explicit `encoding="utf-8"`, so configs load consistently across platforms. The change is backward-compatible: existing UTF-8/ASCII configs load exactly as before, no config edits or migration are required, and no public API signature, configuration option, or environment variable changed (fixes #3316, PR #3337). * Fixes a `TypeError` when instantiating the Amazon Neptune Analytics graph adapter (`NeptuneGraphDB`). `GraphDBInterface` declares `is_empty()` as an abstract method, but the Neptune adapter never implemented it, so the class was abstract and any attempt to construct it (including test collection in `cognee/tests/test_neptune_analytics_graph.py`) raised `TypeError: Can't instantiate abstract class NeptuneGraphDB with abstract method is_empty`. The adapter now implements `async is_empty() -> bool`, which runs a small openCypher node-existence query (`MATCH (n) RETURN true LIMIT 1`) and returns `True` when the graph contains no nodes and `False` otherwise; it relies on Neptune's openCypher support. This is a non-breaking bug fix that only adds the required method — no public SDK function, configuration option, or environment variable changed (closes #3407, PR #3457). * Fixes embedding retries wasting the full back-off window on deterministic "context window too small" failures. When an over-length embedding input is split down to a single string that still exceeds the model's context window but can no longer be divided, the engines now raise a new terminal `EmbeddingContextWindowTooSmallError` (a subclass of `EmbeddingException`, default message `Text is too short to split further but exceeds context window.`) and add it to their `retry_if_not_exception_type` set, so the failure returns immediately instead of consuming the \~128-second retry/back-off window. This applies to `LiteLLMEmbeddingEngine`, `FastembedEmbeddingEngine`, and `OpenAICompatibleEmbeddingEngine`; generic `EmbeddingException` failures remain retryable. The `OpenAICompatibleEmbeddingEngine` also now imports the shared `EmbeddingException`/`EmbeddingContextWindowTooSmallError` from `cognee.infrastructure.databases.exceptions` instead of defining a local `EmbeddingException`. No public API signature, configuration option, or environment variable changed; code that already catches `EmbeddingException` continues to catch the new subclass (fixes #3319, PR #3424). * Fixes `.txt` prompt templates being HTML-escaped on the wire. `render_prompt` configured its Jinja2 environment with `autoescape=select_autoescape(["html", "xml", "txt"])`, and because every prompt template shipped with Cognee is a `.txt` file, every interpolated variable in every rendered LLM prompt was HTML-escaped for all providers — apostrophes became `'`, triplet arrows in retrieval context became `-->`, ampersands became `&`, and angle brackets became `<`/`>`, including the user's own question in completion prompts. Autoescape now covers only markup templates (`["html", "xml"]`), so `.txt` prompts render their content verbatim while `.html`/`.xml` templates remain escaped. The user-visible effect is restored prompt fidelity, reduced token waste, and fewer subtle parsing/extraction issues; the change is internal to prompt rendering and non-breaking — no public API signature, configuration option, or environment variable changed (SDK-203, PR #4115). * Fixes DLT orphan cleanup leaving forgotten rows in the per-dataset graph and vector stores under `ENABLE_BACKEND_ACCESS_CONTROL`. When re-ingesting a DLT source (or a document source such as Notion/Slack/Google Drive) after rows were removed upstream, Cognee reconciles the corpus by deleting rows no longer present. Under access control the graph and vector engines are dataset-scoped, but the cleanup ran outside the dataset DB context — most visibly on the background-ingest path, where `orphan_cleanup` runs before any pipeline establishes that context — so `delete_data_nodes_and_edges` resolved the *default* engines and the graph + vector purge silently targeted the wrong database, leaving the forgotten row's chunks and entities in place and still retrievable (only the relational record was removed). The per-orphan deletion now runs inside `set_database_global_context_variables(dataset.id, dataset.owner_id)`, so the graph, vector, and relational stores are all purged for the correct dataset. Cleanup remains best-effort: partial failures are logged and retried on the next ingest rather than failing the add. No public API signature, configuration option, or environment variable changed (SDK-189, PR #4090). *** ## v1.4.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0)** Release that bumps the package version from `1.3.0` to `1.4.0` and refreshes `uv.lock`. The release cut itself introduces no functional code, public API, configuration, or environment-variable changes; the entries below are the accumulated work promoted from the development branch in this release (SDK-197). No migration or action is required for existing integrations. ### Highlights * Fixes `RAG_COMPLETION` and `TRIPLET_COMPLETION` searches ignoring the `node_name` filter. The public `search()` API already accepted `node_name` (and `node_name_filter_operator`) to restrict results to specific node sets, but for these two search types the argument was silently dropped: `CompletionRetriever` and `TripletRetriever` never received it, so their vector lookups (`DocumentChunk_text` / `Triplet_text`) searched the whole collection and returned chunks/triplets from outside the requested node set(s). Both retrievers now accept `node_name` and `node_name_filter_operator`, the search-type factory (`get_search_type_retriever_instance`) forwards them, and they are passed through to the vector search so results are scoped to the given node set(s) using the chosen `AND`/`OR` operator. The change is backward-compatible: `node_name` defaults to `None` (no filtering), so calls that never set it behave exactly as before, and no public API signature, configuration option, or environment variable changed (COG-5868, PR #4053). * Fixes two `cognee-mcp` bugs that surfaced on the first `remember` of a clean direct-mode (stdio) MCP session. First, the session-backed `remember()` flow now completes the session-to-graph bridge cleanly instead of tripping over dataset setup on the first write. Second, MCP startup migration output no longer pollutes the stdio JSON‑RPC channel, so clients do not misread migration chatter as protocol data. The `remember()` / `cognee.remember()` signatures are unchanged and no configuration option or environment variable changed; the only user-visible tradeoff is a brief one-time delay on the first `remember` while the session is initialized and bridged (SDK-192, PR #4091). * Fixes `improve()` runs failing to persist agent-trace feedback on multi-tenant (`ENABLE_BACKEND_ACCESS_CONTROL=true`) deployments. The agent-trace-feedback persistence path now forwards the authenticated `user` into `cognee.add()` and `cognee.cognify()`: `cognify_agent_trace_feedback` accepts a `user` parameter and passes it to both calls, and `persist_agent_trace_feedbacks_in_knowledge_graph_pipeline` supplies the pipeline's `user` to that enrichment task. Previously these `add`/`cognify` calls ran as the default user, which has no write ACL on multi-tenant deployments, so trace persistence raised a `403 PermissionDeniedError` and the `improve()` run showed errored memify-pipeline stages while feedbacks were silently skipped. The `user` parameter these internal pipelines and tasks already accepted is unchanged, and no public `improve()`/`memify()` API signature, configuration option, or environment variable was modified (COG-5893, PR #4097). *** ## v1.3.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.3.0)** Release that bumps the package version from `1.2.2` to `1.3.0` and regenerates the dependency lockfiles (`poetry.lock`, `uv.lock`, `cognee-mcp/uv.lock`). The release cut itself introduces no functional code changes; the entries below are the accumulated work promoted from the development branch in this release. Deployers upgrading should re-lock and reinstall to pick up the refreshed dependency graph. ### Highlights * Fixes document classification for text-like and unknown file extensions. `classify_documents` previously looked up the document class with `EXTENSION_TO_DOCUMENT_CLASS[data_item.extension]`, which raised `KeyError` during ingestion for extensions the map didn't cover — including common text formats (`md`, `json`, `xml`, `yaml`) and uppercase variants such as `.PDF` or `.CSV`. The extension is now normalized to lowercase before lookup, `md`/`json`/`xml`/`yaml` are mapped to `TextDocument`, and any unrecognized extension falls back to `TextDocument` instead of crashing the pipeline. Files that previously failed to ingest are now classified and processed. The change is backward-compatible: already-mapped extensions classify exactly as before, and no public API signature, configuration option, or environment variable changed (fixes #3657, PR #3662). * Fixes the CLI `cognify` command's `--ontology-file` flag, which previously had no effect. The command passed `ontology_file_path=` to `cognee.cognify()`, but `cognify()` accepts only a `config` object and silently swallowed the unsupported argument through `**kwargs`, so the ontology was never loaded. The command now translates `--ontology-file` into the canonical ontology `config` structure (an `rdflib` resolver with fuzzy matching, built with the same factory `cognify()` uses for its env-based fallback), validates up front that every referenced path exists and otherwise raises a clear `Ontology file not found: <paths>` error, and accepts multiple ontology files as a comma-separated list. Separately, a failed CLI command now always prints its error message: `cognee cognify` failures raise a `CliCommandException` whose `raiseable_exception` field is unset, and the entry point previously printed the message only when that field was set, so the command exited with code `1` but no explanation. Exit codes are unchanged (still `1` on failure), and no new flag, public API signature, or environment variable was introduced (PR #3997). * Improves concurrency and throughput of LanceDB subprocess mode (`VECTOR_DB_SUBPROCESS_ENABLED=true`) by replacing the session-wide RPC lock with id-based routing. Each async RPC now carries a per-request id and a main-process reader thread routes responses to per-call futures, so concurrent `call_async` operations run in parallel instead of serializing behind a single lock. A new `SUBPROCESS_WORKER_MAX_INFLIGHT` environment variable (default `16`) bounds how many async operations a worker runs at once; it must be `> 0` or worker initialization raises `ValueError` rather than silently degrading. Failure semantics also change: a per-call timeout or cancellation now resolves only that call and no longer tears down the entire subprocess session — the session ends only on genuine crash/shutdown/respawn events, which propagate a `SubprocessTransportError` to any still-pending calls. Synchronous calls (such as the Kuzu graph backend) continue to run serially and are unchanged. In internal Locust benchmarks, `/api/v1/add` average latency dropped from \~1371ms to \~246ms and p95 from \~9300ms to \~520ms, with overall throughput up \~21% (PR #2826). * Fixes propagation of the authenticated user into the memify session- and feedback-persistence pipelines, correcting multi-tenant attribution. The `persist_sessions_in_knowledge_graph_pipeline` and `persist_agent_trace_feedbacks_in_knowledge_graph_pipeline` functions now set the session user context (`set_session_user_context_variable(user)`) before running memify, so persisted sessions and agent-trace feedbacks are recorded against the intended authenticated user instead of a default/missing user. The `user` parameter these pipelines already accept is unchanged — no public API signature, parameter, or environment variable was modified, and no data migration is required. Logs and stored knowledge-graph entries may now show different (correct) user associations; custom memify pipeline hooks that relied on the previous missing-user behavior should be verified (PR #3950). * Adds optional per-stage LLM model routing so the extraction, summarization, and query stages can each run on a different model or provider. Each stage reads an optional `LLM_<STAGE>_*` environment group — `LLM_EXTRACTION_*`, `LLM_SUMMARIZATION_*`, and `LLM_QUERY_*`, each accepting `MODEL`, `PROVIDER`, `ENDPOINT`, `API_KEY`, and `API_VERSION` — whose set fields override the base `LLM_*` values for that stage while any unset field falls back to `LLM_*`. Because extraction runs once per chunk and dominates token spend, a common setup routes a cheaper or local model for extraction while keeping a stronger model for summarization and query-time reasoning (for example `LLM_EXTRACTION_MODEL="ollama_chat/llama3.1"`, `LLM_EXTRACTION_PROVIDER="ollama"`, `LLM_EXTRACTION_ENDPOINT="http://localhost:11434"`). Under the hood `LLMConfig.stage_config(stage)` returns a copy of the base config with any stage overrides applied, and a `pipeline_stage(stage)` context manager sets the existing `llm_config` ContextVar to that merged config for the duration of the stage; the client cache key is already derived from the context config, so each stage transparently gets its own cached client. This is fully backward-compatible and requires no action for single-model setups: the stage fields default to empty, so with no `LLM_<STAGE>_*` variables set the effective config is identical to today, and no extraction, summarization, retrieval, or SDK call signature changed. See the "Per-Stage Model Routing" section of [LLM Providers](/setup-configuration/llm-providers) for the full field reference and examples (PR #3961). * Fixes the missing exception type in error logs. When an exception is logged, `setup_logging()`'s structlog `exception_handler` processor records the exception class name in the `exception_type` field. That field guarded its assignment with `hasattr(exc_type, __name__)`, which used the module's own `__name__` (the string `"cognee.shared.logging_utils"`) rather than the literal `"__name__"`; since an exception class never has an attribute by that name the check was always false, so `exception_type` was never added to the log event. The guard now checks `hasattr(exc_type, "__name__")`, so logged exceptions record their type (e.g. `ValueError`) alongside the existing `exception_message`, identifying what failed rather than only that something failed. No public API, configuration option, or environment variable changed (PR #3998). * Fixes the `exception_type` field being silently omitted from logged exceptions. The custom `exception_handler` structlog processor in `cognee.shared.logging_utils.setup_logging` checked `hasattr(exc_type, __name__)`, where the unquoted `__name__` resolved to the module's name rather than the literal attribute name — so the check almost never passed and `event_dict["exception_type"]` was never set. The argument is now the string `"__name__"`, so log records for exceptions correctly capture the exception class name (`exception_type = exc_type.__name__`). This only affects the metadata attached to logged exceptions; no public API signature, configuration option, or environment variable changed (fixes #3709, PR #3849). * Restores a synchronous `get_vector_engine()` as a deprecated backward-compatibility shim and establishes `get_vector_engine_async()` as the canonical async accessor for the vector engine. Both are exported from `cognee.infrastructure.databases.vector`. `get_vector_engine()` is safe to call synchronously from any context — it does no async work when constructing the engine handle — but it now emits a `DeprecationWarning`, and the returned adapter's methods (`embed_text`, `search`, `get_connection`, ...) remain coroutines that must be awaited inside a running event loop. Released users who called `get_vector_engine()` without `await` are unaffected. Dev users who adopted the unreleased async form `await get_vector_engine()` should switch to `await get_vector_engine_async()`, which keeps a uniform "await the engine getter" contract alongside `await get_graph_engine()` (PR #3967). * Speeds up graph extraction for inputs with many chunks by removing a quadratic scan in `extract_graph_from_data`. DLT row chunks (whose graph is built deterministically from schema metadata rather than by the LLM) are excluded from the extraction path; previously each chunk was matched against the DLT set with a repeated list-membership check that triggered a full Pydantic `__eq__` comparison per pair, so the filter cost scaled with `len(data_chunks) × len(dlt_chunks)`. The function now partitions `data_chunks` into DLT and non-DLT lists in a single pass and returns `integrated + dlt_chunks`, making the extraction hot path linear in the number of chunks (a micro-benchmark reports roughly a 9,000× speedup at 4,000 chunks). Outputs are identical and the change is internal to `extract_graph_from_data` — no public API signature, configuration option, or environment variable changed (fixes #4015, PR #4017). * Preserves external ontology IRIs end-to-end and adds an RDF/SPARQL read surface plus RDF ingestion. `DataPoint` gains an optional `ontology_uri` field (defaults to `None`) that carries the external IRI a node is grounded in, threaded through `expand_with_nodes_and_edges` so persisted nodes keep their identifier instead of collapsing it to a local label. A new read surface (`cognee.modules.graph.rdf`) exposes the memory graph as RDF: `graph_data_to_rdf`, `export_memory_graph_to_rdf`, `serialize_memory_graph`, and `query_memory_graph_sparql`. Ungrounded nodes receive minted IRIs under `https://cognee.ai/graph/...` so the RDF is well-formed, `is_a` maps to `rdf:type` (individual→class) or `rdfs:subClassOf` (class→class), and other relations become predicate IRIs. RDF ingestion (`cognee.modules.ontology.rdf_xml.rdf_ingest` — `ingest_rdf`, `load_rdf_graph`, `build_datapoints_from_rdf`) parses TBox/ABox into `EntityType`/`Entity` datapoints that preserve verbatim IRIs, with identity derived from the IRI so re-ingesting the same RDF is idempotent. The change is backward-compatible: `ontology_uri` defaults to `None`, no DB migration is required, and the RDF surface rides on the existing `rdflib` ontology dependency (ensure `rdflib` and any parser backends are available in your runtime to use RDF export/ingest). See [Ontologies → RDF read/write surface](/core-concepts/further-concepts/ontologies#rdf-readwrite-surface) (PR #3928). *** ## v1.2.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.2)** Patch release that bumps the package version from `1.2.1` to `1.2.2` and refreshes `uv.lock`. This release introduces truth-subspace retrieval improvements, opt-in feedback weighting, and reliability fixes for S3-backed LanceDB setups. ### Highlights * Adds the truth subspace builder, which compiles accepted session learnings into centroids and slots that can be used to align and rerank retrieval results. * Adds opt-in truth-subspace reranking and learned feedback weighting for graph search. The default influence remains `0.0`; enable it with `DEFAULT_FEEDBACK_INFLUENCE` or per-call `feedback_influence` values. * Adds `build_truth_subspace` to the Improve API so truth-subspace indexes can be rebuilt as part of the enrichment flow. * Tracks the active dataset through request-local context so retrieval and background tasks can keep dataset-scoped truth state aligned. * Fixes LanceDB dataset provisioning for S3-backed system roots by avoiding direct local directory creation for S3 paths. * Adds demos and tests for truth-subspace building, reranking, feedback influence, and graph truth-state persistence. ### Other fixes * Removes the Sentry and Langfuse third-party observability integrations while keeping the OpenTelemetry tracing layer intact. The `Observer.LANGFUSE` enum value, the Langfuse branch in `get_observe()`, and the `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_HOST` configuration fields and environment variables are gone, and Sentry initialization is dropped from the API. The `@observe` decorator now maps only to OpenTelemetry — it emits an OTEL span when tracing is enabled (`COGNEE_TRACING_ENABLED=true`) and is a no-op otherwise — so existing `@observe` usage keeps working unchanged with no call-site edits. **Packaging (breaking):** the `monitoring` extra is removed in favor of the existing `tracing` extra, which installs only the OpenTelemetry API/SDK and OTLP exporters. Migrate with `pip install cognee[tracing]` in place of `pip install cognee[monitoring]`. Users who relied on Sentry or Langfuse should switch to an OTLP-compatible backend; configure it via `OTEL_EXPORTER_OTLP_ENDPOINT` and related `OTEL_*` variables (see [OpenTelemetry Tracing](/integrations/opentelemetry-tracing)). Lockfiles (`uv.lock` / `poetry.lock`) that still reference the removed `sentry-sdk` / `langfuse` packages should be regenerated. * Fixes a crash when running a pipeline in the background (`run_in_background=True`) with no explicit `datasets`. The background runner (`run_pipeline_as_background_process`) now reads the effective `user` from the run's `params` first and only falls back to the default user when none was supplied, then resolves the run across all datasets that user has write access to. Previously `user` was bound only on the fallback path, so the usual case — a user passed in `params` — left `user` unassigned and raised `UnboundLocalError: cannot access local variable 'user'` before the run started. No API or CLI changes are required. * Fixes a lock-starvation bug in single-session `improve()`. When `improve()` is called with one `session_id`, it holds a per-session lock so concurrent auto-improve, idle-watcher, and `SessionEnd` runs serialize instead of duplicating work. Previously the lock was released only after a successful run (and on early stage 1–2 failures), so an exception in a later stage — default enrichment (`memify`), the global context index, or the graph-to-session sync — left the lock held permanently, and every subsequent `improve()` for that session silently returned `{}` until the process restarted. All stages are now wrapped in a single `try/finally`, so the session lock is always released on exit regardless of which stage fails. No public signature, parameter, return type, configuration option, or environment variable changed. The fix prevents the issue from recurring; sessions already stuck from before the upgrade still need a process restart to clear the held lock (closes #3313, PR #3317). * Fixes a CLI startup crash on a fresh, uninitialized database (for example a new Postgres) when no `--user-id` is passed. When `resolve_cli_user()` resolves the default user, it now catches `DatabaseNotCreatedError`, runs the database migrations to create the schema, and retries — so the command proceeds with the default user instead of failing on first run. The recovery is automatic when resolving the default user, including omitted `--user-id` and non-strict fallback-to-default paths; normal calls against an already-initialized database are unaffected and incur no extra overhead. No new flag, configuration option, or environment variable is introduced (fixes #3267, PR #3308). * Allows one automatic retry on structured-output (instructor) calls in the generic LLM API adapter (`LLM_PROVIDER="custom"` and other generic OpenAI-compatible providers). The adapter's `acreate_structured_output` now passes `max_retries=2` to instructor on both the primary and the content-policy fallback request, where it previously allowed no retry. When the model returns output that fails instructor's schema parsing or validation, instructor reissues the request once before surfacing an `InstructorRetryException`. The user-visible effect is fewer transient structured-output failures, at the cost of a slight latency increase on the rare request that is retried. No public API signature, configuration option, or environment variable changed, and the existing retry-warning logs are unchanged (PR #3413). *** ## v1.2.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.1)** Patch release that bumps the package version from `1.2.0` to `1.2.1` and refreshes `uv.lock`. This release follows `v1.2.0` with targeted reliability fixes for dataset-scoped ingestion, background task lifetime, and dataset helper authorization. ### Highlights * Fixes `remember(..., dataset_id=...)` so it now forwards `dataset_id` to `add()`. Previously `dataset_id` was used only to build the `cognify()` target while `add()` silently ingested raw data into the default `main_dataset`, so `cognify()` ran on the intended (but empty) dataset and produced no new graph. Ingestion and graph building now target the same dataset. No API or migration changes are required; callers who passed `dataset_id` and saw missing results should upgrade. * Anchors fire-and-forget background tasks so Python's garbage collector can no longer abort them mid-run. Background syncs (`cognee.api.v1.sync.sync.sync`) and background pipeline runs (`run_pipeline_as_background_process`) now hold a strong reference to their in-flight `asyncio.Task` in a module-level set (`_BACKGROUND_SYNC_TASKS` / `_BACKGROUND_PIPELINE_TASKS`) until the task finishes, with a done-callback that discards the reference on completion. Previously the event loop kept only a weak reference, so the GC could collect a still-running task and silently abort a background sync or pipeline run. This fixes those intermittent silent aborts; the only side effect is a small, transient increase in retained memory while tasks run (released as each task completes). No public API signature, request/response schema, configuration option, or environment variable changed. * Fixes `cognee.datasets.has_data()` raising `AttributeError`. The method now forwards the full `User` object to its internal authorization helper instead of `user.id`, so calls succeed and return the expected `bool`. No signature, parameter, or behavioral change beyond the method no longer crashing. *** ## v1.2.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.0)** Release that promotes accumulated `dev` work after the `v1.2.0` development builds, bumping the package version to `1.2.0` and refreshing the lockfile (`uv.lock`). Highlights include ChromaDB search enhancements, BM25 lexical chunk search, search-answer reference evidence, session-context guidance enabled by default, and a range of Postgres/Neptune adapter, visualization, logging, and memory-stability fixes. ### Highlights * Adds ChromaDB vector search support for `include_payload=False`, so callers can omit metadata payloads from returned `ScoredResult` values when they only need ids and scores. * Adds ChromaDB `node_name` filtering for `search()` and `batch_search()`, including `OR` and `AND` semantics through `node_name_filter_operator`. * Prevents Entity and EntityType node id collisions by namespacing generated ids by node category. * Excludes internal `EntityType` taxonomy nodes and their `is_a` edges from the schema inventory output (`get_schema_inventory` and the visualize schema inventory endpoint). Consumers no longer receive a separate `EntityType` type group or `is_a` relationship aggregates; entity instances are still grouped under their resolved semantic type. * Improves ontology parsing for file-like inputs with filename/content-type detection, RDFLib fallback formats, and clearer initialization errors when parsing fails. * Reaps subprocess database workers deterministically at interpreter exit. The `cognee_db_workers` harness now registers an `atexit` handler that force-terminates any still-live LanceDB/Kuzu worker processes on shutdown, instead of relying on garbage-collector and `__del__` ordering that is not guaranteed to run at interpreter exit (notably for Windows `spawn` daemon workers). This helps avoid leftover worker processes and shutdown hangs when running with `graph_database_subprocess_enabled=true` or `vector_db_subprocess_enabled=true`. * Offloads the Ollama adapter's blocking client calls off the asyncio event loop. The `LLM_PROVIDER="ollama"` adapter wraps a synchronous OpenAI-compatible client, so its chat-completion, audio-transcription, and image/vision calls previously ran inline and blocked the running event loop for the full duration of each Ollama request, serializing concurrent async callers (for example the per-chunk extraction that `cognify()` fans out). These calls are now dispatched through `asyncio.to_thread`, so they execute in worker threads and no longer stall the loop. Public async signatures are unchanged and no configuration or migration changes are required; because requests now run in worker threads, any objects shared with the Ollama client should be thread-safe. * Serializes concurrent decodes on the shared llama.cpp local in-process model. The `LLM_PROVIDER="llama_cpp"` local (in-process) adapter now guards calls into its single `llama_cpp.Llama` instance with a lock, so the per-chunk extraction that `cognify()` fans out via `asyncio.gather`/`asyncio.to_thread` no longer decodes on the same non-thread-safe instance concurrently. This helps avoid native `GGML_ASSERT` crashes from corrupted KV-cache/logits state during local llama.cpp runs; in-process requests are now processed one at a time (use server mode for parallel decoding). * Simplifies the structured-output schema sent to the LLM during graph extraction when a custom `graph_model` (a `DataPoint` subclass) is used. `extract_content_graph` now converts the model to a plain `BaseModel` that keeps only the fields you declare on each subclass — DataPoint infrastructure fields (such as `id`, `created_at`, `version`, `type`, `belongs_to_set`) and the `metadata` field are dropped from the schema the LLM is asked to fill — and then rehydrates the LLM result back into your original `DataPoint` model via `model_validate`. The LLM extracts only your domain fields, while declared `metadata` defaults (for example `{"index_fields": ["name"]}`) are preserved on the rehydrated objects, so indexing behavior is unchanged. This is a no-action change for callers of the high-level extraction, `cognify`, and `remember` APIs. * Fixes Neptune (`GRAPH_DATABASE_PROVIDER="neptune"`) edge writes for relationship types that contain spaces, hyphens, or openCypher reserved words. The adapter now backtick-quotes (and escapes embedded backticks in) the relationship type when interpolating it into the generated openCypher `MERGE` statements for both single-edge and batched (`UNWIND`) edge upserts, preventing query syntax errors and unsafe interpolation. Also fixes the batched-edge fallback path so that when a batch insert fails, the per-edge retry iterates the edges for that relationship instead of the relationship grouping map. No configuration or migration changes are required, but generated/logged openCypher will now show backtick-quoted relationship-type names. * Reuses a single `aiohttp.ClientSession` across anonymous telemetry requests instead of opening a new session per call. This avoids a repeated DNS + TCP + TLS handshake to the telemetry endpoint on every event, helping lower latency and connection churn for telemetry. The shared session is created lazily inside the running event loop and rebuilt transparently when the loop changes (for example across tests or `asyncio.run` boundaries) or after it is closed; telemetry stays best-effort and never raises. No new configuration is required, and telemetry can still be turned off with `TELEMETRY_DISABLED=true`. * Tolerates a missing `dataset_database` table on PostgreSQL during startup migrations and pruning. The `run_startup_migrations()` vector step and the graph/vector prune routines now also catch the asyncpg `ProgrammingError` / `UndefinedTableError`, in addition to the SQLite `OperationalError` already handled. Running against a fresh PostgreSQL/pgvector database (for example the pgvector example) now skips the step with a warning instead of crashing with an undefined-table error. * Reduces peak memory use of the Postgres graph adapter (`GRAPH_DATABASE_PROVIDER="postgres"`) for graph node and edge relational upserts. `add_nodes`/`add_edges` now stream each batch to Postgres in fixed-size chunks (1000 rows per `INSERT ... ON CONFLICT` statement) instead of compiling one large multi-thousand-row statement, and JSONB property columns are serialized once at execute time via an engine-level `json_serializer` (the UUID/datetime-aware `JSONEncoder`) rather than a per-row `json.loads(json.dumps(...))` round-trip. This helps avoid the transient allocation churn and memory spikes seen on large single-batch writes (the commit reports roughly a 20x reduction). The number of rows written, the upsert/conflict semantics, and the data stored are unchanged; no configuration or migration changes are required. * Switches `CHUNKS_LEXICAL` search to BM25 ranking. Lexical chunk searches now rank exact-term matches with BM25 instead of the previous Jaccard-style scorer, and the retriever filters default stop words unless explicitly configured otherwise. API signatures stay the same, but result ordering can change for `SearchType.CHUNKS_LEXICAL`. * Adds lightweight references (Evidence) to completion-style search answers via a new `include_references` flag (default `true`) on `search()`, `recall()`, and the `POST /api/v1/search` and `POST /api/v1/recall` request bodies. When enabled, a deterministic `Evidence:` block is appended to the answer text, assembled in-process (no extra LLM call) from the retrieved chunk payloads, falling back to entity → chunk → document graph traversal when chunk metadata is missing. The response schema and return types are unchanged — Evidence is added to the answer text only. Because this changes default answer text, snapshot and evaluation baselines will diff; set `include_references=False` to restore the exact prior output. Older indexes lacking the new `document_id`/`document_name` chunk fields use the graph fallback where available or omit the Evidence block silently. * Disables local-variable rendering in logged exception tracebacks. `setup_logging()` now configures the console renderer with `RichTracebackFormatter(show_locals=False)`, so when an exception is logged the traceback no longer expands each frame's local variables. In the retrieval/search path those locals can hold graph objects carrying embedding vectors and deep node/edge references, and rendering them recursively spiked memory to multiple GB and OOM-killed the process (notably in CI) whenever an exception was logged mid-search. Tracebacks themselves are still logged; only the per-frame locals dump is omitted. No configuration changes are required. * Restores Cognee's safe uncaught-exception hook. `setup_logging()` now installs `sys.excepthook` so non-`KeyboardInterrupt` exceptions are logged through structlog before Python's default traceback is printed, and falls back to plain traceback output if rich rendering itself fails. No configuration changes are required. * Propagates relational `DATABASE_CONNECT_ARGS` SSL settings to the Postgres maintenance, PGVector, and graph Postgres engines, so connections to managed Postgres that enforce SSL (for example Neon, RDS/Aurora, Azure Database for PostgreSQL) succeed. Previously only the main relational engine received these args, so CREATE/DROP DATABASE maintenance, per-dataset PGVector engines, and the `GRAPH_DATABASE_PROVIDER="postgres"` graph engine could fail with missing-SSL errors. The maintenance engine also maps the libpq `sslmode` key to the asyncpg `ssl` key, and rewrites a Neon `-pooler.` host to its direct endpoint because CREATE/DROP DATABASE cannot run through Neon's PgBouncer pooler. No configuration schema change is required — supply asyncpg SSL options via the existing `DATABASE_CONNECT_ARGS` and they are now honored across Cognee's Postgres engines; the env unset stays a no-op for in-cluster Postgres. Deployments on managed Postgres with enforced SSL should upgrade. * Fixes two interaction glitches in the graph visualization (`visualize_graph`) story view. Clicking a node no longer displaces it: the click-vs-drag threshold is raised from 3px to 6px so trackpad jitter on a plain click is no longer treated as a drag that reheated the force simulation and sent the clicked node (and, in the **Force** and **Flow** layouts, the whole layout) flying off the canvas. In the **Story** layout, dragging now drives the node position directly instead of reheating the pinned grid, and a released node snaps back cleanly to its lane. Separately, the pipeline stage-header pills (shown in the **Story** and **Flow** layouts) are now drawn in a final pass after edges, nodes, and labels, so a dense graph panned toward the top of the viewport can no longer paint over them. Generated visualization HTML changes only; no API, configuration, or migration changes are required. * Retries the Kuzu/Ladybug JSON extension load on the live connection when it is missing at runtime. In the subprocess graph worker (`graph_database_subprocess_enabled=true`), if `LOAD EXTENSION` fails with a "not been installed" error, the worker now runs `INSTALL` on the active connection and retries the load once; if that `INSTALL` fails it raises with the real underlying cause instead of the generic load error. This recovers from cases where the best-effort warm-up install on the throwaway database did not complete (for example a transient network error while downloading the extension on a fresh machine). The warm-up install path also now logs its failure cause to stderr (`[ladybug worker] warm-up INSTALL JSON failed: ...`) instead of swallowing it silently, so these conditions are diagnosable from worker/CI logs. The retry may perform an extension install on the live connection and add a small startup delay; no configuration or migration changes are required. * Relaxes the bundled Ladybug graph-store dependency from the `ladybug==0.16.0` pin to `ladybug>=0.16.0,<0.18`, so installs can pick up the `0.17.x` line. The database migration worker's storage-version table now maps the `0.17` on-disk format (catalog code `41`) to `0.17.1`, so an existing `0.16.x`/`0.17.x` graph database is recognized as current and is not flagged for legacy migration (migration still targets only pre-`0.15.0` databases). No manual database schema changes are required; deployers upgrading should re-lock dependencies (refresh `uv.lock`) and redeploy database workers to pick up the new range. * Adds a session-context guidance layer and turns it on by default. The cache `AUTO_FEEDBACK` setting now defaults to `true` (previously `false`), so when `CACHING` is enabled, session-capable completion searches run one additional structured-output LLM call per answered turn under the resolved session (`default_session` when `session_id` is omitted) to analyze the current turn against the previous one. The analysis can rewrite the turn into an effective query used for retrieval, accumulate durable per-session guidance grouped into `goals`, `rules`, `preferences`, and `lessons_learned` that can be injected into later answers, and **gate** a follow-up turn — returning a short acknowledgement (the analysis reply, or `"Got it."`) instead of running retrieval and completion. The step fails open to answering the original query when analysis errors or no session is available. Because guidance and the effective query can change retrieval inputs, session answers and turn gating may differ from history-only sessions, and per-turn latency and token usage increase. Set `AUTO_FEEDBACK=false` to disable and restore plain conversation-history replay. See [Sessions and Caching](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback). * Bypasses the Instructor structured-output pipeline when `acreate_structured_output` is called with `response_model=str` on the default OpenAI, generic, and Ollama LLM adapters. Plain-text requests are now sent directly to the provider and the model's raw string content is returned, instead of being wrapped in Instructor's JSON/tool-call schema. This avoids repeated parse failures and retry storms on local llama.cpp-compatible servers that don't honor those schemas, and can lower latency for string completions. Rate limiting still applies to these direct calls. Passing a Pydantic model is unchanged — it still returns a validated model instance — so this is a no-action change for callers. * Fixes the condition that gates name-to-UUID resolution of the `datasets` argument in `search()`. The check was wrapped in a single-element list (`[all(...)]`), which is always truthy, so the name-resolution path ran for any non-`None` `datasets` value. It now runs only when every entry in `datasets` is a string. Passing dataset names (the documented usage) is unaffected; the only behavior change is that non-string entries supplied through `datasets` (for example already-resolved UUIDs) are no longer forced through name-based authorization lookup — pass UUIDs via `dataset_ids` as before. No API signature, default, or migration change is required. * Corrects two `.env.template` knob names that the config loader was ignoring. The template previously listed `LLM_MAX_TOKENS` and `EMBEDDING_MAX_TOKENS`, but Cognee's settings classes read these values from `LLM_MAX_COMPLETION_TOKENS` (default `16384`) and `EMBEDDING_MAX_COMPLETION_TOKENS` (default `8191`). Anyone who copied the old template and set the chunk-sizing limits under the previous names had them silently ignored, so chunk sizing fell back to the defaults. If you relied on those entries, rename them to `LLM_MAX_COMPLETION_TOKENS` / `EMBEDDING_MAX_COMPLETION_TOKENS` in your `.env`. No code or schema changes are required. The same `.env.template` update also documents additional already-supported settings (LLM/embedding tuning and rate limiting, chunking, session cache, graph/vector connection and subprocess tuning, Langfuse monitoring, llama.cpp, and the auth-token secrets) as commented examples with their defaults. * Fixes an `UnboundLocalError` in file metadata extraction (`get_file_metadata`) when the underlying file-like object cannot seek. `content_hash` is now initialized before the seek/hash attempt, so when `file.seek(0)` raises `io.UnsupportedOperation` (the error is still logged), metadata is returned with an empty `content_hash` instead of crashing. This makes ingestion more robust for non-seekable file-like inputs; no configuration or API changes are required. * Preserves structured search completions in the result payload. `SearchResultPayload.completion` now models a single `dict`, a Pydantic `BaseModel`, and a list of models in addition to the previous `str` / list-of-string / list-of-dict shapes, and a custom serializer dumps model instances to their `dict` representation. This fixes searches that pass a non-string `response_model` (typed LLM output via `retriever_specific_config`): the structured object is kept as-is instead of being dropped or coerced into an empty model. The default string-answer path is unchanged; no configuration or migration changes are required. * Caps LLM retries and recovers from over-length embedding input. The structured-output and transcription adapters (anthropic, azure\_openai, gemini, generic\_llm\_api, llama\_cpp, mistral, ollama, openai) now stop on a fixed number of attempts (`stop_after_attempt`) instead of the previous time-based `stop_after_delay(128)` window, and the instructor retry counts were lowered (for example structured-output generation tops out at 3–4 attempts, while Bedrock's and the other adapters' inner instructor `max_retries` drop to 1–2). This makes transient failures fail faster and at lower cost, with a small reduction in resilience to intermittent errors — watch your LLM error/latency/cost metrics after upgrading. Separately, `LiteLLMEmbeddingEngine` now recovers from over-length embedding input: a context-window error or a `400 BadRequestError` matching `maximum input length` triggers recursive split-and-pool (splitting the batch, or splitting a single string into overlapping halves and averaging the resulting vectors) instead of failing, while other 400 errors still fail fast. No API or configuration changes are required. See [Embedding Providers → Timeout and Retry Behavior](/setup-configuration/embedding-providers). * Bounds the input-data preview persisted in the `pipeline_runs.run_info` column so a single run cannot grow the table without limit. On pipeline run start, error, and completion, the audit-only `run_info` data is summarized: a list of `Data` records is still reduced to their IDs and empty input is still recorded as `"None"`, but any other payload is now stringified and truncated to a 512-character preview ending with `... [truncated, <N> chars total]` instead of being stored verbatim. `run_info` is never read back during processing; persist large raw inputs (for example text passed to `add()`/`cognify()`) elsewhere if you need the full payload. No configuration or migration changes are required. * Fixes `forget(everything=True)` under multi-tenant per-dataset database isolation (`ENABLE_BACKEND_ACCESS_CONTROL=true`). The `everything` branch no longer runs inside a single-dataset database context; the per-dataset context is now established per dataset inside the underlying delete-all flow. Previously, entering a single-dataset context with no dataset reference could try to create a `dataset_database` row for a non-existent dataset and fail the operation. Single-dataset, single-item, and `memory_only` modes are unchanged, and the public `forget()` signature, return shapes, and error messages are unchanged. * Forwards `FALLBACK_ENDPOINT` to the OpenAI adapter's content-policy fallback request (`LLM_PROVIDER="openai"`). Previously this `api_base` override was not applied, so the fallback completion always went to the default OpenAI endpoint even when `FALLBACK_ENDPOINT` was set; now the fallback request is routed to the configured base URL. Deployments that set `FALLBACK_ENDPOINT` to an OpenAI-compatible proxy or alternate endpoint will see their fallback traffic go there. `FALLBACK_ENDPOINT` remains optional for `openai` — when unset, the fallback still uses the default OpenAI endpoint. * Caps the `instructor` dependency at `<1.15.3` (previously `<2.0.0`) and lowers the `litellm` minimum to `>=1.83.7` (previously `>=1.84.0`). This pins structured-output extraction to a known-good `instructor` range and widens the compatible `litellm` window; lockfiles (`poetry.lock`, `uv.lock`) are refreshed to match. No API or behavioral changes — callers using the high-level `cognify`/`search` APIs are unaffected. Developers and deployers should re-lock and reinstall dependencies to pick up the new constraints. * Detects Markdown, JSON, XML, and YAML files by extension during file-type guessing. `guess_file_type` now returns deterministic types for `.md`/`.markdown` (`text/markdown`), `.json` (`application/json`), `.xml` (`application/xml`), and `.yaml`/`.yml` (`application/yaml`) instead of relying on content-based detection, which has no magic-number signature for these formats and fell back to `text/plain`/`txt`. The recorded file metadata (`mime_type` and `extension`) for these files now reflects their actual format. Loader selection is unchanged — `TextLoader` already handled these extensions — so no action or migration is required. * Adds a `GET /api/v1/proposals/{proposal_id}` endpoint for reviewing a stored skill-improvement proposal before applying it. The endpoint takes a required `dataset_id` query parameter and returns the proposal's `status` (`proposed`/`applied`), `confidence`, `rationale`, `model_name`, and before/after procedures (`old_procedure`/`proposed_procedure`); it is read-only and never mutates the graph (applying still goes through `POST /api/v1/remember/entry` with `skill_improvement`). It returns `403` when the caller is not authorized for the dataset and `404` when the proposal is not found. * Adds no-code, inline skill ingestion. `POST /api/v1/remember` (with `content_type=skills`) now accepts `skills_text` (a `SKILL.md` markdown body as a string) and `skill_name` (the skill name/slug, defaults to `skill`) form fields, so a skill can be ingested without uploading a file — when `skills_text` is set and no files are uploaded, it is written to a temporary `SKILL.md` and ingested through the existing skills pipeline. A new `POST /api/v1/skills` endpoint exposes the same inline ingestion via a JSON body (`skills_text`, optional `skill_name`, and one of `dataset_name`/`dataset_id`). ### Notes * Includes a behavior-preserving cleanup of the LiteLLM embedding engine (`LiteLLMEmbeddingEngine`): no public `__init__` signature, env-var (`MOCK_EMBEDDING`, `EMBEDDING_ENDPOINT`), or default changes, and embedding behavior is unchanged. * Deployers upgrading should re-lock dependencies (refresh `uv.lock`) and reinstall, then rebuild/redeploy to pick up the updated dependency set. * The ontology parser update improves file-like parsing behavior; upload endpoint format restrictions should be documented separately if they change. *** ## v1.1.3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.3)** Patch release focused on API-mode robustness and dependency safety. It enables remote pipeline status checks for MCP/API deployments, improves vector retrieval behavior for empty input, and tightens the `instructor` dependency range. ### Highlights * Enables `cognify_status` in API mode. The MCP can resolve dataset IDs remotely and read pipeline status from `GET /api/v1/datasets/status`, so self-hosted API deployments can check background pipeline status without local database access. * Adds API-mode support to `CogneeClient.get_pipeline_status`, which now queries the server's `/api/v1/datasets/status` endpoint instead of raising `NotImplementedError`. * Makes LanceDB retrieval return an empty list when called with an empty id list, preventing avoidable errors for callers that sometimes have no vector ids to fetch. * Pins `instructor` below `1.15.3` and refreshes lock metadata. Deployers with exact dependency pins should re-lock or reinstall against the updated constraints. * Refreshes the README with clearer Cognee positioning, branding, and a research paper link. *** ## v1.1.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.2)** Patch release with a refreshed public frontend, improved Cloud UI workflows, and a Postgres graph adapter compatibility fix for asyncpg/PostgreSQL 16. ### Highlights * Syncs the public frontend with the SaaS application, bringing updated dashboard, search, dataset, connection, onboarding, knowledge graph, and graph model editor experiences. * Adds conversation-based search history and refreshed multi-dataset search flows in the frontend. * Improves connection and onboarding flows with a connection modal, step-by-step agent setup guidance, new quickstart assets, and updated loading visuals. * Adds memory customization UI support for datasets, including graph models, custom prompts, and ontology-related configuration. * Fixes Postgres graph neighborhood expansion under asyncpg/PostgreSQL 16 by casting recursive CTE seed parameters to `text[]`. ### Notable Changes * Bumps the package version from `1.1.1` to `1.1.2` and refreshes lockfiles. * Aligns frontend API routes and local development behavior with the OSS backend. * Updates API key, tenant, configuration, dataset, ingestion, ontology, search-history, session, analytics, and user frontend modules. * Adds frontend assets for quickstarts, agent integrations, loading states, and graph previews. * Adds regression coverage for Postgres graph neighborhood seed array typing and retries a flaky usage-logger e2e path in CI. ### Fixes and Improvements * **Postgres neighborhood query parameter typing**: The Postgres graph adapter's `get_neighborhood` query now casts the seed parameter to `text[]` (`unnest(CAST(:seeds AS text[]))`) in its recursive CTE seed row. Deployments using `GRAPH_DATABASE_PROVIDER=postgres` with asyncpg/PostgreSQL 16 should no longer hit parameter type inference errors when expanding neighbors from seed node ids. * **Cloud UI refresh**: Dashboard, dataset, dataset detail, connections, search, onboarding, knowledge graph, and graph model editor screens were refreshed and aligned with current Cloud workflows. * **Search and dataset workflows**: Search now supports conversation history and multi-dataset recall flows, while dataset pages add improved status polling, graph access, and memory customization entry points. * **Connect Agent flow**: The frontend adds clearer connection setup prompts, modal-based setup guidance, and integration visual assets. * **Frontend resilience**: Error handling, loading states, analytics logging, tenant context, user configuration, and local fetch behavior were updated across the public frontend. *** ## v1.1.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.1)** Patch release that promotes accumulated `dev` work after `v1.1.1.dev0`, with agent-management APIs, graph visualization updates, custom graph-model support in `remember`, and backend stability fixes. ### Highlights * Adds agent management and connection endpoints for listing, creating, inspecting, registering, unregistering, and deleting agents and their active connections. * Reworks graph visualization with a pipeline-aware Story layout, Schema view, improved labels, legends, and modular visualization components. * Adds `graph_model` support to the `remember` REST endpoint, letting API callers pass a JSON-serialized graph schema into ingestion. * Expands graph and retrieval behavior with local Neo4j dataset handling, global context graph bucketing, improved edge text, and `node_name` filtering for chunk retrieval. * Improves LLM, PGVector, remember/session, prune, forget, and graph-projection error handling. ### Notable Changes * Bumps the package version from `1.1.0` to `1.1.1` and refreshes the release lockfiles. * Splits agent lifecycle and connection handling into dedicated modules and API routes, including persisted agent connection state and agent-session names. * Adds SDK/API support for retrieving specific agent configuration and for inspecting current agent connections. * Adds local Neo4j dataset database handling and updates graph database selection to recognize that handler. * Reworks global context index internals with graph bucketing, scoring, build, update, load, summarize, and persistence flows. * Improves edge indexing and rendering by preserving natural edge descriptions, generating fallback edge text from metadata, and rendering relationship labels inside edge markup. * Updates CI and test coverage across database adapters, agents, visualization, global context indexing, retrieval filters, and LLM configuration. ### Fixes and Improvements * **Remember custom graph models**: The `remember` REST endpoint now accepts an optional `graph_model` form field, parses the JSON schema into a graph model, and forwards it into the ingestion flow. * **Agent lifecycle and connections**: Agent endpoints now separate agent resources from agent connections, support agent-session names, persist connection metadata, mark unregistering agents inactive, and expose connection detail. * **Graph visualization**: Story view spacing, column pinning, schema rendering, edge-label rendering, and fallback labeling were improved so generated graph views are easier to inspect. * **Graph ingestion and retrieval**: Edges with unprojectable endpoints are skipped instead of failing graph projection, `KnowledgeGraph` subclasses follow the knowledge-graph integration path, chunk retrieval receives `node_name` filters, and `forget` can handle dataset values that are string UUIDs. * **PGVector metadata consistency**: `create_collection` now reflects SQLAlchemy metadata only after the table-creation transaction commits, avoiding stale metadata entries when table creation rolls back. * **LLM adapters**: Generic LLM API transcription and Ollama image transcription now raise clear `ValueError` messages for empty responses, Mistral guards against `None` messages before reading content, and OpenAI instructor mode is honored. * **Session remember routing**: `remember(session_id=...)` now routes through the JSON `/entry` endpoint in API mode, and using `custom_prompt` with `session_id` raises a clear `ValueError`. * **Operational stability**: Prune errors and dataset lookup issues are handled more defensively, brittle batch-query test settings were adjusted, and optional LLM configuration can be passed through CI. *** ## v1.1.0.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.0.dev1)** Developer preview release on the way to `v1.1.0dev1`. This release includes API, retrieval, permissions, storage-runtime, and backend consistency changes. ### Highlights * Adds database subprocess workers for LanceDB and Kuzu so native database work can run outside the main Cognee process. The wheel now includes the `cognee_db_workers` package. * Exposes more ingestion controls through the public API and remote client paths, including chunk sizing and background execution options for `remember()` and `cognify()`. * Adds `dataset_ids` support to `recall()`, making shared-dataset retrieval more reliable when dataset names are not owned by the calling user. * Expands permission management with DELETE endpoints for dataset permissions, roles, and user-role membership. * Improves session visibility so parent users can see sessions created by child-agent users where appropriate. ### Notable Changes * Adds `graph_database_subprocess_enabled` and `vector_db_subprocess_enabled` configuration, plus Kuzu tuning variables for threads, buffer pool size, and max DB size. * Keeps `belongs_to_set` metadata consistent across dataset deletion and shared-node/vector upserts in LanceDB, PGVector, and Neo4j paths. * Adds `include_payload` behavior to Neptune Analytics vector search. * Improves Postgres hybrid batching by respecting embedding-engine batch size. * Improves infer-schema text sampling and prompting. * Rewrites the examples README into a fuller index and adds performance-testing support with Locust. * Deprecates `.env.example` as the canonical template in favor of `.env.template`. * Bumps the package version from `1.0.9` to `1.1.0.dev1` and refreshes lockfiles. *** ## v1.0.3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.3)** Patch release with bug fixes and stability improvements on top of v1.0.2. ### Highlights * Promotes accumulated `dev` work to `main` for the `v1.0.3` release * Adds session lifecycle APIs, unified memory/session handling, and dashboard support * Introduces dataset queueing for async context management and ingestion flows * Ships new relational migrations, including session lifecycle tables and `parent_user_id` * Expands recall/remember and cloud routing behavior, plus frontend onboarding and Connect Agent updates ### Notable Changes * Added session endpoints, metrics, and supporting persistence work * Added dataset queue infrastructure and follow-up fixes for background processing * Added database migrations for new tables and user/dataset ownership handling * Updated recall, remember, improve, and search-related API behavior * Added frontend work for Connect Agent, dashboard/activity views, API keys, and onboarding * Included guide updates, workflow/tooling changes, and dependency updates such as `litellm` and `onnxruntime` ### Bug Fixes * **PostgreSQL null-byte compatibility**: Embedded null bytes (`\x00`) in node or edge string fields no longer cause errors when using PostgreSQL as the relational backend. Null bytes are now automatically stripped from all string values (including nested attributes) before writes to the relational store. This sanitization is transparent — affected strings are silently cleaned rather than rejected. * Fixed duplicate graph nodes caused by `DataPoint.id` being regenerated during graph construction. The original `id` is now preserved when converting DataPoint instances into graph nodes, ensuring node identity is stable across graph extraction passes. *** ## v1.0.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.2)** Patch release with bug fixes and stability improvements on top of v1.0.1. ### Bug Fixes * **LanceDB schema migration**: "contained null values" errors (raised when old rows lack a field required by a newer DataPoint schema) are now treated as recoverable schema drift. The affected table is automatically rebuilt from the current schema instead of raising a hard failure. * **cognee-mcp Docker image build**: Added missing `build-essential` and `libpq-dev` system packages to the builder stage so that `cognee[postgres]` can compile `psycopg2` from source on Linux. ### Dependency Updates * Bumped `llama-index-core` requirement from `>=0.13.0,<0.14` to `>=0.14.20,<0.15` for the `llama-index` extra. * Pinned `nltk>=3.9.3,<4` explicitly in the `docs` extra to satisfy `unstructured`'s dependency until `unstructured` v0.21.0. *** ## v1.0.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.1)** Patch release with bug fixes on top of v1.0.0. *** ## v1.0.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.0)** ### Highlights * **New high-level API**: `remember`, `recall`, `improve`, and `forget` cover the full memory lifecycle in four operations * Session-aware memory via `session_id` — short-term context that can be promoted into the permanent graph * Unified `recall` replaces the previous `search` call with automatic retrieval strategy selection * Legacy operations (`add`, `cognify`, `search`, `memify`) remain available as lower-level building blocks ### New Features * `cognee.remember(data, session_id=...)` — ingest and graph in one call; supports permanent or session memory * `cognee.recall(query, session_id=...)` — query across both the permanent graph and session cache * `cognee.improve(...)` — enrich an existing graph with feedback-based weighting and session promotion * `cognee.forget(dataset=..., session_id=...)` — delete data, datasets, or full session memory *** ## v0.5.4.dev1 **Released:** March 5, 2026\ **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.4.dev1)** ### Highlights * Developer preview release focused on quality, performance, and developer ergonomics * Faster ingestion and sync * Improved search relevance and new filtering options * Stability fixes for memory creation, deletion, and CLI workflows * Internal refactoring and dependency upgrades ### New Features * Bulk import CLI for faster batched ingestion * Search filters for tags and date ranges * Optional per-collection ingestion throttling ### Improvements * Lower latency for ingestion and sync * Better search ranking * More robust deletion and duplicate handling * Clearer CLI messages and debug logs ### Bug Fixes * Fixed duplicate memories under concurrent ingestion * Fixed partial state after deletion * Fixed CLI export formatting issues * Fixed intermittent retrieval failures under load *** ## v0.5.3 **Released:** February 27, 2026\ **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.3)** ### Highlights * New graph visualization improvements * Expanded permissions and user management work * SessionManager and cache/session persistence work * Search and graph retrieval improvements * Multiple stability and CI/CD fixes ### Notable Changes * Added role-based permission checks and permission endpoints * Added graph visualization updates, including note set coloring * Added return type hints to API functions * Added chunk associations for the memify pipeline * Added vector filtering based on node sets * Fixed delete flow bugs, health check issues, MCP issues, and several config/integration issues *** ## v0.5.3.dev1 **Released:** February 20, 2026\ **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.3.dev1)** ### Highlights * Added vector filtering based on node sets * Added principal Cognee configuration * Fixed health check issues * Fixed FalkorDB adapter port bug * Fixed Ollama image ingestion argument issue ### Notes * Includes a small set of targeted fixes and feature work on top of `v0.5.3.dev0` * Introduced one new contributor in this release

> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt > Use this file to discover all available pages before exploring further. # Changelog > Recent Cognee releases Cognee releases with highlights and links to the full release notes on GitHub. ## Unreleased Changes queued for the next release. This section is updated as unreleased work is merged and is folded into a versioned release section when the release is published. ### Highlights * Fixes a cancelled pipeline run being left in progress forever instead of reaching a terminal status. `run_tasks()` wraps a run's task execution in a `try`/`except` whose handler runs the rollback, writes the terminal `pipeline_runs` row through `log_pipeline_run_error`, and yields `PipelineRunErrored` — but it caught only `Exception`, and `asyncio.CancelledError` has been a `BaseException` since Python 3.8. A run interrupted by cancellation (a graceful server shutdown or restart, or any other `CancelledError` delivered to the task driving the run) therefore skipped that entire handler, and its `pipeline_runs` record stayed at `DATASET_PROCESSING_STARTED` indefinitely — which for a custom pipeline running with `use_pipeline_cache=True` means the dataset-level cache check keeps reporting the dataset as "already being processed" and skipping re-runs, since that check short-circuits on `DATASET_PROCESSING_STARTED` while letting `DATASET_PROCESSING_ERRORED` through. The handler now catches `asyncio.CancelledError` alongside `Exception`, so a cancelled run is finalized exactly like any other failed run — rollback handler first, then a terminal row with `outcome` `FAILED`, `error_class` `CancelledError` and a scrubbed `error_message`, then a `PipelineRunErrored` event — and **cancellation still propagates**: the error is re-raised at the end of the handler, so this inserts one cleanup step rather than swallowing or delaying the cancel. Behavior on the `Exception` path is unchanged. Note this covers *cooperative* cancellation only; a hard kill (`SIGKILL`, container OOM, power loss) runs no cleanup code at all and still leaves a `DATASET_PROCESSING_STARTED` row for the [startup stale-run reaper](/core-concepts/building-blocks/pipelines) to reclaim. No public API signature, configuration option, environment variable, or migration ships with the fix (CLO-365, PR #4680). * Adds a **[GitHub App organization connector](/integrations/github-integration)**: an admin installs a configured GitHub App into an org, and every repository the installation covers is cloned and indexed into the deterministic [code graph](/guides/code-graph) — one dataset per installation, named `github_` (the account login lowercased, non-alphanumeric runs collapsed to `_`), so backend access control isolates at the org boundary rather than per repository. The indexed content is reachable through `SearchType.CODE` only; the code route produces no chunks and no embeddings, so completion and chunk search types do not cover it. The connector reuses the existing generic `authorize`/`callback`/`connection`/`status` routes and the encrypted credential store unchanged — **no new provider-specific endpoints**. **Nothing durable is stored but the installation id**: the credential's encrypted token payload is empty, and \~1-hour installation tokens are minted on demand from the app's private key via a hand-rolled RS256 JWT (using the existing `cryptography` dependency — no new packages). The callback's `installation_id` arrives on an unauthenticated endpoint and is never trusted directly: the OAuth `code` is exchanged for a user token, that user's access to the installation is confirmed against `GET /user/installations`, and the credential is built from the app-JWT-authenticated installation record — which is why the app must have **"Request user authorization (OAuth) during installation" enabled**. An initial sync fires detached after connect (so the browser redirect is immediate), and webhooks keep it fresh: `push` to a repository's **default branch** re-indexes it, `installation_repositories` indexes additions while **removals are logged only** (indexed data is retained — `forget()` stays a human decision), and `installation` `deleted`/`suspend` revokes the credential. All handling is idempotent, so redeliveries are cheap and safe. Disconnect (`DELETE /api/v1/integrations/{provider}/connection`) is **non-destructive**: it revokes the local credential so no further tokens are minted, but deletes no indexed data and does not uninstall the app — `revoke_remote` stays a no-op for GitHub because the GitHub-side equivalent would remove the installation from the whole org. **New configuration** (all optional; a deployment without them boots normally and fails loudly at use time, surfacing as a 503 from `/authorize` rather than a 500): `GITHUB_APP_ID`, `GITHUB_APP_SLUG`, `GITHUB_APP_PRIVATE_KEY` (accepts literal `\n` escapes so a PEM fits on one env line), `GITHUB_CLIENT_ID`, `GITHUB_CLIENT_SECRET`, `GITHUB_WEBHOOK_SECRET` (verifies `X-Hub-Signature-256` *and* signs the OAuth `state`), and `GITHUB_FRONTEND_BASE_URL` — documented in `.env.template`, with the required app config being `Contents: Read-only`, the **Push** and **Installation repositories** events, a callback URL at `/api/v1/integrations/github/callback`, and a webhook URL at `/api/v1/integrations/github/events`. There is no `GITHUB_REDIRECT_URI`; the GitHub App takes its callback URL from its own settings. Alongside it, **one new generic endpoint**, `POST /api/v1/integrations/{provider}/events`, is a provider-agnostic webhook receiver dispatched on the provider's registered `WebhookVerifier` (HMAC over the raw request bytes) — unauthenticated by design, since providers cannot send a bearer token; a provider that registers no verifier 404s, the same answer an unknown provider gets, so the route leaks nothing about which providers are configured. Deliveries are acked as soon as the signature checks out and handled detached, so a provider's delivery timeout is never in play. The route is declared `include_in_schema=False` and so does not appear in the OpenAPI spec. The `OAuthIntegration` extension seam gains four **optional, defaulted** hooks — `exchange_callback()` (for callbacks carrying more than a `code`), `webhook_verifier()` + `handle_webhook()`, and `on_installed()` (post-connect background work) — so existing Slack and third-party adapters are unaffected and future providers get webhooks with zero router changes. Token-leak hardening lands in the existing code path: `resolve_repo_source()` gained a `credentials` parameter that injects auth as environment-level git config instead of into the URL, strips URL userinfo from clone directory names (stable across hourly token rotations), rewrites the persisted git remote to the credential-free URL, pulls from the explicit credentialed URL, and scrubs tokens from logs and git error output — and `remember()` redacts repo specs in both result items and failure logs. **New public SDK kwarg**: `remember(..., content_type="code", repo_credentials="")`, rejected with `ValueError` for any other `content_type`. **No breaking change and no Alembic revision** — the connector stores its credential in the existing `integration_credentials` table (CLO-486, PR #4647). * Adds a **[Linear workspace connector](/integrations/linear-integration)** built as a Linear *agent* install: the authorize URL carries `actor=app` with `read,write,app:assignable,app:mentionable`, so Cognee joins the workspace as an app user members can **@mention or delegate issues to**. Those arrive as `AgentSessionEvent` webhooks (`created` from a mention or delegation, `prompted` from a follow-up) and are answered from memory — the handler posts a `thought` activity **first**, before any search, because Linear marks a session unresponsive without an activity within **10 seconds**, then runs a `HYBRID_COMPLETION` search across every dataset the connecting user can read (Linear's own `promptContext` and workspace `guidance` are prepended to the question to ground retrieval) and replies with a `response` activity via `agentActivityCreate`; every failure path ends in an `error` activity so a session never hangs. Alongside the agent loop, **issues sync as text**: `Issue` `create`/`update` deliveries and an install-time backfill of the **50 most recently updated issues** go through `remember(self_improvement=False)` — enrichment stays a human/scheduled decision — into **one dataset per workspace**, named `linear_` (the workspace's Linear URL slug lowercased, non-`[A-Za-z0-9_]` runs collapsed to `_`), so backend access control isolates at the workspace boundary. Each issue renders to deterministic plain text (identifier, title, URL, state, description — nothing volatile), so a re-sync of an unchanged issue produces byte-identical content. Issue **deletions are logged and dropped**; `forget()` stays a human decision. **Webhook security** is HMAC-SHA256 over the raw body against the `Linear-Signature` header with a constant-time compare, plus a **60-second `webhookTimestamp` replay guard** evaluated only *after* the HMAC passes (a timestamp from unverified bytes proves nothing) — both failures are a `401`, so a host clock drifting more than a minute rejects otherwise-valid deliveries. The token exchange is enriched with one GraphQL `viewer`/`organization` query because Linear's token response carries no workspace identity while every webhook envelope routes by `organizationId`, which becomes the credential's account id. An `OAuthApp` `revoked` delivery revokes the local credential, and — unlike GitHub — **`revoke_remote` is actually implemented**: Linear's token-revoke endpoint kills only Cognee's token without touching the app install, so a disconnect calls it best-effort. **No new endpoints and no router changes**: the connector rides the generic `/api/v1/integrations/{provider}/authorize|callback|events|connection` routes, and registration is a single import side effect in `cognee/api/client.py`. **New configuration** (all optional; a deployment without them boots normally and fails at use time, surfacing as a 503 from `/authorize` rather than a 500): `LINEAR_CLIENT_ID`, `LINEAR_CLIENT_SECRET`, `LINEAR_WEBHOOK_SECRET` (verifies `Linear-Signature` *and* signs the OAuth `state`), `LINEAR_REDIRECT_URI`, and `LINEAR_FRONTEND_BASE_URL` — documented in `.env.template`, with the required app config being agent capabilities enabled, a callback URL at `/api/v1/integrations/linear/callback`, and a webhook URL at `/api/v1/integrations/linear/events` subscribed to agent session, issue, and app-revoked events. Unlike the GitHub connector the stored credential holds a **real access token**, so `INTEGRATION_CREDENTIALS_KEY` (or the `INTEGRATION_CREDENTIALS_KEYS` keyring) is what protects it at rest. There is **no frontend Connect button yet** — drive the flow through `POST /api/v1/integrations/linear/authorize`. **No breaking change and no Alembic revision** — the connector stores its credential in the existing `integration_credentials` table (COG-6323, PR #4663). * Adds **in-flight progress for `add`/`cognify`**, so a client can show "3 of 10 files done" while a run is still going instead of only start and finish. Two surfaces ship it. **`GET /api/v1/datasets/status/progress`** ([dataset management](/cognee-cloud/functionality/dataset-management#in-flight-progress)) takes the same `dataset`/`pipeline` query parameters as `/status` and the same flat-versus-nested shaping (flat for zero or one pipeline, defaulting to `cognify_pipeline`; nested `{dataset_id: {pipeline_name: …}}` for more than one), but every value is `{status, progress}` instead of a bare status — `progress` being `{completed_items, total_items, current_stage}`, or `null` before the run's first tick and once it reaches a terminal state. Errors, including asking for a dataset you cannot read, are a `409` as on `/status`. Separately, the existing `/cognify/subscribe/{pipeline_run_id}` WebSocket ([cognify](/python-api/cognify)) now forwards **`PipelineRunProgress`** messages — one each time a processed result exits the run's task chain — carrying `current_stage`, `stage_index`, and `stage_total`; results stream through every stage before surfacing, so `current_stage` in practice always names the chain's final task and `stage_index` equals `stage_total`, making these messages a liveness heartbeat rather than a stage-by-stage tracker. The two channels deliberately carry different signals: progress messages have **no `payload` key** (the graph snapshot the other statuses attach is not recomputed per tick, since ticks are frequent and the graph has not changed shape), and their `completed_items`/`total_items` are always **`null`** — liveness is the WebSocket's signal, item counts are the endpoint's. Subscribe for live updates while a run is in flight; poll the endpoint to recover granular progress after a page refresh or a dropped subscription. Under the hood, progress is metadata inside the started state rather than a new run status — **`PipelineRunStatus` gains no member** — persisted by updating the run's existing `DATASET_PROCESSING_STARTED` row in place under a new `run_info["progress"]` key, the one exception to `pipeline_runs` being append-only ([pipelines](/core-concepts/building-blocks/pipelines)); inserting a row per tick would grow the table without bound as batches accumulate. Writes are throttled to roughly 20 per run (`max(1, total_items // 20)`, first and last item always persisted) to keep write pressure off the default SQLite backend, so the snapshot advances in steps on a large batch; concurrent ticks race last-write-wins with no locking, and a late tick can never resurrect a finished run — status readers pick the newest row, so a tick landing after the terminal row updates the older `STARTED` row invisibly, and the defensive path for a missing `STARTED` row drops the tick rather than inserting a late one. Progress failures are logged and swallowed, so reporting can never fail the run, and an item that errored still counts as completed so `completed_items` always reaches `total_items`. **No breaking change**: `/status` keeps its exact response shape, `cognify()` never returns or yields `PipelineRunProgress` (it is WebSocket-only), and no configuration option or environment variable ships. One Alembic revision does — `d1e2f3a4b5c6`, a composite `(dataset_id, pipeline_name, created_at)` index on `pipeline_runs` covering the latest-run lookup both status endpoints run, built with `CREATE INDEX CONCURRENTLY IF NOT EXISTS` on Postgres so a large table is not locked for the build. It applies through the usual upgrade path, so there is no manual step unless you run with `ENABLE_AUTO_MIGRATIONS=false` — then apply it with `cognee-cli upgrade` (CLO-557, PR #4535). * Makes `query` a **required** body field on `POST /api/v1/recall` and `POST /api/v1/search`. The field was semantically required all along, but it shipped with its OpenAPI example wired up as a functional default — `Field(default="What is in the document?")` on `RecallPayloadDTO` and `SearchPayloadDTO` — so a body that omitted `query` passed validation and was answered *as though the caller had asked that placeholder question*, across every dataset they could read. The default is replaced with `Field(..., examples=["What is in the document?"], description=...)`, which keeps the string as the schema example that prefills the interactive reference and Swagger "Try it out" while removing it as a value the server substitutes on the caller's behalf. **Breaking for callers that relied on the implicit placeholder:** a request with no `query` now fails Pydantic validation, and Cognee's global `RequestValidationError` handler turns that into **`400`** — not FastAPI's default `422` — with the usual `{"detail": [...], "body": ...}` shape naming the missing field. The one-line fix is to send an explicit `query` string; requests that already pass a real one behave exactly as before. **The Python SDK is unaffected** — `recall()` and `search()` already take `query_text` as a required positional argument — as are the CLI, MCP, and every curl sample in these docs, all of which pass a query. No configuration option, environment variable, or migration ships with the change. See [Requests without a `query`](/cognee-cloud/functionality/search-and-recall#requests-without-a-query) (fixes #4641, PR #4642). * Fixes namespaced Ollama model names and Hugging Face GGUF paths failing with `litellm.BadRequestError: LLM Provider NOT provided` on the `litellm_native` [structured output backend](/setup-configuration/structured-output-backends). `_qualify_model` (`cognee/infrastructure/llm/structured_output_framework/litellm_native/get_native_client.py`) prefixes an unroutable `LLM_MODEL` with its configured provider so LiteLLM, which routes on a provider-qualified name, can dispatch it — but it short-circuited on `"/" in model`, treating any slash as proof the name already carried a provider. Ollama names do not follow that rule: they can be namespaced (`library/phi4`), and a GGUF pulled from Hugging Face keeps its full path (`hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF`), which is the documented way to run one under Ollama. Those names reached LiteLLM unqualified and raised the provider-missing error before a request was sent. The slash check is dropped and the `litellm.get_llm_provider()` probe immediately below it — which was already doing the job the shortcut stood in for — now decides for every name, so `library/phi4` becomes `ollama/library/phi4` and the GGUF path becomes `ollama/hf.co/bartowski/…`. **The conservative guarantee is unchanged:** anything LiteLLM already resolves is returned untouched, verified by the two back-compat tests that pass either way — `ollama/phi4`, `openai/gpt-5-mini`, and `gpt-4o` all pass through as written, and `openai`/`azure` are still never prefixed. Only names that were unroutable before change behavior. Note that `LLM_PROVIDER="ollama"` must be set explicitly for a namespaced id, since `library` and `hf.co` are not prefixes [provider inference](/setup-configuration/llm-providers#provider-inference) recognises and an unset provider raises `ProviderNotDeducibleError` at configuration load. The cost is one extra `get_llm_provider` call for slash-containing names, a local lookup with no network round trip. No public API signature, configuration option, or environment variable changed, and no migration ships (fixes #4617, PR #4620). *** ## v1.5.3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.3)** Patch release that bumps the package version in `pyproject.toml` from `1.5.2` to `1.5.3` and regenerates `uv.lock` to match (PR #4637). It ships no functional code change; its substance is a packaging move: the `cryptography` dependency cap is relaxed from `<50` to `<51` and the lockfile moves to `cryptography` 50.0.0 (PR #4636) — downstream consumers pin `cryptography>=50.0.0` for PYSEC-2026-3552/3553/3554, and cognee's own usage (Fernet, AES-GCM) is stable across 50.x, so the two constraints no longer conflict in one environment. Alongside it, the release pipeline gains automation that bumps the `cognee-mcp` lockfile and publishes the MCP image on each release (PR #4623), and the `cognee-mcp` lockfile is bumped to cognee 1.5.2 (PR #4622) — CI and housekeeping changes with no entry of their own. Like v1.5.2 below, this cut is taken on the release line: work merged to the development branch in the meantime (the entries under Unreleased above) is not part of it. No public API signature, configuration option, or environment variable changed, and no Alembic revision ships — no migration is required when upgrading from v1.5.2. *** ## v1.5.3.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.3.dev1)** Development pre-release that bumps the package version in `pyproject.toml` to `1.5.3.dev1` and regenerates `uv.lock` to match — the lockfile change records the new `cognee` version and nothing else, with no dependency versions moved and no resolver timestamp refresh. The bump itself introduces no functional code, public API, configuration, or environment-variable change (PR #4678). It moves *from* `1.5.1`, not from `1.5.3`: the v1.5.2 and v1.5.3 cuts above were taken on the release line, so the development branch still carried the `1.5.1` marker, and this bump is what moves it onto the 1.5.3 line. **Neither this build nor `1.5.3` is a superset of the other.** This build carries development work the two release-line cuts do not — including four of the five entries under Unreleased above (PRs #4647, #4663, #4642, #4620; the cancelled-pipeline-run fix, PR #4680, merged after this cut and is not in this build), among other merges not logged individually here — while the fixes those cuts shipped are *not* in it: the `litellm_native` non-strict schema demotion (PR #4621, in v1.5.2) is absent, and the `cryptography` cap is still `>=43.0.0,<50` rather than the `<51` that v1.5.3 relaxed it to (PR #4636). If you need either of those, use `1.5.3`. Under PEP 440 the `.dev1` marker sorts *before* the already-published `1.5.3`, so a plain `pip install cognee` resolves the stable release and even `--pre` prefers `1.5.3` over it; reaching this build takes an explicit `cognee==1.5.3.dev1`. **Upgrading:** the bump adds no Alembic revision of its own, but three revisions ship in this cut that are in neither `1.5.1` nor `1.5.3` — `c4e8a1f6b3d7`, a composite `pipeline_runs (created_at, id)` index matching the activity feed's `ORDER BY created_at DESC, id DESC`; `b3d5f7a9c1e2`, which adds a non-nullable `has_full_metrics` boolean to `graph_metrics` with a `false` server default; and `d1e2f3a4b5c6`, a composite `pipeline_runs (dataset_id, pipeline_name, created_at)` index for the latest-run lookup behind `/status` and pipeline progress. So a deployment coming from either of those releases must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`); coming from `1.4.x`, the revisions introduced in the v1.5.0.dev1 section below apply as well. All three are inspector-guarded and skip work that is already present, so re-running them is a no-op. The new column is backfilled by the migration itself — existing rows with a computed `diameter` are set to `true`, the rest keep the `false` default — so no manual data action is required. *** ## v1.5.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.2)** Hotfix release on the 1.5.x line: the cut bumps the package version in `pyproject.toml` from `1.5.1` to `1.5.2` and carries exactly one fix — the `litellm_native` structured-output repair in the highlight below — plus adjustments to the Docker-compose end-to-end tests that ship alongside it and get no entry of their own. It is taken on the release line, not the development branch, so work merged to `dev` in the meantime (the entries under Unreleased above) is not part of it. No Alembic revision ships in this cut, so no migration is required when upgrading from v1.5.1. ### Highlights * Fixes the `litellm_native` [structured output backend](/setup-configuration/structured-output-backends) permanently abandoning the native path when a provider's strict mode rejects a response model's schema. Strict structured output accepts only a restricted schema subset, and some of cognee's own models fall outside it — the session-analysis `SessionTurnAnalysis` produces a `oneOf` from its discriminated union, and DataPoint-derived models like `EntityList`/`RuleSet` carry a free-form `metadata` dict that becomes `additionalProperties` — so the provider answered them with a schema `BadRequestError`. The adapter treated that as "native mode is broken" and dropped to the prompted-JSON fallback for the rest of the process, re-paying the failed strict request on every later call (for `SessionTurnAnalysis`, one failed request on every answered session turn). Now a schema-classified rejection is retried once with an explicit **non-strict** `json_schema` payload — the raw `model_json_schema()` with `strict: false`, which the provider accepts as guidance instead of enforcing — while conformance is still validated app-side against the original Pydantic model. The demotion is remembered in a module-level set keyed on `(llm_model, response_model.__name__)`, so the failed strict request is paid once per process rather than per call, and schemas strict mode does accept (the majority, including `KnowledgeGraph` extraction) keep their grammar-constrained guarantee untouched; if the non-strict retry is also rejected, prompted JSON remains the final fallback. A second, related repair: a native-path `ValidationError` — nearly unreachable under strict mode but routine the moment anything runs non-strict — now routes to the self-correcting JSON fallback, which feeds the validation error back to the model on retry, instead of bubbling into the outer retry loop that blindly re-sent the identical prompt (no error feedback) for up to 240 seconds of full-price LLM calls. New unit tests pin all three behaviors: the non-strict retry succeeding (with the next call skipping the strict attempt entirely), the prompted-JSON fallback when non-strict is also rejected, and exactly two calls on invalid native output. No public API signature, configuration option, environment variable, or migration ships with the fix (COG-6271, PR #4621). *** ## v1.5.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.1)** Patch release that closes out the 1.5.0 development line: the cut bumps the package version in `pyproject.toml` from the development marker `1.5.0.dev5` — the value the development branch carried after the v1.5.0 stable cut — to `1.5.1` and regenerates `uv.lock` to match. The lockfile change records the new `cognee` version and a `uv`-emitted `exclude-newer` compatibility placeholder only — no dependency versions moved — and the cut itself introduces no functional code, public API, configuration, or environment-variable change. Most of the work shipping in this release is logged under the v1.5.0.dev3, v1.5.0.dev4, and v1.5.0.dev5 pre-release sections below; the highlight in this section covers the work merged after the v1.5.0.dev5 cut, shipping for the first time in this release. **Upgrading:** the cut adds no Alembic revision of its own, but two revisions ship in this release that were not part of `1.5.0` — `a7f3c9e1b5d2`, introduced in the v1.5.0.dev3 pre-release below, which adds nullable operation-record columns to `pipeline_runs` (triggering user and tenant, operation name, start/end timestamps, outcome with error class and scrubbed message, token spend, originating surface, session/parent linkage, and a background-launch flag), and `c7e2a9b4d1f3`, new in this cut, which adds nullable `created_at` and `last_used_at` to `user_api_key`. So a deployment coming from `1.5.0` must run migrations when upgrading (`cognee.run_migrations()`, or `alembic upgrade head`); coming from `1.4.x`, the five revisions introduced in v1.5.0.dev1 apply as well. Both new revisions only add nullable columns and are inspector-guarded, so existing rows keep `NULL` (there is no backfill) and re-running them is a no-op — no data action is required. ### Highlights * Fixes overwriting a locally stored file on **Windows** raising `PermissionError` when another handle holds that file open — a regression from the change that made `LocalFileStorage.store()` atomic (PR #4581), so far published only in the `1.5.0.dev5` pre-release below. That change writes the payload to a sibling temp file (`...tmp`) and swaps it into place with `os.replace`, so a concurrent reader can never observe a half-written file; on Windows, though, replacing a file that another handle holds open requires that handle to have been opened with DELETE sharing, which an ordinary `open()` does not grant, so `os.replace` raises and a store that previously succeeded started failing. `store()` now catches `PermissionError` around the swap and falls back to the pre-atomic write, copying the temp file over the destination in place in 4 MiB chunks; the temp file is unlinked either way, on the fallback path as on the atomic one. Both branches of `store()` — text and binary stream — share the fallback, and it is triggered by the exception rather than by a platform check, so **POSIX keeps the atomic swap and never takes it**. The trade-off on the fallback path is explicit: the write is *not* atomic, so a reader holding an open handle sees the new bytes and can observe the file mid-write. That is the behavior local storage had on Windows before PR #4581, so nothing regresses relative to any stable release (the atomic swap, and with it this regression, has shipped only in the `1.5.0.dev5` pre-release) — but if you need snapshot semantics on Windows, coordinate readers and writers yourself rather than relying on `store()`. Alongside the fix, the unit tests added with the atomic store are adjusted to match: the atomic-store test now asserts the old-handle snapshot guarantee only on POSIX and elsewhere asserts just that the store succeeds, and `test_identify_data` patches `get_relational_engine` with `patch.object` on an explicitly imported module instead of a dotted `mock.patch` string — on Python 3.10 `mock.patch` resolves a dotted target with `getattr`, which lands on the `identify` function that the package `__init__` rebinds over the submodule of the same name and raises `AttributeError`, while 3.11+ resolves it through `pkgutil.resolve_name` and finds the module. Together these restore the red `windows-latest` and Python 3.10 CI jobs. No public API signature, configuration option, environment variable, or migration ships with the fix (COG-6241, PR #4596). *** ## v1.5.0.dev5 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.0.dev5)** Development pre-release that bumps the package version in `pyproject.toml` from `1.5.0.dev4` to `1.5.0.dev5` and regenerates `uv.lock` to match. The lockfile change records the new `cognee` version and a refreshed resolution timestamp only — no dependency versions moved — and the cut itself introduces no functional code, public API, configuration, or environment-variable change. This is a development cut taken on the branch that continues past the v1.5.0 stable release below, so under PEP 440 its `.dev5` marker still sorts *before* `1.5.0`: a plain `pip install cognee` resolves the stable release, and even `--pre` prefers `1.5.0` over it, so reaching this build takes an explicit `cognee==1.5.0.dev5`. **Upgrading:** no new Alembic revision ships in this cut, so no migration is required when moving between builds of the 1.5.0 line — but a deployment coming from `1.4.x` still needs the five revisions introduced in v1.5.0.dev1 below (`cognee.run_migrations()`, or `alembic upgrade head`). The highlights below are the work merged after the v1.5.0.dev4 cut, shipping for the first time in this pre-release; a CI-only test repair (PR #4587) ships alongside them and gets no entry of its own. Everything in the v1.5.0.dev4 and v1.5.0.dev3 sections below ships in this pre-release as well. ### Highlights * Cuts the number of relational-database sessions an added file costs from **\~11.1 to \~4.1**, and statements from **\~14.6 to \~6.6** — measured over a 164-PDF `add()` with real s3fs and asyncpg against Postgres configured the way the cloud pods are, `POOL_ARGS='{"poolclass": "nullpool"}'`. That configuration is what makes the count matter: with [NullPool](/setup-configuration/relational-databases) every SQLAlchemy session is a brand-new connection — TCP, TLS, and SCRAM-SHA-256, about 14 ms of event-loop CPU each on asyncpg 0.30 before any network latency — and NullPool is there deliberately, so this fix reduces the sessions needed rather than re-introducing pooling. The sessions were per *file*, not per batch: on the `add()` path the pipeline fans out one `run_tasks_data_item_incremental` per item and calls the tasks with a single-item list, so every "per call" lookup in `ingest_data` was really per file, and PR #4571's batched `identify_many()` was always resolving one hash. Two places shrank. **`run_tasks_data_item_incremental`, 4 sessions → 2:** its pre-check resolved the content's row id and then fetched that row by id just to read `pipeline_status`, which a new `identify_data_by_hash()` returns in one lookup; after the tasks ran it resolved the fresh content again and *then* opened another session to write the status, and the status session now does the resolution itself via `identify_data_by_hash(session=...)`. **`ingest_data`, 7 sessions → 2:** the task now accepts the pipeline's `ctx` and reuses `ctx.dataset` — which `run_pipeline` had already resolved and write-checked for the run — instead of re-resolving the dataset on every call, `get_dataset_data()`'s read of *every* `Data` row in the dataset (O(N²) rows over a 164-file add, just to build a membership map) is replaced by the targeted `Data.id.in_(...)` query that already returns the only ids that can be in it, and `identify_many()` gained an optional `session` so it shares one read session with that query while the commit keeps its own, so no connection is held idle across the loader/S3 work in between. `add()` also hands its resolved dataset id to the task so the non-`ctx` fallback stays on the cheap branch. **Permission semantics are unchanged:** `ctx.dataset` is reused only when it demonstrably is the dataset the caller selected — by id, or by name for a dataset the caller owns in the caller's tenant — and anything else still falls through to full resolution plus the write-permission check, which matters if you compose the built-in `ingest_data` into a [custom pipeline](/core-concepts/building-blocks/pipelines) pointed at another dataset. `identify()` and `identify_data()` now build their `(dataset, owner, tenant, content_hash)` filter from one shared `content_hash_predicates()` helper, and `identify_many()` replicates the same four predicates, so they cannot disagree on which row wins for a hash. No public API signature, configuration option, environment variable, or migration ships with this fix — `add()`'s signature and the `/add` endpoint are untouched, and deploying a build containing it is enough to pick it up (CLO-590, PR #4589). * Cuts the number of S3 requests an added file costs from **13 to 4** — PUT 2 → 1, HEAD 6 → 2, GET 5 → 1, measured per file at steady state over an instrumented 20-file `add()` on the S3 backend — and removes every event-loop-blocking metadata read from the `add()` path (a 12-file upload run performed 120 of them before, 0 after). The waste was self-inflicted: the payload was uploaded twice, because the incremental pre-check and `ingest_data` each stored the same item, and then downloaded and md5-hashed several more times, every read recomputing a byte-identical hash of content this process had just written — and those reads bridged to async through `run_sync`, which parks the event loop in a `thread.join()`, so the pipeline's 20-way item concurrency was mostly notional. The hash is now computed once, where the bytes already are: `save_data_to_file` returns the metadata it computed from the in-memory payload, the pre-check publishes it — plus where the payload landed — on `ctx.extras`, and `ingest_data` consumes that handoff instead of repeating the upload and the read-back; the handoff is matched on item identity with a stored-path fallback, so anything handing `ingest_data` a different object still does the full work itself. Loaders describe their own output the same way: [`LoaderResult`](/core-concepts/further-concepts/loaders) gained an optional `file_metadata` field, and the new `store_derived_text` helper stores a loader's extracted text and fills that field in one step — returning a plain `str` path is still fully supported and simply keeps the old read-back (over S3, a HEAD plus a full GET of content the loader had in memory). Async callers get metadata through new `aget_metadata` / `aget_identifier` accessors on the ingestion data types, so no `run_sync` bridge remains on the add path. **Storage semantics change in three ways.** Uploaded source files are now stored under content-addressed keys, [`/`](/core-concepts/main-operations/legacy-operations/add) instead of the bare filename written with `overwrite=True`, so two different uploads sharing a name no longer silently clobber each other and re-adding identical bytes is idempotent; the basename stays the user's real filename because the code-graph route keys node identity on it, loaders select by suffix, and dlt derives its source name from it — derived text and raw text keep their flat `text_.txt` names. Local writes are atomic (write to a temp file, then `os.replace`), so a content-addressed key another reader may hold open can no longer be observed half-written. And a replaced or deleted original is now actually reclaimed: `remove_data_file_if_unreferenced` runs on delete and on content-changed update, ref-counted across both location columns and all datasets, and never touches user-owned paths. Two mechanical notes: both storage backends stream file-like payloads in 4 MiB chunks, so the peak memory an upload adds is one chunk rather than the whole file, and the s3fs client's botocore connection pool is raised from its default 10 to **32** — above the pipeline's default per-dataset item concurrency of 20, so concurrent items are limited by the network rather than the pool; that value is a fixed internal constant with no environment variable. **Operator impact:** the new key shape applies to files stored after the upgrade — existing objects stay at the paths recorded on their `Data` rows and remain readable, and no migration rewrites them — but tooling that asserts on flat filename keys inside `DATA_ROOT_DIRECTORY` needs updating. The rows themselves are unchanged: a 12-file corpus produces `Data` rows identical to the previous behavior across both the upload and local-path flows, on every column including `content_hash` and `data_size`. Loader `load()` implementations may now return `LoaderResult` where callers and tests previously assumed a plain `str`. Beyond that additive `LoaderResult.file_metadata` field, no public API signature, configuration option, or environment variable changed, and no migration ships — deploying a build containing it picks everything up (COG-6241, PR #4581). *** ## v1.5.0.dev4 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.0.dev4)** Development pre-release that bumps the package version in `pyproject.toml` from `1.5.0.dev3` to `1.5.0.dev4` and updates `uv.lock` to match — the lockfile change records the new `cognee` version and a refreshed resolver `exclude-newer` timestamp only, with no dependency versions moved. Unlike the neighboring cuts, the tag does not point at a pure version bump: the same merge (PR #4590) also adds diagnostic `logger.info` lines around the ingestion path's `store_to_dataset` flow to help troubleshoot `add()`, a logging-only change with no behavior difference. No new Alembic revision ships since v1.5.0.dev3, so no migration is required when moving between these builds; a deployment coming from `1.4.x` still needs the five revisions introduced in v1.5.0.dev1 below (`cognee.run_migrations()`, or `alembic upgrade head`). ### Highlights * Stops the `POST /v1/search` telemetry event from carrying the raw text of a request, bringing it in line with the convention the recall path already followed. The `Search API Endpoint Invoked` event previously shipped four request fields verbatim; they now carry sizes under the same property keys, so downstream event schemas keep their columns: `query` and `system_prompt` become the character length of the string (`0` when unset), `node_name` becomes the number of node-set filters passed (`0` when unset), and `code_query` becomes the character length of the structured operation dict's string form (`0` when unset). Every other property on the event — `endpoint`, `search_type`, `datasets`, `dataset_ids`, `top_k`, `only_context`, `verbose`, `skills`, `tools`, `max_iter`, `include_references`, and `cognee_version` — is unchanged, and `cognee.recall` already reported `query_length` this way. **This is a telemetry-only change**: search request handling, results, and options are identical, and no public API signature, configuration option, or environment variable ships with it, so no migration is required. Analytics that read those four properties as strings need to be updated to read integers; [`TELEMETRY_DISABLED=true`](/setup-configuration/overview#observability--telemetry) still turns collection off entirely. A wiring test (`cognee/tests/unit/api/test_search_router_event_properties.py`) now pins the event's property set against future drift (COG-6244, PR #4588). *** ## v1.5.0.dev3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.0.dev3)** Development pre-release that bumps the package version in `pyproject.toml` from `1.5.0.dev1` to `1.5.0.dev3` and updates `uv.lock` to match; the tag points at the version-bump merge itself. The development branch moves straight from `dev1` to `dev3` because the `1.5.0.dev2` marker was set on a separate cut that this branch does not carry. The lockfile change records the new `cognee` version and a refreshed resolver `exclude-newer` timestamp only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself, and the bump introduces no code, public API, configuration, or environment-variable change of its own. One new Alembic revision ships since v1.5.0.dev1 — `a7f3c9e1b5d2`, which extends `pipeline_runs` into a general operation record (PR #4561) — so run migrations when upgrading from that pre-release (`cognee.run_migrations()`, or `alembic upgrade head`); coming from `1.4.x`, the v1.5.0.dev1 migrations below also apply. What the changed version value does trigger on its own is the migration runner's vector-adapter storage sync (for example, LanceDB columns), which runs after the chain on a `cognee_version` mismatch. The entries below are the work merged to the development branch since v1.5.0.dev1. ### Highlights * Rewrites the [Postgres graph adapter](/setup-configuration/graph-stores) as a bare-bones reference implementation over two ordinary tables, and fixes concurrent writers in separate processes losing each other's updates. **Nothing changes for deployments that configure the backend through `GRAPH_DATABASE_PROVIDER`:** `postgres_demo` is now the canonical value, `postgres` is still accepted and resolves to the same adapter, no public adapter method was removed, deprecated, or changed shape, and no configuration option, environment variable, or migration ships with the rewrite. **What does break is direct imports** — the module moved from `cognee/infrastructure/databases/graph/postgres/` to `cognee/infrastructure/databases/graph/postgres_demo/` and the class from `PostgresAdapter` to `PostgresDemoAdapter`, with the old package deleted rather than aliased, so `from cognee.infrastructure.databases.graph.postgres.adapter import PostgresAdapter` now raises `ModuleNotFoundError`. Repoint such imports at `cognee.infrastructure.databases.graph.postgres_demo.adapter.PostgresDemoAdapter`, or better, obtain the engine from `get_graph_engine()`, which never needed updating. The concurrency fix is the one behavior change an operator will notice: the adapter previously guarded writes with an in-process `asyncio.Lock`, which cannot coordinate anything outside its own interpreter, so two workers attaching provenance or removing node-set tags at the same time read-modified-wrote over each other. Every write entrypoint — `add_nodes`/`add_edges` (and the single-item `add_node`/`add_edge` that delegate to them), `delete_nodes`, `delete_edge_triples`, `delete_graph`, `attach_node_source_refs` / `attach_edge_source_refs`, `remove_node_source_refs` / `remove_edge_source_refs`, `remove_belongs_to_set_tags`, and `set_graph_metadata` — now takes one transaction-scoped Postgres advisory lock (`pg_advisory_xact_lock`) before touching a row, with `FOR UPDATE` row locks added on the read-modify-write provenance and tag paths; edge identities are locked in sorted order. **The tradeoff:** concurrent writers queue instead of running in parallel, so a write-heavy multi-worker deployment sees writes take turns; the database user must be able to acquire advisory locks (standard PostgreSQL installations allow this); and reads never take the lock, so they are unaffected. A plain row-lock scheme was not enough — batches inserting the same node wait on each other's unique index entry and a cascading delete takes rows in a different order, which Postgres reports as a deadlock — and because the lock is transaction-scoped it is released on commit *and* on rollback, so a failed write cannot strand it. Internally, short static SQL replaces the previous recursive CTEs, aligned-array `unnest`, and grouped provenance writes, with traversal and metrics computed in Python; the table schema (`graph_node`, `graph_edge`, `graph_metadata`, and the `text[]` provenance columns) is untouched, so there is no data action on upgrade, and the `postgres_graph` / `postgres_graph_shared` dataset-database handlers accept both provider names. The Postgres graph store remains a **demo feature** and is still not production-ready — use a graph-native backend such as Kuzu or Neo4j for the graph layer in production — and it still does not support raw Cypher, so `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` continue to raise `SearchTypeNotSupported` (SDK-63, PR #4573). * Fixes indexing mutating the caller's `DataPoint` — which, on a DataPoint that declares more than one index field, also made its vector collections embed the wrong field's text. `index_data_points()` walks `metadata["index_fields"]` and, for each field, prepares a copy of the DataPoint whose `index_fields` is narrowed to just that field. That narrowed marker is not cosmetic: every in-tree adapter that implements `index_data_points` (the LanceDB, PGVector, and Turso vector adapters, plus the hybrid Neptune Analytics adapter) reads `metadata["index_fields"][0]` off the copy — directly, or through `DataPoint.get_embeddable_data()` — to decide which attribute to embed. The copy was a shallow `model_copy()`, so copy and original shared one `metadata` dict and every narrowing assignment landed on that shared dict. Two consequences followed for a DataPoint with two or more index fields: the caller's own `index_fields` list came back replaced by a single-element list, and — because all per-field copies are prepared before any embedding batch is dispatched — every copy ended up pointing at the **last** declared index field, so a `Product` with `metadata={"index_fields": ["name", "description"]}` had its `description` text embedded into both the `Product_name` and `Product_description` collections. The copy is now `model_copy(deep=True)`: each per-field copy owns its own `metadata`, the caller's list survives indexing unchanged, and `Product_name` embeds `name` while `Product_description` embeds `description`. **Single-index-field DataPoints were never affected** — the narrowing rewrote the list to a value-equal one and the embedded field was already correct — and that covers every built-in type in the default ingestion path (`Document`, `DocumentChunk`, `Entity`, `EntityType`, `TextSummary`, each declaring exactly one index field), so ordinary `remember()` / `cognify()` graphs need no attention. What is affected is custom DataPoint subclasses declaring several embeddable fields, plus the in-tree multi-field models (`WebPage`, `WebSite`, `ScrapingJob` in the web scraper, the schema and translation task models, `SkillRun`, `SkillImprovementProposal`, and `Tool`, whose two `Embeddable` annotations auto-derive `index_fields=["name", "description"]`). The multi-field `GraphitiNode` is **not** on that list: the temporal-awareness pipeline indexes it through its own copy-and-narrow loop, which never routes through this function and already got per-copy `metadata` in the fix for PR #3580. **Data action for those:** rows already written to the non-last collections hold the wrong field's embedding and the fix does not rewrite them. On the default LanceDB backend, re-adding the affected DataPoints (`add_data_points()`, or re-running graph building over the source data) is enough — its `merge_insert` upsert rewrites the stored vector along with the payload. On PGVector and Turso the `ON CONFLICT (id)` clause updates only the row's `payload` and keeps the existing vector, so delete the stale rows from the affected collections first (`delete_data_points`) and then re-add, or the wrong embeddings will survive. One cost note: `deep=True` duplicates nested DataPoint fields per indexed field as well, so preparing the copies is slightly more expensive for large nested objects; what gets written is unchanged. No public API signature, configuration option, environment variable, or migration ships with this fix — deploy a build containing it to pick it up (PR #4487). * Changes the Ollama provider's default [instructor mode](/setup-configuration/llm-providers#llm-instructor-modes) from `json_mode` to `json_schema_mode`, so an unset `LLM_INSTRUCTOR_MODE` now sends the Pydantic schema to Ollama as a JSON-Schema decoder constraint instead of only asking for `response_format: {"type": "json_object"}` and describing the schema in prompt text. Under `json_mode` the model's output was validated after the fact, and a local model frequently failed that validation on the first attempt — measured against `llama3.1:8b` with Cognee's own graph-extraction prompt, the real `KnowledgeGraph` model, and `max_retries=0`, first-attempt validity goes from **2/5** to **5/5**; the `json_mode` failures were `InstructorRetryException` with missing required fields (`summary`, `Node.description`, `Node.label`). Corroborated one layer down at the Ollama API with the same prompt and schema: `format: "json"` produced schema violations in 6/6 calls (three returning zero nodes), the full JSON Schema in 0/6. Ollama has supported JSON-schema structured outputs since **0.5**, and this aligns it with `openai` and `bedrock`, the other table entries whose endpoints can *enforce* a schema rather than describe one. **If you run an Ollama older than 0.5**, set `LLM_INSTRUCTOR_MODE=json_mode` to keep the previous behavior — that escape hatch is the existing config override and is unchanged, and it takes precedence over the provider default. No public API signature, configuration option, or environment variable changed, and no migration ships (PR #4560). * Stops a stable release from publishing a `cognee/cognee-mcp` image whose tag does not match the `cognee` library inside it — the drift that shipped the `1.4.1` and `1.5.0` MCP images (issue #4360). The MCP image installs `cognee` from PyPI through `cognee-mcp/uv.lock`, which can only be re-locked *after* the new version is on PyPI; that manual lock bump was missed twice and failed silently, so the image was built against whatever older `cognee` the lock still pinned. The release workflow's Docker job now runs a **Check MCP lockfile ships the released cognee version** step that reads the `cognee` entry out of `cognee-mcp/uv.lock` and compares it exactly against the version being released, positioned after the `cognee/cognee` image is pushed and **before** either `cognee-mcp` image build, so a mismatch fails the job instead of pushing a skewed image. The failure is annotated on `cognee-mcp/uv.lock` and names the remedy: once the new `cognee` is on PyPI, run `uv lock --upgrade-package cognee` in `cognee-mcp/`, merge the lock bump, then re-run the job — the GitHub release, the PyPI publish, and the `cognee/cognee` image all precede this step, so a failure blocks only the MCP image, not the release itself. **The check runs only on `main` releases:** dev canaries are exempt because their `.devN` version cannot be in the lock before it is published. Alongside the guard, `cognee-mcp`'s own floor moves from `cognee[postgres-binary,docs,neo4j]>=1.4.2,<2.0.0` to `>=1.5.0,<2.0.0` and its lockfile is re-resolved to `cognee` 1.5.0 (the pre-existing `[tool.uv] exclude-newer-package = { cognee = "0 days" }` escape from the 2-day supply-chain window is what lets a release-day `uv lock` see the new version at all, and is unchanged here). This is release-pipeline plumbing only: no public API signature, configuration option, environment variable, or migration ships with it, and nothing changes for users of the published MCP image beyond the guarantee that its tag and its bundled `cognee` agree (SDK-425, PR #4567). * Stops an `ERROR`-level `PermissionDeniedError raised (Status code: 403)` line from being logged every time a user who has no datasets at all reads memory — the state every fresh install is in before its first ingestion, and the reason a plain `recall()` against a new deployment printed a 403 error line while otherwise behaving normally. `get_specific_user_permission_datasets()` has two raise sites: one for *requested* dataset ids the caller cannot access, and one for "the caller has zero datasets carrying this permission at all", which is reachable only when no dataset ids are passed. The second is an ordinary state rather than an authorization failure, and its callers already treated it as one — `get_readable_datasets()` (and `get_permitted_dataset_ids()` on top of it) converts it into an empty list — but the log line is emitted inside the exception constructor, before any caller gets a chance to catch it. The base `CogneeApiError` has always accepted `log` and `log_level` arguments and dispatched to the matching logger method; `PermissionDeniedError.__init__` just never forwarded them, so every instance logged at `ERROR`. It now takes `log: bool = True` and `log_level: str = "ERROR"` and passes them to the base class, and the zero-dataset raise site passes `log_level="DEBUG"`. **The exception itself is unchanged**: it is still raised, still carries status 403 and the same `Request owner does not have permission: [] for any dataset.` message, and the global `CogneeApiError` handler still maps it to a 403 response anywhere it propagates uncaught. In practice no in-tree caller lets this particular raise reach a client as a 403 today: read paths convert it to an empty list, and `POST /v1/sync` with no dataset ids catches it in its blanket `except Exception` and answers `409` with `{"error": "Cloud sync operation failed"}`. The requested-datasets raise — `Request owner does not have necessary permission: [] for all datasets requested.` — is untouched and still logs at `ERROR`, because asking for a specific dataset you cannot read is a real denial. **Operator action:** anyone alerting or grepping on `ERROR`-level `PermissionDeniedError raised (Status code: 403)` lines will stop seeing them for the zero-dataset case; set [`LOG_LEVEL=DEBUG`](/setup-configuration/overview#environment-variable-quick-reference) to keep observing it. No configuration option, environment variable, or migration ships with this fix, and the only signature change is the two additive keyword arguments on `PermissionDeniedError` — deploy a build containing it to pick it up (COG-6268, PR #4619). * Stops aiohttp's `Unclosed client session` warning from being printed when a Cognee process exits. Telemetry reuses a single process-wide `aiohttp.ClientSession`, created lazily and bound to the event loop it was built on, so that each `send_telemetry()` call skips a DNS + TCP + TLS handshake to the collector — but nothing ever closed it, so the session was still open when the interpreter (or the loop that owned it) went away and aiohttp complained at garbage collection. Long-running servers and one-shot SDK scripts alike ended their run with the warning. A new `close_telemetry_session()` in `cognee.shared.utils` awaits the outstanding fire-and-forget telemetry tasks, closes the session, and clears the module-level references; it is idempotent, safe to call from any loop, and `_get_telemetry_session()` transparently rebuilds a session if telemetry fires again afterwards. Three call paths use it: the FastAPI `lifespan` shutdown block awaits it alongside the existing graph and vector engine `cache_clear()` calls, so the server closes the session on the loop that owns it; an `atexit` hook, registered the first time a session is created, is the last-resort path for CLI runs and SDK scripts that never go through the server; and the loop-change branch of `_get_telemetry_session()` now closes the stale session before replacing it instead of dropping it unclosed. Telemetry behavior is otherwise unchanged — still best-effort, still silently skipped when there is no running loop, and still switched off entirely by [`TELEMETRY_DISABLED`](/setup-configuration/overview#observability-%26-telemetry) — so the only user-visible difference is the missing warning. No public SDK signature, configuration option, environment variable, or migration ships with this fix (COG-6270, PR #4619). * Sends the configured [`LLM_TEMPERATURE`](/setup-configuration/llm-providers#temperature-and-seed) on local inference servers even when the variable is unset, fixing extraction on Ollama running at whatever the model itself defaults to — `1.0` for several Ollama models — while `docs/ollama_models.md` told users extraction wants `0.0` (issue #4631). `LLMConfig.fold_sampling_params_into_llm_args` folds `llm_temperature` into `llm_args`, the dict every adapter merges into each completion call, and it did so only when the field was in `model_fields_set`. That gate exists for a real reason its docstring names — the default gpt-5 family rejects any temperature but the provider default, so an unset field must not silently send `0.0` there — but the restriction belongs to the hosted OpenAI reasoning models, not to every provider, so Ollama, llama.cpp, and LM Studio inherited a workaround for a limit they do not have. The gate now also passes when `is_local_llm(self.llm_provider, self.llm_model)` is true, the same predicate the validator directly below it (`default_local_rate_limit_budget`) already uses to give local servers a smaller default RPM budget for the same class of reason. `llm_temperature` already defaulted to `0.0`, so no new value is introduced — only the gate changed. **Who is affected:** a deployment on `ollama` or `llama_cpp`, or on an `lm_studio/`-prefixed model, that never set `LLM_TEMPERATURE` now gets `temperature: 0.0` where it previously got the model's own sampling default; extraction becomes deterministic, which is the documented recommendation. **Hosted providers, vLLM included, are unchanged** — `is_local_llm` deliberately excludes vLLM, which serves with continuous batching and is treated as a cloud endpoint throughout, so an unset variable there still sends nothing and the gpt-5 default path is untouched. **Both escape hatches still work and take precedence:** set `LLM_TEMPERATURE` to any value to fold that value instead, or give a `temperature` key directly in `LLM_ARGS`, which wins over the dedicated field (`self.llm_args = {**folded, **(self.llm_args or {})}`) — that is how a local deployment restores its pre-upgrade sampling. `.env.template` and `docs/ollama_models.md` are updated to match, and four unit tests in `cognee/tests/unit/infrastructure/llm/test_llm_config.py` pin the local/non-local boundary and the two precedence rules. No public API signature or new configuration option ships, and no migration is required — deploy a build containing the fix to pick it up (PR #4634). * Makes the [`neo4j` dataset database handler](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-how-to-use-them) fail actionably when the Neo4j server behind it cannot host per-dataset databases. That handler gives each dataset its own database inside one DBMS via `CREATE DATABASE`, which is an Enterprise/AuraDB feature — a Community server serves exactly one database and rejects the command. Previously **every** `neo4j.exceptions.Neo4jError` raised by a system-database command was flattened into one generic `EnvironmentError` reading *"Local Neo4j multi-user mode requires a Neo4j deployment that supports CREATE/DROP DATABASE and credentials with database-management privileges."* — indistinguishable by type from any other OS-level failure, conflating the two distinct causes and naming no remedy. Provisioning now (1) probes the server edition with `CALL dbms.components() YIELD edition` **before** `CREATE DATABASE` runs, and (2) translates the `Neo.ClientError.Statement.UnsupportedAdministrationCommand` code the command returns if the probe could not run — the probe is deliberately best-effort, so a server that restricts `dbms.components()` is caught by the second path rather than failing on the probe itself. Either path raises the new `Neo4jMultiDatabaseSupportError`, a `CogneeConfigurationError` subclass (HTTP 422) exported from `cognee.infrastructure.databases.exceptions`, whose message spells out the four ways forward: connect to a Neo4j Enterprise or AuraDB deployment; keep the same server and set `GRAPH_DATASET_DATABASE_HANDLER=neo4j_community` to isolate each dataset in [its own Docker container](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community) (needs a reachable Docker daemon); switch to a backend with built-in multi-tenancy such as the default `ladybug`/`kuzu`; or set `ENABLE_BACKEND_ACCESS_CONTROL=false` and accept that all datasets share one graph database with per-dataset isolation lost. A rejection carrying a `.Security.` code — the other cause the old message lumped in — is separated out into `DatabaseCredentialsError`, telling the operator the configured credentials lack database-management privileges (e.g. the admin role) rather than implying the server is the wrong edition. The translation sits in the shared system-query helper, so `DROP DATABASE` on dataset deletion and the `SHOW DATABASES` readiness poll report the same way; anything that is neither an unsupported-command nor a security code still raises the original generic `EnvironmentError`, unchanged. **Nothing that worked before changes:** an Enterprise/Aura deployment provisions exactly as it did, and this only replaces the error a Community setup was already failing with. No public API signature, configuration option, environment variable, or migration ships with this fix — deploy a build containing it to pick it up (SDK-375, PR #4606). * Fixes two ways a small or local model could take down a run, both surfacing on the same nightly suites. First, [`STRUCTURED_OUTPUT_FRAMEWORK=litellm_native`](/setup-configuration/structured-output-backends) now qualifies a bare `LLM_MODEL` with its LiteLLM provider prefix. Cognee keeps provider and model as separate settings and the `instructor` path did its own per-provider dispatch, but LiteLLM routes on a provider-qualified model name — so a config that is perfectly valid under `instructor`, `LLM_PROVIDER=ollama` with `LLM_MODEL=phi4`, reached LiteLLM as a bare `phi4` and died with `litellm.BadRequestError: LLM Provider NOT provided. You passed model=phi4` before a request was sent. A new `_qualify_model()` in `get_native_client.py` prefixes the model for the five providers whose LiteLLM prefix is unambiguous — `ollama`, `anthropic`, `gemini`, `mistral`, and `bedrock` — and is deliberately conservative everywhere else, so it cannot re-route a configuration that works today: an already-qualified name (`ollama/phi4`) is returned untouched, and so is any bare name `litellm.get_llm_provider()` already resolves on its own. **`openai` and `azure` are excluded by design** — LiteLLM already resolves bare OpenAI model names, and Azure needs a deployment-specific form Cognee will not guess at, so Azure users keep writing the qualified model their deployment expects. The `instructor` and BAML backends are untouched. Second, `SummarizedContent.description` — the field that is unused and kept only for backwards compatibility — no longer fails validation on non-string model output. It is still part of the JSON Schema handed to the LLM, so a model is free to fill it, and smaller local models routinely answered with a list of bullets; strict validation then failed the whole structured-output call, retries exhausted, and an entire `cognify()` run died on a field nothing reads (`ValidationError: 1 validation error for SummarizedContent / description / Input should be a valid string ... input_type=list`, observed nightly on the llama-cpp suite). A `mode="before"` field validator now coerces instead of rejecting: `None` becomes `""`, a list or tuple is joined with newlines, anything else is stringified. **`summary` — the field actually consumed — keeps strict validation**, so a model that fails to produce a usable summary still errors. Being a model-level change, it applies to every structured-output backend. The rest of the PR is CI-only (nightly timeouts, taking the nightly off the PR gate, Windows runner flags, a perf-bench backend cap, and cloud tenant-creation retries) and has no user-facing surface. No public API signature, configuration option, environment variable, or migration ships with either fix — deploy a build containing them to pick them up (CLO-594, PR #4600). * Fixes `TikTokenTokenizer.decode_token_list()` failing on every non-empty input with `TypeError: 'int' object is not an instance of 'Sequence'`. The method looped over the token ids and handed each one to tiktoken's `Encoding.decode` as a bare `int` (`self.tokenizer.decode(i)`), but that method takes a *sequence* of ids, so the first iteration raised and the method could never return anything but the empty list it short-circuits to for empty input. Each id is now wrapped before decoding (`self.tokenizer.decode([i])`), which is the per-token decode the list comprehension was already written to express: `decode_token_list(tokenizer.extract_tokens("hello world foo"))` returns one string per token, and joining them reproduces the original text. The pre-existing coercion of a non-list argument to a single-element list is unchanged, as is the method's signature. **Nothing in Cognee itself was affected:** `decode_token_list` is not declared on `TokenizerInterface` — which specifies only `extract_tokens`, `count_tokens`, and `decode_single_token` — and no code in the repository calls it, so the token counting that drives chunk sizing and the sibling `decode_single_token` (which goes through `decode_single_token_bytes` and was always correct) never routed through the broken path. Only code calling this method directly on the TikToken adapter sees a difference. A unit test (`cognee/tests/unit/infrastructure/llm/test_tiktoken_adapter.py`) round-trips `extract_tokens` into `decode_token_list` and asserts the pieces rejoin into the original string. No public API signature, configuration option, environment variable, or migration ships with this fix — deploy a build containing it to pick it up (fixes #4594, PR #4607). * Fixes Vertex AI batch-size rejections failing an embedding run instead of being recovered. `LiteLLMEmbeddingEngine.embed_text` has long had a recursive split-and-retry path for embedding requests a provider rejects as too large, but because the embeddings API returns these as a plain `400 BadRequestError` rather than LiteLLM's `ContextWindowExceededError`, the engine has to recognize them by message — and the guard matched only OpenAI's wording, `maximum input length`. Vertex AI words its per-request instance cap differently (`2048 instance(s) is allowed per prediction`), so the error fell through the guard, was re-raised, and killed the run even though the engine already knew how to recover from it. The guard's regex now also matches `instance(s) is allowed per prediction`, routing Vertex's rejection into the same recovery. **Which branch of that recovery does the work matters here:** a Vertex instance cap limits how many texts one request may carry, not how long any single text is, so it is the batch-halving branch that resolves it — the batch is split in half, each half embedded in parallel, and the recursion repeats until every request is under the cap — not the single-string mean-pooling branch that handles a genuinely over-length text. **Fast-fail behavior for genuinely bad requests is unchanged:** the match is still kept narrow and case-insensitive over these two phrasings, and any other `400 BadRequestError` is re-raised unchanged. Affected are deployments embedding through Vertex AI on the LiteLLM engine — directly or via a LiteLLM-compatible gateway — with a batch larger than the model's instance cap; lowering `EMBEDDING_BATCH_SIZE` under the cap remains the cheaper path, since each recovery costs extra round trips, but it is no longer required to get the request through. A new unit case in `cognee/tests/unit/infrastructure/test_embedding_context_window_fallbacks.py` pins the behavior. No public API signature, configuration option, or environment variable changed, and no migration ships — deploy a build containing this fix to pick it up (PR #4569). * Fixes `from cognee.tasks.web_scraper import *` failing outright with `AttributeError: module 'cognee.tasks.web_scraper' has no attribute 'BeautifulSoupCrawler'`. The package's `__all__` still listed `BeautifulSoupCrawler`, a name left behind by a rename that nothing in the package defines — the crawler it named is `DefaultUrlCrawler`, exported alongside it, and the similarly named `BeautifulSoupLoader` under `cognee/infrastructure/loaders/external/` is an unrelated loader. With the dead entry dropped, `__all__` lists exactly the four names the module resolves: `fetch_page_content` and `DefaultUrlCrawler`, imported eagerly, plus `web_scraper_task` and `cron_web_scraper_task`, which the module's `__getattr__` loads lazily from `cognee.tasks.web_scraper.web_scraper_task` on first access — that lazy import still needs the optional `apscheduler`, unchanged here. **Only star imports and tooling that reads `__all__`** (documentation generators, re-export checks) were affected; targeted imports such as `from cognee.tasks.web_scraper import DefaultUrlCrawler` always worked, and no name that exists today changes behavior. The same rename residue in the `DefaultUrlCrawler.__init__` docstring — which opened with *"Initialize the BeautifulSoupCrawler."* — is corrected, and a new unit test (`cognee/tests/unit/tasks/web_scraper/test_public_exports.py`) asserts that every name in `__all__` resolves, skipping a name whose optional dependency is missing, so a future rename cannot leave the list stale again. No public API signature, configuration option, or environment variable changed, and no migration is required — deploy or reinstall a build containing this fix to pick it up (PR #4465). * Fixes a [stage-routed](/setup-configuration/llm-providers#per-stage-model-routing) LLM config keeping the rate-limit default that was derived for the **base** provider. `LLMConfig.stage_config()` builds a stage's effective configuration with `model_copy(update=...)`, which does not re-run validators, so the provider-dependent default `default_local_rate_limit_budget` had already derived for the base provider survived onto the copy — even though pointing a stage at a different provider is the entire purpose of the method. The shape shows up in the documented worked example for stage routing: with an OpenAI base and `LLM_EXTRACTION_PROVIDER="ollama"`, `stage_config("extraction")` reported an `llm_rate_limit_requests` of `60`, while the identical configuration built directly reported `10` — the smaller budget local inference servers get because they process requests near-serially. The rate-limit logic moves out of the validator body into a module-level `_apply_local_rate_limit_default()` that both the validator and `stage_config()` call, so a stage now re-derives the default from its own effective provider and model. Detection is unchanged (`ollama` and `llama_cpp` by provider, `lm_studio/` by model prefix, vLLM deliberately excluded as a regular provider), an explicitly configured `LLM_RATE_LIMIT_REQUESTS` still wins, and a stage routed to another cloud provider keeps `60`. **Scope worth knowing before expecting a throughput change:** this corrects the *resolved configuration*, not the pacing. Every runtime reader of the budget — the dispatch seam in `cognee/shared/rate_limiting.py`, the legacy `llm_rate_limiter` singleton, and the overload-policy warning — builds from the process-wide base config rather than the per-stage one, so client-side throttling is not repartitioned per stage; size it for the server that receives the high-volume stage by setting `LLM_RATE_LIMIT_REQUESTS` explicitly. One asymmetry follows from the base validator's own assignment marking the field as set: a config whose *base* provider is local keeps `10` on a stage routed to a cloud provider. A stage that sets only `LLM__MODEL` to a non-local model still keeps the base provider and its budget, as before. Three unit tests in `cognee/tests/unit/infrastructure/llm/test_stage_routing.py` pin the local-budget case, the explicit-override case, and the cloud-to-cloud case. No public API signature, configuration option, environment variable, or migration ships with this fix — deploy a build containing it to pick it up (PR #4635). * Fixes three traversal helpers on the default [Ladybug (Kuzu) graph adapter](/setup-configuration/graph-stores) generating Cypher that Ladybug's parser rejects, so `get_predecessors()` and `get_successors()` answered with an empty list on every call and `get_disconnected_nodes()` raised (issue #4365). The two directional helpers returned `properties(m)`, whose argument Ladybug binds as `(LIST, STRING)` rather than as a node, so the statement failed at compile time; because both wrap their query in a `try`/`except` that logs and returns `[]`, the failure surfaced as *no predecessors* / *no successors* rather than as an error. They now project the fields explicitly — `RETURN {id: m.id, name: m.name, type: m.type, properties: m.properties}` — and pass each row through `_parse_node_properties`, which merges the stored JSON `properties` blob into the flat dictionary. That is the same projection-plus-parse pattern `get_neighbors()` already used, so the two helpers now return the adapter's standard flat node dictionary (`id`, `name`, `type`, plus the stored DataPoint fields kept in that blob) — the shape their callers were always written against. `get_disconnected_nodes()` filtered on `WHERE NOT EXISTS((n)-[]-())`, an unsupported pattern predicate, and has no `except` around it, so it propagated the parser error to the caller; it now filters on `WHERE NOT (n)-[:EDGE]-()`, which Ladybug accepts, and still returns a `List[str]` of node ids. **Naming the relationship type is equivalent here rather than narrowing:** the adapter's schema declares exactly one rel table (`EDGE`), so "has no `EDGE` relationship" and "has no relationship" select the same rows — a second rel table would break that equivalence. Exposure is narrow: none of the three methods is declared on `GraphDBInterface`, and none runs on the default `remember()` / `cognify()` / `recall()` path — the only in-tree caller is `remove_disconnected_chunks`, exported from `cognee.tasks.chunks` but not wired into any built-in pipeline. So this reaches you if you call these helpers directly on the engine returned by [`get_graph_engine()`](/guides/graph-engine-adapters) or drive that task from a custom pipeline; the Neo4j and Neptune adapters carry their own implementations and were never affected. **The change is the adapter file only** — it touches no test file, so the two `xfail` markers in `cognee/tests/integration/infrastructure/graph/test_kuzu_adapter.py` that name these two bugs still stand and have to be lifted separately; the third marker in that file (`get_model_independent_graph_data` format mismatch) is an unrelated bug this fix does not address. No public API signature, configuration option, or environment variable changed, and no migration ships — deploy a build containing the fix to pick it up (PR #4525). * Fixes the Neo4j graph adapter (`GRAPH_DATABASE_PROVIDER="neo4j"`) accumulating a duplicate copy of every edge on each re-cognify, and dropping stored edge properties when reading a node's connections. Two independent contract violations in `Neo4jAdapter`, both in places where the `ladybug`, `postgres`, and `neptune` adapters already behaved correctly. **`has_edges`** is the batch existence check that `graph_db_interface` declares as taking `(source_id, target_id, relationship_name)` triples and returning the subset of them that already exists in the graph. The Neo4j implementation matched endpoints on Neo4j's *internal* node identifier — `WHERE id(a) = edge.from_node AND id(b) = edge.to_node` — which is an integer that never equals the string UUID Cognee writes into the `id` property, so the query matched nothing no matter what the graph contained; it also returned raw booleans rather than tuples. Its only production caller is the cognify dedup step, `find_existing_edge_identities` in `cognee/modules/graph/utils/retrieve_existing_edges.py`, which `extract_graph_from_data` uses to subtract already-stored edges before writing the rest — so it was told that none of the extracted edges existed, every one of them was written as new, and re-cognifying the same content against a Neo4j graph added another copy of every edge each time. The query now matches on `a.id`/`b.id` with both endpoints constrained to the `__Node__` base label, like every other query in the adapter, and returns the existing subset as `(from_node, to_node, relationship_name)` string tuples — mirroring the equivalent Neptune fix in #2384. **`get_connections`** returned each edge as `{"relationship_name": ...}` and nothing else: the driver's `result.data()` flattens a relationship to `(start_props, type, end_props)` and discards its properties. Both the predecessor and successor queries now also return `properties(relation)`, which is merged into the edge dict, so stored edge properties — `edge_text`, weights, timestamps, and anything else written onto the relationship — survive the read. That loss was most visible on deletion: `legacy_delete` derives the `EdgeType` vector-row id for a chunk's `contains` edges from `edge["edge_text"]`, and with that key absent it fell back to the relationship name, computed a different id, and left the real vector rows behind. **Impact:** no public API signature, configuration option, environment variable, or migration changed, and no other graph backend is touched — Neo4j's runtime behavior now matches what the interface already documented. The fix is forward-looking only: duplicate edges already written to a Neo4j graph, and vector rows already orphaned by an earlier delete, are not cleaned up by upgrading. Rebuilding the affected dataset — `forget(dataset=..., memory_only=True)` to drop the graph and vectors while keeping the raw files, then `cognify()` — is what clears duplicates that are already stored. Unit tests in `cognee/tests/unit/infrastructure/databases/graph/test_neo4j_edge_contract.py` cover the returned tuple shape, `id`-property matching, edge-property merging, and edges that carry no properties (fixes #4187, PR #4188). * Fixes `DefaultUrlCrawler.fetch_urls` rejecting the `List[str]` its own signature advertises. The method is typed `urls: Union[str, List[str]]` and is built for many URLs — it creates one `asyncio` task per URL, bounds them with the `concurrency` semaphore, and drains them through `asyncio.as_completed` — but its input guard normalized a `str` to a one-element list and took an `else: raise ValueError(f"Invalid urls type: {type(urls)}")` branch for everything else, so `await crawler.fetch_urls(["https://a/", "https://b/"])` failed with `ValueError: Invalid urls type: ` before a single request went out. The guard now raises only for a type that is neither `str` nor `list`, so a list is normalized through and every URL in it is fetched. **The blast radius reaches past the class**, because `fetch_page_content` computes a normalized `url_list` for its log lines but forwards the caller's original `urls` to the crawler: `fetch_page_content(["https://a/", "https://b/"])` hit the same `ValueError`, and so did **every** `web_scraper_task` / `cron_web_scraper_task` call routed to the built-in crawler — that task normalizes its own `url` argument to a list before calling the helper, so even a single-URL string reached `fetch_urls` as a `list`. The built-in crawler is the backend selected when neither `TAVILY_API_KEY` nor `KEENABLE_API_KEY` is set, and also whenever `extraction_rules` or a `soup_crawler_config` is passed; the `tavily` and `keenable` backends normalize or pass lists through themselves and were never affected. **The `remember()` / `add()` URL-ingestion path was never affected either** — `save_data_item_to_storage` calls `fetch_page_content` with one URL string per data item — so the flow in [Web URL ingestion](/guides/web-url-ingestion) needs no attention. Everything else about the method is unchanged: the per-URL SSRF validation, the robots.txt check (a disallowed URL still maps to an empty string), the concurrency limit, and the per-URL error swallowing that fails one page to an empty string instead of aborting the batch. Passing a single `str` behaves exactly as before, and the result is still a `Dict[str, str]` keyed by URL. No public API signature — `fetch_urls`'s already promised list input; only its runtime behavior now matches — configuration option, or environment variable changed, and no migration is required (PR #3536). * Removes one relational-database session per ingested file from the ingestion path. Before the main ingest loop runs, `ingest_data` walks the batch once to resolve each item's `data_id`, and that pre-loop called `await ingestion.identify(...)` for every item — each of those calls opening its own `db_engine.get_async_session()`. An N-file `add()` therefore opened N sessions before any row was written, and the churn was pure overhead: every one of those queries filtered the same dataset on the same four predicates and differed only in the content hash it looked up. The pre-loop now touches the relational database not at all — it saves each file and computes its content hash, both pure CPU/storage work — and collects the batch's unique hashes into a set that a single new `identify_many(hashes, user, dataset_id)` resolves in **one** session with a `content_hash IN (…)` query. **Dedup semantics are deliberately unchanged.** `identify_many()` applies the identical `(dataset_id, content_hash, owner_id, tenant)` filter that `identify()` uses, so the two can never disagree on which row wins for a hash; it returns a `{content_hash: data_id}` map in which a hash with no existing row is simply absent rather than mapped to `None`, and it keeps the first hit per hash via `setdefault`, matching `identify()`'s `.limit(1)`. Dedup *within* one batch — two identical items in a single `add()` sharing the first minted id — and the fresh `uuid4()` on a miss both behave exactly as before. `identify()` itself still ships and is still exported from `cognee.modules.ingestion`, now alongside `identify_many`; the ingestion pre-loop no longer calls it, but its signature and behavior are untouched. Large batches are split into statements of at most **900** hashes to stay under SQLite's default `SQLITE_MAX_VARIABLE_NUMBER` of 999 — conservative but harmless on Postgres, which has no meaningful limit — and the chunking is per statement, not per connection: every chunk runs inside the same session. **One scope caveat:** items carrying an explicit pinned `data_id` (the dlt / `update()` path) take a separate branch that calls `resolve_data_id()` once per item, and that is still a round trip each. Those items get strictly cheaper regardless — they previously ran an `identify()` query whose result the pin resolution immediately discarded — but a pinned-heavy batch keeps per-item database traffic. No public API signature, configuration option, environment variable, or migration ships with this change — deploy a build containing it to pick it up (CLO-590, PR #4571). * Fixes the file-storage probe behind `/health` destroying a user file that happens to be named `health_check_test`. To prove storage is writable, `HealthChecker.check_file_storage()` writes a small temporary file and deletes it again — but the name was a fixed literal on both branches: locally, `os.path.join(data_root_directory, "health_check_test")` opened with mode `"w"` (which truncates an existing file) and then `os.remove`d, and on S3 the relative path `"health_check_test"` passed to `storage.store()` and then `storage.remove()`. Nothing checked whether that name was already taken, so a single health check silently overwrote and then deleted the object sitting there. The temporary name now carries a fresh `uuid.uuid4()` per check — `health_check_test_` — on **both** the local and the S3 branch, making a collision with real data effectively impossible. **Who was at risk:** only deployments holding an object named exactly `health_check_test` at the top level of `DATA_ROOT_DIRECTORY` (or of the configured `s3://` root) — but for those, the loss repeated on every probe, so a readiness or liveness check polling `/health` on an interval would keep deleting the file as fast as it was restored. **Nothing else about the endpoint moves:** the same four critical components are probed, `check_file_storage()` still returns `ComponentHealth(status=HEALTHY, provider="local"` / `"s3", details="Storage accessible")` on success and `UNHEALTHY` with `Storage test failed: …` on failure, `file_storage` still counts toward the `healthy` / `degraded` / `unhealthy` roll-up in `get_health_status()`, and the `HealthResponse` shape is unchanged — so existing probes and dashboards need no adjustment. A regression test (`cognee/tests/unit/api/test_health_checker.py::test_health_check_does_not_delete_existing_file`) plants a file under the old literal name at the data root and asserts the check still reports `HEALTHY` while leaving the file's contents intact. No public API signature, configuration option, environment variable, or migration ships with this fix — deploy a build containing it to pick it up (PR #4528). *** ## v1.5.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.0)** Stable release that closes the 1.5.0 line: the cut bumps the package version in `pyproject.toml` from `1.5.0.dev2` — the marker the development branch carried after the macOS 13/14 install fix in this section's highlights set it — to `1.5.0` and regenerates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved — and the cut itself introduces no functional code, public API, configuration, or environment-variable change. Most of the work shipping in this release is logged under the v1.5.0.dev1 and v1.5.0.dev2 pre-release sections below; the highlights in this section cover the work merged after the v1.5.0.dev2 cut, shipping for the first time in this release. One availability note: everything that was reachable only from a pre-release build of this line is now in a stable release — including the Slack integration's `/cognee-remember` slash command, so `pip install cognee` (no `--pre`) is enough to get it. **Upgrading:** no new Alembic revision ships in this cut, but the five revisions introduced in v1.5.0.dev1 — `b8c1d3e5f7a9`, `c5d7e9f1a3b5`, `f2b4c6d8e0a1`, `d6e8f0a2b4c6`, and `e5a7b9c1d3f4` — are part of this release, so a deployment coming from `1.4.x` must run migrations when upgrading (`cognee.run_migrations()`, or `alembic upgrade head`). The `1.5.0` number also matches the `cognee_version` tag on the `rekey_fork_document_ids` data-chain migration, but that field is audit-only — revision slugs are the only gate for the data-migration chain, so the migration is neither required nor unlocked by the version number. What the version change itself triggers is the migration runner's vector-adapter storage sync (for example, LanceDB columns), which runs after the chain on a `cognee_version` mismatch. ### Highlights * Restores automatic `ladybug` selection on macOS 13 and 14, removing the manual `ladybug==0.17.1` pre-install step that v1.5.0.dev2 (below) documented as the escape hatch. The single markerless `ladybug>=0.16.0,<=0.18.2` requirement is replaced by two lines with complementary markers: `ladybug>=0.17.0,<0.18` when `sys_platform == 'darwin'` **and** `platform_version` contains `'Darwin Kernel Version 22.'` or `'Darwin Kernel Version 23.'` (macOS 13 and 14), and `ladybug>=0.17.0,<=0.18.2` on the exact complement. Exactly one line is active on any machine, so macOS 13/14 resolve ladybug 0.17.1 from its prebuilt `macosx_13_0` wheels — no sdist build, no `no member named 'atomic_ref' in namespace 'std'` failure, nothing to pre-install — while Linux, Windows, and macOS 15+ resolve the newest ladybug exactly as before; `uv.lock` now carries both 0.17.1 and 0.18.1 under the matching markers. (A follow-up merged before the v1.5.0 cut repinned the complement line to `ladybug==0.19.0`, so the release as shipped resolves 0.19.0 outside macOS 13/14 and its `uv.lock` carries 0.17.1 and 0.19.0; the macOS 13/14 line ships exactly as described here.) **The markers use only substring operators (`in` / `not in` on `platform_version`), and that is load-bearing:** substring tests are plain string operations in every marker evaluator (packaging ≤ 24, packaging ≥ 25 as vendored in current pip, uv, poetry), whereas the ordering comparison on `platform_release` that v1.5.0.dev1 shipped is unexpressible across those generations — the invalid literal `'24.'` evaluates silently `False` on macOS under packaging ≥ 25 (ladybug skipped, `import cognee` broken), while a valid literal crashes packaging ≤ 24 on Linux kernel strings like `6.8.0-45-generic`, since both sides of an `and`/`or` are evaluated. The enumerated set does not expire: Darwin 22/23 is closed, and any future macOS falls through to the newest-ladybug line. A guard test (`cognee/tests/unit/test_ladybug_requirement.py`) re-checks the partition against every importable marker evaluator and forbids `platform_release` from returning to these markers. Separately, the lower bound moves from `0.16.0` to `0.17.0` on **both** lines, so no platform resolves ladybug 0.16.x any more. Machines on 0.17.x are on a supported on-disk storage format (format code `41`) and the `ladybug_migrate` worker upgrades it forward if they later move to 0.18+, so no data action is required. The PR also moved the development branch's version marker to `1.5.0.dev2`. No public API signature, configuration option, or environment variable changed, and no migration is required (PR #4499). * Pins the `ladybug` graph engine to exactly `0.19.0` and reshapes the batch edge write that 0.19.x crashes on — this is the follow-up the previous highlight refers to. The requirement for Linux, Windows, and macOS 15+ moves from `>=0.17.0,<=0.18.2` — a range whose lockfile resolved to 0.18.1 — to `ladybug==0.19.0`, for the storage fix that release carries; the separate macOS 13/14 requirement (`>=0.17.0,<0.18`, selected by the `'Darwin Kernel Version 22.' / '23.' in platform_version` markers) is untouched, so those machines stay on 0.17.x, which has no `macosx_15_0`-only wheel problem but also does not receive the storage fix. **0.19.0 is pinned rather than 0.19.1** because `extension.ladybugdb.com` publishes no v0.19.1 JSON extension — a 404 on every platform checked (linux\_amd64, linux\_arm64, osx\_arm64) — and 0.19.1 segfaults Cognee's DB worker mid-write in CI; 0.19.0 carries the same storage fix and its extension build exists, so do not expect 0.19.1 extension binaries and do not raise the pin past it. The pin bump alone was not enough: on 0.19.x, `LadybugAdapter.add_edges` died mid-write in every CI run, inside `add_data_points` → `add_edges`, as a SIGSEGV in the native engine that surfaces through the subprocess worker as `Subprocess exited unexpectedly (exit code -11)`. The cause is query shape, not data — 0.19.0 introduced a row-driven primary-key lookup for `MATCH` (`LadybugDB/ladybug#722`) that the adapter's two separate `MATCH` clauses for the edge's endpoints land on. Both endpoints are now bound in **one** comma-separated clause (`MATCH (from:Node {id: edge.from_id}), (to:Node {id: edge.to_id})`), which keeps the primary-key index seeks, costs nothing — \~80s to write 20,000 edges on 0.17.1, 0.18.2, and 0.19.0 alike, matching the old syntax on the versions where it worked — and writes an identical graph, so this is an internal query-generation change with no interface or behavior contract attached. Alongside it, `cognee_db_workers/ladybug_migrate.py` registers on-disk storage code `43` as `0.19.0`, so a store written by 0.19.0 is recognized instead of failing with `Could not map version_code to proper Ladybug version.`; stores left at code `41` on macOS 13/14 are still upgraded forward when those machines later move to a newer line. Every pinned version's storage code must exist in that mapping, so bumping the pin stays a two-file change. **Upgrade action:** re-sync your environment against the updated `pyproject.toml` / `uv.lock` so the installed engine matches the pin — no public API signature, configuration option, or environment variable changed, and no Alembic migration ships, so this is a reinstall, not a data action (COG-6185, PR #4512). * Fixes graph database opens failing on Windows with `RuntimeError: Could not find lbug C API shared library.` — the failure the 0.19.0 pin above exposes, since 0.18.x Windows wheels were self-contained. ladybug's Windows wheels stopped vendoring OpenSSL in 0.19.0 while its native extension still imports `libssl-3-x64.dll` and `libcrypto-3-x64.dll`, so `import ladybug._lbug` raised `ImportError`, ladybug silently fell back to its C-API backend, and that backend's shared library ships in no wheel — the failure surfaced only at the first database open. Cognee now supplies those two DLLs itself from CPython's own OpenSSL 3 (`cognee_db_workers/_windows_openssl.py`): it copies them into a per-interpreter cache directory under the names ladybug's import table asks for and registers that directory with `os.add_dll_directory()`, at package import time in the parent process and in every spawned DB worker. No configuration is required, and it is a no-op off Windows and on installs whose ladybug wheel vendors OpenSSL itself. It cannot help interpreters that ship no OpenSSL 3 — CPython 3.10 on Windows links OpenSSL 1.1, and embedded distributions ship no `DLLs` directory — so use Python 3.11 or newer on Windows (COG-6185, PR #4513). * Fixes the `rekey_fork_document_ids` data migration never completing on a large graph, which left the affected dataset permanently failed with `migration_last_error: TimeoutError: Subprocess call exceeded 300.0s deadline`. On a dataset of \~100k nodes and \~295k edges the re-key burned three consecutive 300-second worker-subprocess deadlines inside `get_edge_delete_data` and never got past it. Three things were fixed on the default Ladybug graph backend and the migrations that drive it. **First, edge-identity and node-id queries are now chunked index seeks.** `get_edge_delete_data`, `delete_edge_triples`, `get_node_delete_data`, and the node/edge provenance read and write helpers in the Ladybug adapter each issued a single statement of the scan-planned form `MATCH (a:Node)-[r:EDGE]->(b:Node) WHERE a.id = e.s AND b.id = e.t …` (or `MATCH (n:Node) WHERE n.id IN $ids`), which plans as a cartesian scan over the whole node/edge set. They now match on inline property maps — `MATCH (a:Node {id: e.s})-[r:EDGE]->(b:Node {id: e.t})` and `UNWIND $ids AS nid MATCH (n:Node {id: nid})` — so the planner uses primary-key seeks, and each call is split into fixed-size chunks, the same treatment `add_nodes` and `add_edges` already had. **Second, `_migrate_graph` no longer snapshots the whole graph's provenance.** It snapshotted all edges even though only remapped edges and at-risk survivors (edges incident to a node whose neighbor was remapped) ever have their snapshot read back; it now partitions the edge list first and snapshots only those. **Third, provenance restore and move are batched instead of per-artifact.** Both loops previously did one read + write + checkpoint subprocess round-trip per node or per edge — the checkpoint page churn alone can exhaust the Ladybug store's size cap (the `kuzu_max_db_size` setting; Neo4j and Postgres graph stores have no such cap). `_migrate_graph`'s restore now groups artifacts by provenance profile (identical `source_ref_keys` and run refs, which a dataset's subgraph overwhelmingly shares) and `_rekey_graph_provenance` groups them by pipeline run id, attaching each group in one call; because `attach_node_source_refs` / `attach_edge_source_refs` apply the transition per artifact, the batched calls are state-identical to the loops they replace. On the fixture above, a re-key that never finished now completes in about 28 minutes end to end with every call inside the default 300-second worker deadline and the default 32 GB store cap, and the unaffected "keeper" dataset's migration drops from \~12 s to \~7 s. **Operational notes (Ladybug only):** a bulk re-key still runs for tens of minutes, so give the store headroom under `kuzu_max_db_size` and do not kill a worker mid-checkpoint — an interrupted run can leave `.lbug.shadow` / `.wal.checkpoint` recovery files that block the next open, and repeated interrupted runs can exhaust the cap even at a small on-disk size. **Known limitation:** a forked dataset can come out of the graph re-key with edge rows whose endpoint no longer resolves (measured \~17.5k on the fixture — 12,989 empty-target and 4,572 empty-source), which makes the formatted-graph endpoint answer `500` on response validation for that dataset; keeper datasets are unaffected. That is pre-existing `_migrate_graph` edge handling that was unreachable at this scale before the re-key could complete, not something this fix introduces, and it is tracked separately. No public API signature, configuration option, or environment variable changed, and no new migration ships — the migration chain is the same, it now finishes (COG-6112, PR #4498). * Fixes OTLP log export blocking the thread that emits each log line, and makes `COGNEE_TRACING_ENABLED=false` an actual off switch. The [OpenTelemetry log bridge](/integrations/opentelemetry-tracing#logs) attached its OTLP log exporters — both the gRPC and the HTTP path — through a `SimpleLogRecordProcessor`, which exports every log record as its own synchronous network round trip on the emitting thread. Spans were already batched and metrics already exported periodically; logs were not, so with an OTLP endpoint configured (including one derived from `LANGFUSE_*` keys) each of the many log lines a `cognify()` run emits became a blocking call to the collector, with no error to point at. Both OTLP log paths now use `BatchLogRecordProcessor`, so records are buffered and exported on a background thread; console output (`console_output=True`) stays per-record as before. **Because log records are now buffered, call `disable_tracing()` before the process exits** — it shuts the logger provider down and force-flushes the pending batch, and that flush costs a short delay where shutdown previously had nothing to flush. Separately, an explicit off value for `COGNEE_TRACING_ENABLED` — `false`, `0`, or `no` — is now authoritative in both places it was previously overridden: config derivation from `LANGFUSE_*` keys ended with an unconditional enable that overwrote it, and `is_tracing_enabled()` combined the config field and the env var with an `or` behind a module-level latch, so the variable could switch tracing on but structurally never off. Set before tracing initializes, it now disables tracing and all OTLP traffic even with Langfuse keys present; flipped off after tracing was already enabled in the process, it stops new spans and metric recordings, though already-attached exporters — the log bridge and the periodic metric reader — keep running until `disable_tracing()` shuts them down. **An unset variable is unchanged and is not a veto** — `LANGFUSE_*` keys alone still auto-enable tracing, so existing key-only setups keep working. No public API signature, configuration option, or migration changed; deploy the fix to pick it up, and upgrade if you run with Langfuse or another OTLP backend (PR #4507). *** ## v1.5.0.dev2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.0.dev2)** Development pre-release that bumps the package version from `1.5.0.dev1` to `1.5.0.dev2` and updates `uv.lock` to match. No new Alembic revision ships in this cut, so no migration is required on upgrade — but if you are coming from `1.4.x`, the v1.5.0.dev1 migrations below still apply. One fix ships beyond the version bump. ### Highlights * Makes `ladybug` an unconditional dependency, fixing pip installs of cognee on macOS failing at import with `ModuleNotFoundError: No module named 'ladybug'`. The macOS-version environment marker that v1.5.0.dev1 shipped on the requirement (`sys_platform != 'darwin' or platform_release >= '24.'`, from the macOS 13/14 install fix below) cannot be expressed correctly across installer generations: `packaging <= 24.x` needs the invalid `'24.'` literal to hit PEP 508's string-comparison fallback (and crashes on valid version literals against Linux kernel strings like `6.8.0-45-generic`), while `packaging >= 25` — vendored in current pip — dropped that fallback, evaluated the marker `False` on macOS, and silently skipped installing `ladybug`. The marker is now gone entirely, and a guard test (`cognee/tests/unit/test_ladybug_requirement.py`) pins the requirement markerless so it cannot quietly return. The trade-off: on macOS 13/14, where ladybug 0.18.x has no wheel and the sdist build fails, installation now fails loudly at install time instead of skipping cleanly — the escape hatch is pre-installing `ladybug==0.17.1` (its `macosx_13_0` wheels satisfy the unchanged `>=0.16.0,<=0.18.2` range) before installing cognee. *** ## v1.5.0.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.5.0.dev1)** Development pre-release that opens the 1.5.0 line: the package version moves from `1.4.2` to `1.5.0.dev1`, with `uv.lock` updated to match. A `v1.5.0.dev0` tag was cut first but never reached PyPI — the release pipeline's publish gate rejected the build's `Metadata-Version: 2.5` metadata — so v1.5.0.dev1, cut together with the gate fix, is the first published build of the line. Unlike recent marker-only bumps, this cut moves real dependencies in the lockfile (among others, dropping the distributed/Modal execution stack and bumping `enola` — see the entries below), and five new Alembic revisions ship: `b8c1d3e5f7a9` (adds the `provenance_entries` table), `c5d7e9f1a3b5` (adds `dataset_id` to `data`), `f2b4c6d8e0a1` (adds `system_metadata` to `data`), `d6e8f0a2b4c6` (backfills dataset-scoped data rows and drops the `dataset_data` link table), and `e5a7b9c1d3f4` (adds `dataset_id` to queries and results). Existing deployments must run migrations when upgrading (`cognee.run_migrations()`, or `alembic upgrade head`). The entries below are the work merged to the development branch since the v1.4.2 cut. ### Highlights * Adds the `/cognee-remember ` slash command to the self-hosted [Slack integration](/integrations/slack-integration), which previously answered *"Command `/cognee-remember` is not yet supported."* — saving was reachable only through the **Remember this** message shortcut, which by definition can only capture what Slack already had on screen. The command writes free text (a decision from a call, a conclusion recorded afterwards) through a new `remember_note` path into the same `slack` dataset and `node_set` as the shortcut, so both arrive as one Slack-origin body of memory rather than two the graph cannot relate; the stored text reads *In Slack, @user noted: …*, deliberately distinct from the shortcut's *In #channel, @user said: …*, so a note never puts words in someone's mouth. It acks inside Slack's 3-second window with an ephemeral `💾 Remembering: __` (the ack preview is truncated at 200 characters; the stored note is not) and performs the save detached, confirming afterwards via `response_url` with `✅ Remembered — it'll be recallable shortly.` or, on failure, `Could not save that. Please try again.` — a detached save reports its own failure as a message instead of raising, since no caller is left to catch it. The detached split is required rather than cosmetic: `cognee.remember` resolves — and on a first save creates — the target dataset before returning even with `run_in_background=True`. **Reinstall required:** Slack does not grant a new slash command to an already-installed app, so an existing install must be reinstalled to the workspace (the tracked `slack-app-manifest.yml` gains the command and a note saying so); until then the command is refused client-side and nothing reaches your backend, which looks like a broken server. Every reply is ephemeral, matching `/cognee-ask` (PR #4479). * Fixes the Slack integration telling an unlinked member to paste an API key, and clarifies that such a member is **refused rather than attributed to the installer**. `/cognee-ask` and the **Remember this** shortcut both replied ``Link your own Cognee account first: `/cognee-link ` (create a key from your Cognee account's API Keys settings)`` — an argument that never existed, left over from before linking became a magic link. The wording now lives once in `handle_slack_link.py` as `NOT_LINKED_MESSAGE` (*"I don't know which Cognee account you are yet. Run `/cognee-link` to connect yours, then try again."*) and is shared by all three entry points, so they cannot drift again. The underlying rule is unchanged but was easy to misread: `resolve_owner_user_id` returns the member's own linked account, falls back to the workspace credential's owner **only for the Slack user who completed the OAuth Connect**, and otherwise returns `None` — which every handler turns into that refusal *before* any read or write, so a note is never saved into someone else's memory because its author never linked. Empty-input replies now carry an example (e.g. `` `/cognee-ask why did we choose Neon for v2?` ``) instead of `Usage: ...`, and `/cognee-ask`'s ack changed from `Searching your memory for: "..."…` to `🔎 Recalling: _..._`, matching the cloud integration's wording (PR #4479). * Fixes `GET /api/v1/datasets/{dataset_id}/data` returning `500` for any dataset that actually holds data, which broke the dataset detail view (a dataset with no data rows returned an empty list before reaching the failure). The handler built each row with `dict(**jsonable_encoder(data), dataset_id=dataset_id)`, and since `Data` gained its own `dataset_id` column (the `c5d7e9f1a3b5` revision in this same cut) the encoded row already carried that key, so the keyword form raised `TypeError: dict() got multiple values for keyword argument 'dataset_id'`. A dict literal now resolves the duplicate, with the requested dataset id still winning — the column is nullable, so the row's own value cannot be relied on. The endpoint's path, parameters, and response shape are unchanged, so clients need no update beyond deploying the fix; no configuration option, environment variable, or migration ships with it (PR #4479). * Makes the length cap on dlt `ColumnValue` nodes opt-in. Cell values selected for value nodes — the columns picked with the `column_value_columns` kwarg on `add()` (`remember()` does not accept it), or `DLT_COLUMN_VALUE_COLUMNS` — were silently dropped when the value ran longer than a hardcoded 256 characters. That constant is gone, replaced by the `dlt_max_column_value_length` ingestion setting (env `DLT_MAX_COLUMN_VALUE_LENGTH`), which defaults to `0`: no cap, so every selected value becomes a node regardless of length. A positive value restores the old shape of the behavior, skipping — not truncating — selected cells longer than the bound. **Impact:** only runs that actually select column values are affected, since the selection is empty by default, but those runs can now emit more `ColumnValue` nodes than before, and each unique value costs one embedding. Set `DLT_MAX_COLUMN_VALUE_LENGTH` when ingesting free-text-heavy columns or selecting columns with `"*"`; `256` reproduces the previous cap exactly. The setting is environment/config-only — there is no per-call kwarg for it — and no migration is required (PR #4469). * Fixes graph writes against the default Ladybug store failing with `Catalog exception: function TiMESTAMP does not exist.`, which broke `remember()` and `cognify()`. `LadybugAdapter` (`GRAPH_DATABASE_PROVIDER="ladybug"`, the default backend) generates its Cypher with lowercase `timestamp(...)` casts around the `created_at` / `updated_at` parameters, and ladybug (Kuzu) **0.17.1** does not resolve that name as a function — it rejects the statement with the mixed-case spelling shown above, its own rendering of the unresolved name. All 13 cast sites now emit uppercase `TIMESTAMP(...)`, covering every write path that stamps a timestamp: the single-node create in `add_node`, the single-edge upsert helper `_edge_query_and_params` behind `add_edge`, the batched `add_nodes` and `add_edges` statements, and the three property-update executors (`_execute_node_feedback_updates`, `_execute_node_truth_state_updates`, `_execute_edge_feedback_updates`). The batched node and edge writes are the ones ingestion drives, so the failure surfaced as a broken cognify rather than as an isolated adapter error; a `remember` → `recall` round trip through cognee-mcp completes again with `created_at` / `updated_at` written on both nodes and edges. **Who hit this:** the failure was observed on ladybug 0.17.1 — exactly the version macOS 13/14 installs of this cut landed on, whether resolved automatically by the marker split in the install fix below or pre-installed as the escape hatch the v1.5.0.dev2 entry above recommends — because the `>=0.16.0,<=0.18.2` requirement these builds carried otherwise resolved to 0.18.x, whose macOS wheels are tagged `macosx_15_0`; so 0.17.1 installs were the most likely to run into it. (v1.5.0 has since replaced that range, pinning `>=0.17.0,<0.18` on macOS 13/14 and `0.19.0` everywhere else.) The uppercase spelling is what the fix standardizes on across the adapter; no lowercase cast remains. **Impact:** no public API signature, configuration option, environment variable, or migration changed, and no data action is required — deploy a build containing this fix to pick it up. Rows that failed to write were never persisted, so there is nothing to clean up; re-run any ingestion that errored out this way (fixes #4474, PR #4475). * Fixes `brute_force_triplet_search()` mutating a `collections` list passed in by the caller. The function appends `"EdgeType_relationship_name"` so the edge collection is always searched, but that append landed on the caller's own list object — so the entry leaked back out and stayed there. `TripletSearchContextProvider` keeps the list it was constructed with as `self.collections` and hands the same object to one search per entity, so a configured list was silently and permanently extended after the first search. The provided list is now copied before the edge collection is appended; the `collections=None` branch already built a fresh default list and is unchanged, as is the default set itself (`Entity_name`, `TextSummary_text`, `EntityType_name`, `DocumentChunk_text`, `DltRow_text`). Retrieval results do not change — the same collections are searched, edge collection included. Nothing changes for `search()` or any other public API, since the in-repo caller (`GraphCompletionRetriever`) rebuilds its collection list on every call; only code that calls `brute_force_triplet_search()` or constructs a triplet-search context provider directly with its own list sees a difference, and code that relied on finding the appended entry in that list afterwards must now add `"EdgeType_relationship_name"` itself. No signature, configuration option, environment variable, or migration ships with this fix (SDK-275, fixes #3481, PR #4471). * Raises the per-attempt embedding deadline in `LiteLLMEmbeddingEngine` from 30 to 300 seconds. The `asyncio.wait_for` guard around `litellm.aembedding()` is measured per attempt and starts *before* any network I/O, so waiting for a free connection in the local HTTP pool and event-loop scheduling delays counted against the 30-second budget — under high cognify concurrency healthy requests were cancelled for standing in Cognee's own queue (an observed 421-item cognify against OpenAI `text-embedding-3-large` produced 3,336 timeout retries and a failed run after 64 minutes, against only 2 genuine provider rate-limit errors). The new value matches the deadline `OpenAICompatibleEmbeddingEngine` already used, so the two engines no longer differ. **Behavior change:** the retry window is unchanged at 128 seconds and is evaluated *between* attempts, so a request that genuinely hangs now consumes the full 300 seconds on its first attempt and is not retried afterwards — one hung request can block its task for up to 5 minutes, where the shorter deadline left room for retries inside the window. Requests that fail fast (connection refused, rate limits, `5xx`) still retry through the full 128-second window as before. Bounding embedding concurrency remains the other half of avoiding long stalls. No public API signature, configuration option, or environment variable changed, and no migration is required; the timeout stays hardcoded and is still only overridable by subclassing the engine (PR #4485). * Adds per-file `labels` and `external_metadata` multipart form fields to `POST /api/v1/add` and `POST /api/v1/remember` — the HTTP equivalent of the Python SDK's `DataItem(label=..., external_metadata=...)`, which was previously the only way to attach either to an upload. Each field is sent as **one JSON part** whose entries pair positionally with the uploaded files (the Nth entry applies to the Nth file): `labels` is a JSON array of strings with `""` skipping a file, and `external_metadata` is a JSON array of objects with `null` or `{}` skipping a file. A single part is used instead of a repeated form field because Swagger UI collapses repeated multipart array fields into one comma-joined part, which would silently corrupt per-file pairing; for the same reason the comma-separated form (`finance,people,`) is accepted equivalently for `labels` — so a label can contain a comma only via a client that sends real JSON — while `external_metadata` has no such fallback and must always be valid JSON. Validation returns `400` when an entry count does not match the file count (a partial list is ambiguous), when a JSON `labels` array contains non-string entries, when `external_metadata` is malformed or contains the reserved key `node_set` (ingestion writes the request's `node_set` into the stored dict after merging, so it would be silently overwritten — use the `node_set` form field instead), and, on `remember`, when either field is combined with `session_id` or `content_type`, since those paths never create the `Data` records the values are stored on. On merge, your keys win over loader-derived metadata. `GET /api/v1/datasets/{dataset_id}/data` now returns the stored values as `label` and `externalMetadata` — previously they were persisted but unreadable over HTTP — and a re-ingest that omits the label now leaves a previously stored label unchanged instead of clearing it (a provided label still replaces). Separately, this PR raised the default `data_per_batch` — the cap on data items processed concurrently within one dataset pipeline run — from `20` to `2000` for `add()`, `cognify()`, and the cognify endpoint payload, but that raise was reverted before this release was cut (PR #4490), so the shipped default remains `20` throughout. No migration, environment variable, or new permission is involved (PR #4444). * Adds `GET /api/v1/permissions/principals/{principal_id}/datasets`, which lists the datasets a principal holds a permission on — so a client can ask "which datasets does this team have?" without enumerating datasets and testing each one. The principal may be a user, a role, or a tenant, and the optional `permission_name` query parameter selects the permission to list, defaulting to `read` (the other accepted values are `write`, `delete`, and `share`). The response is a JSON list of dataset objects. Visibility is tenant-scoped: the requester's tenant is read off the requester rather than taken as a parameter, so a caller cannot name a different one, and the returned list is always filtered to that tenant. Who may ask depends on the principal's type — a user may ask about themselves, or about any user if they are the tenant owner or hold user-management permission; a role is visible to its members, or to the tenant owner or any user-management holder in the same tenant; and a tenant is visible only to callers currently in it. A role id from another tenant returns `404` rather than that tenant's datasets, matching the cross-tenant scoping already used by the role-members endpoint, and a caller who may not ask about the principal receives `403`. The same check is available to SDK callers as `authorized_get_principal_datasets(principal_id, permission_name, requester_id)`, exported from `cognee.modules.users.permissions.methods`; the pre-existing `get_principal_datasets` performs no authorization and is unchanged. No new model, permission type, environment variable, or migration ships with this change (COG-6158, PR #4447). * Fixes installing Cognee from a repository checkout hard-failing on macOS 13 and 14. `pyproject.toml` pinned `ladybug>=0.16.0,<=0.18.2`, so the resolver always selected 0.18.x — but every ladybug 0.18.x macOS wheel is tagged `macosx_15_0`, so on macOS 14 and older the installer fell back to the sdist, whose CMake build uses C++20 `std::atomic_ref`; the libc++ shipped with those macOS releases does not provide it, and the build died with `no member named 'atomic_ref' in namespace 'std'`. That broke `pip install `, `uv pip install `, and the `uv sync` contributor path in `CONTRIBUTING.md`. The pin is now split across two environment markers — `sys_platform != 'darwin' or platform_release >= '24.'` keeps `>=0.16.0,<=0.18.2`, while `sys_platform == 'darwin' and platform_release < '24.'` takes `>=0.16.0,<0.18` — so macOS 13/14 (Darwin release below 24, i.e. below macOS 15) resolve ladybug 0.17.1 from its prebuilt `macosx_13_0` wheel, and Linux, Windows, and macOS 15+ stay on 0.18.x exactly as before; `uv.lock` now carries both ladybug versions under the matching markers. Machines held at 0.17.x are on a supported on-disk storage format (format code `41` in `cognee_db_workers/ladybug_migrate.py`), and the migrate worker upgrades the format if they later move to 0.18.x, so no data action is required. **If you edit this dependency, keep the trailing dot in `'24.'`:** `packaging` evaluates both sides of an `and`/`or` marker, and version-comparing `platform_release` raises `InvalidVersion` on Linux kernel strings such as `6.8.0-45-generic`, so a plain `'24'` would break installs on Ubuntu/Debian — the trailing dot is not a valid PEP 440 version and therefore forces PEP 508's string comparison, which evaluates correctly on every platform (a comment in `pyproject.toml` records this). No public API signature, configuration option, or environment variable changed, and no migration is required (COG-5974, PR #4228). * Adds `cognee.validate()`, a read-only dataset integrity checker, and the `GET /api/v1/validate` endpoint in front of it. It cross-checks a dataset's graph and vector stores for three problems: **orphaned edges** (an edge endpoint id that is not in the node set — `error`), **identity-id mismatches** (an `Entity` / `EntityType` node whose id is not the one `Type.id_for(name)` derives from its own properties, so a correctly-derived duplicate could coexist unnoticed — `warning`), and **missing vector entries** (an `Entity` or `DocumentChunk` node with no point in its `Entity_name` / `DocumentChunk_text` collection, meaning it exists in the graph but is unreachable by embedding-based search — `error`). It returns a `ValidationReport` with `status` (`healthy` / `degraded` / `unhealthy`, derived from severities: any error → unhealthy, otherwise any warning → degraded), `summary` (`graph_nodes`, `graph_edges`, `node_type_distribution`), and a list of typed `issues`. `validate`, `ValidationReport`, `ValidationIssue`, and `ValidationStatus` are importable from the `cognee` top level. The checker is backend-agnostic — it runs entirely through `GraphDBInterface.get_graph_data()` and `VectorDBInterface.retrieve()`, so no adapter-specific code is involved and every supported graph/vector backend is covered — and never writes to any store, so it is safe against production data; cost is a full graph read plus one batched vector retrieve per collection, and there is no sampling or limit parameter, so it scales with graph size. `dataset` defaults to `main_dataset` and resolves to the datasets the caller can read, with the **first** determining which graph is checked (one graph per call, as with `report()`). **The HTTP endpoint answers an `unhealthy` report with a `503` status code**, `200` for `healthy` / `degraded`, and `500` with `{"status": "error", "reason": ...}` if the check itself raises — so a health probe or a client that raises on non-2xx responses needs to read the body to tell a data-integrity finding from a transport failure. The `dataset` query parameter is repeatable and requires an authenticated user. No migration, configuration option, environment variable, adapter change, or CLI command was added; recommended after `cognify()`, large imports, and migrations (PR #4356). * Fixes re-ingesting a skill creating a duplicate Skill node instead of updating the existing one, and adds `DELETE /api/v1/skills/{skill_id}` so a skill can be removed. A skill's id is a deterministic hash of its dataset id, its source directory and its name, and the storage layer already upserts by node id — but both the inline-text and the file-upload skills paths in `remember()` materialized the `SKILL.md` into a fresh `TemporaryDirectory()` on every call, so the source directory (and therefore the "deterministic" id) differed on every request and each `POST /api/v1/skills` or `POST /api/v1/remember` with `content_type=skills` added another copy. Materialized skills are now staged under a stable per-dataset root (`cognee-skills-` in the system temp directory, with the skill's slug as a subfolder), which keeps the source directory fixed per dataset and skill name, so re-ingesting the same skill name (the `skill_name` field on the inline path; the uploaded `SKILL.md`'s parent-folder name on the upload path) into the same dataset upserts the same node with refreshed content and embedding. The same name in a *different* dataset still resolves to a distinct id, so attaching one skill to several datasets keeps working, and path-based (folder) ingestion is unchanged. The new `DELETE /api/v1/skills/{skill_id}` takes a required `dataset_id` query parameter and requires `delete` permission on that dataset — a separate grant from the `write` that ingestion needs and the `read` the list/fetch routes need — returning `200` with `{"status": "deleted", ...}`, `403` for a dataset you cannot delete in, `404` for an unknown skill id, and `409` when the deletion fails. It is a hard delete of the graph node, its edges and its `Skill_search_text` embedding (the embedding cleanup is best-effort — a vector-store failure is logged without failing the delete) rather than an `is_active=False` soft delete, because a hidden leftover node would be silently resurrected by a later re-ingest now that ids are stable. **Impact:** no migration runs, so Skill nodes duplicated by the old behavior stay in the graph until you remove them — delete the extra copies with the new endpoint (a subsequent re-ingest will then keep updating one node). Deletion is not recoverable; re-ingest the `SKILL.md` to restore a skill (PR #4290). * Applies the session cache's sliding TTL lazily on the SQL backends (`sqlite` and `postgres`), removing a per-write rewrite of the whole session. The sliding TTL — Redis `EXPIRE`-on-write parity, which pushes a session's expiry forward on every write — was translated to SQL as an `UPDATE` over *all* of that session's rows at every write path, so the cost of a write grew with the length of the session and total write cost grew quadratically; on the default SQLite backend this produced extreme WAL write amplification (a reported 64.2 GB `cache.db-wal` against a 200 MB `cache.db`, growing 5–39 GB/hour under steady agent traffic). `log_usage` had the same shape one scope wider, re-stamping every usage-log row for the user on each logged call across the 12 decorated API routes (including `POST /api/v1/recall` and `POST /api/v1/search`) and the MCP tools that share the decorator. The `UPDATE` now skips rows whose recorded expiry lags the new target by less than 5% of the TTL, so each row is rewritten at most once per slack window and a write costs roughly its own bytes. **Behavior change:** session entries and usage logs now expire between 0.95 × `SESSION_TTL_SECONDS` and 1.0 × `SESSION_TTL_SECONDS` after the last write instead of exactly at the TTL — with the 7-day default, up to \~8.4 hours earlier — so treat the TTL as a lower bound with a small slack window; rows written while the TTL was disabled are still stamped on the next write, and Redis keeps exact `EXPIRE` semantics. Setting `SESSION_TTL_SECONDS=0` disables expiry entirely and skips the sliding-TTL writes as well, which is the lightest-I/O setting for long-lived sessions on SQLite. No public API signature, configuration option, or environment variable changed, and no migration is required (COG-6106, PR #4405). * **Breaking: removes multiprocess and distributed (Modal) execution support.** The `COGNEE_DISTRIBUTED` environment variable, the `cognee[distributed]` install extra, and the Modal execution path are no longer supported, and running multiple Cognee processes against the same stores is not a supported configuration — the embedded defaults (Ladybug/Kuzu graph, SQLite, LanceDB) are file-based with process-local locks, so a second process opening the same files can see stale or empty data. Cognee runs as a single process; when more than one process or agent needs the same memory, route all access through a single Cognee service backed by external stores (Neo4j for the graph, Postgres for the relational store, PGVector for vectors). The distributed-execution guide and Modal deployment page have been removed, and the deployment, caching, and configuration docs now consistently describe single-process operation (COG-6050). * Fixes Graphiti temporal-awareness indexing embedding the wrong text into the `GraphitiNode_name` and `GraphitiNode_summary` vector collections, so similarity search against those fields matched on the node's `content` instead of on the field the collection is named for. `index_and_transform_graphiti_nodes_and_edges()` builds one indexable point per entry in `GraphitiNode.metadata["index_fields"]` (`name`, `summary`, `content`) by calling `model_copy()` and then narrowing the copy's `metadata["index_fields"]` to the single field being indexed. Pydantic v2's `model_copy()` is a shallow copy and `DataPoint` does not override it, so every copy taken from a given node shared that node's one `metadata` dict: each field's assignment overwrote the previous ones, and by the time the points were flushed all of that node's copies carried the *last* indexed field — `content` where set, otherwise `summary`. The vector adapters then resolve the text to embed from `metadata["index_fields"][0]` rather than from the `index_property_name` they were called with — directly in `LanceDBAdapter.index_data_points`, and via `DataPoint.get_embeddable_data` on the PGVector and Turso adapters — so that overwritten field name decided what was actually embedded. Each copy now gets its own `metadata` dict before the assignment. The contamination was bounded to copies of a single node within one indexing pass: Pydantic v2 gives each instance its own copy of a mutable field default, so the class-level `GraphitiNode.metadata` default was never corrupted and later nodes and other pipelines were unaffected. `EdgeType` declares exactly one index field, so edge indexing made a single copy per edge type and was never affected. **Operator impact:** if you indexed a Graphiti graph before this fix, re-run `index_and_transform_graphiti_nodes_and_edges()` to correct the affected collections — nodes whose only non-`None` indexable field was the one being indexed were already correct. On the default LanceDB backend the re-run is enough, because its `merge_insert` upsert rewrites the stored vector along with the payload; on PGVector and Turso the `ON CONFLICT (id)` clause updates only the row's `payload` and leaves the existing vector untouched, so delete the stale rows from `GraphitiNode_name` and `GraphitiNode_summary` first (`delete_data_points`) or the wrong embeddings will survive the re-run. No schema change, no migration, and no public API signature, configuration option, or environment variable changed (fixes #3292, PR #3580). * Bumps the pinned `enola` release used by code-graph extraction from `0.1.34` to `0.3.13` and ingests the explainer findings that 0.3.x writes to `insights.json` as a new fact kind. Each finding becomes a synthetic fact of kind `insight`, mapped to a new `CodeInsight` DataPoint whose `name` is the finding's title, whose `description` is the explainer's own prose (used verbatim instead of the generic `kind: k=v` property summary other fact kinds get), and whose `fact_properties` carry `source` (the explainer that produced it), `confidence`, `description`, and `suggested_actions`, each when the finding provides it. Each piece of evidence the finding cites — a symbol, fact, or file — becomes an `evidences` edge from the insight to that node when the target resolves to a fact in the snapshot, so a finding is linked to the code it is about. Findings come from enola's deterministic, LLM-free explainers — hotspots, god-class, dependency-depth, cycles, layers, exported-surface, complexity-outliers, and others. `SearchType.CODE` picks these up with no API change: `insight` is now a valid `kind`/`kinds` value, `CodeInsight` a valid `node_types` value, and `evidences` a usable `relation_types` value. Installation also gains a `darwin-amd64` (Intel macOS) build alongside the existing `darwin-arm64`, `linux-amd64`, `linux-arm64`, and `windows-amd64` ones, and archive extraction handles the 0.3.x tarball layout, which ships `LICENSE` and `NOTICE` next to the binary — the extractor still refuses any archive that does not contain exactly one top-level `enola*` file, or whose members contain a path separator or start with a dot. **Upgrade impact:** the `facts.jsonl` relation shape is unchanged, so existing consumers of the code graph keep working, and a snapshot without `insights.json` is skipped silently while one that cannot be parsed is logged and ignored, rather than failing extraction, so 0.1.x snapshots still extract; re-extracting a repo grows its graph by the number of findings enola reports (129 on the cognee repo itself). The auto-installed binary is version-scoped by filename, so an existing `enola-0.1.34-*` install is not reused — the first extraction after upgrading downloads `0.3.13` unless `ENOLA_PATH` points at your own binary, which still wins over the auto-install (COG-6113, PR #4404). * Fixes the Amazon Neptune graph adapter (`NeptuneGraphDB`, used by `GRAPH_DATABASE_PROVIDER="neptune"`) never releasing its AWS client when the graph engine is dropped from Cognee's engine cache. Graph engines are created through `_create_graph_engine`, which is wrapped in `closing_lru_cache`; that cache closes each entry once it has left the cache and the last caller handle has been released. Its close step starts by checking whether the cached value has a `close` attribute and returns immediately when it does not — and `NeptuneGraphDB` implemented no `close()`, nor does `GraphDBInterface` declare one — so for Neptune the close was silently skipped and the `langchain_aws` `NeptuneAnalyticsGraph` together with its underlying boto3 client was discarded without being closed, leaving the client's pooled connections to be reclaimed non-deterministically by the interpreter instead of released at eviction. The adapter now implements `async close()`, which closes the wrapped boto3 client (`self._client.client`) when it exposes a `close()` method and then clears `self._client`; the client and attribute checks are defensive, so the call is a safe no-op when the client was never initialized and when close runs more than once, as the cache's idempotency requirement expects. The evictions this affects are the ones any provider sees — capacity eviction once more distinct graph configurations are in play than `DATABASE_MAX_LRU_CACHE_SIZE` (default `6`), explicit eviction when a dataset is deleted, and the `cache_clear` behind prune — so the benefit is to long-running services that cycle graph configurations, not to a short script that creates one engine and exits; Neptune is not the per-dataset handler (`GRAPH_DATASET_DATABASE_HANDLER` defaults to `ladybug`), so this is not about per-dataset database isolation. Eviction-time close remains deferred until the last engine handle drops and a failing close is still logged and swallowed rather than raised. The separate `neptune_analytics` provider is served by a different hybrid adapter and is unchanged. No public API signature, configuration option, or environment variable changed and no migration is required; callers continue to obtain engines through `get_graph_engine()` and are not expected to construct or close adapters themselves (PR #3244). * Adds an extraction-oriented image transcription prompt and an optional local OCR pass to `ImageLoader` (`cognee/infrastructure/loaders/core/image_loader.py`). **Default behavior changes:** images are now transcribed with a new prompt template (`transcribe_image_prompt.txt`) that asks for the entities shown and their attributes, the relationships between them, all visible text/numbers/dates/labels transcribed verbatim, and structured content (tables as rows, charts as series and data points, diagrams as element connections), under a 1024-token completion cap — where previously every image got the hardcoded `"What's in this image?"` caption prompt and a 300-token cap. Images therefore produce longer, denser text and cost more tokens per image than before; set `IMAGE_EXTRACTION_ENABLED="false"` to restore the previous caption prompt and 300-token cap. Five environment variables are new: `IMAGE_EXTRACTION_ENABLED` (default `"true"`), `IMAGE_TRANSCRIPTION_PROMPT_PATH` (default `"transcribe_image_prompt.txt"`; a file name resolves inside `cognee/infrastructure/llm/prompts`, an absolute path is loaded from its own directory), `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS` (default `1024`), `IMAGE_TRANSCRIPTION_REASONING_EFFORT` (default `"low"`; `minimal`/`low`/`medium`/`high`, dropped for models without reasoning support), and `IMAGE_OCR_ENABLED` (default `"false"`). With `IMAGE_OCR_ENABLED="true"` and the new `cognee[rapidocr]` extra installed (`rapidocr-onnxruntime`, pip-only — no system binary), a local OCR pass runs off the event loop and its recognized text is appended to the transcription under an `[OCR extracted text]` heading, truncated at 8000 characters; an OCR failure is logged and the vision transcription is kept rather than failing ingestion. Because `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS` also caps reasoning tokens, a small value on a reasoning model (including the default `openai/gpt-5-mini`) can return empty content — the loader logs a warning suggesting a higher cap and continues with empty text for that image. `transcribe_image` gained optional `prompt`, `max_completion_tokens`, and `reasoning_effort` keyword parameters on `LLMGateway` and the LLM interface; existing calls keep working unchanged, and images still reduce to `chunk.text` and feed the existing graph extractor unchanged (partially addresses #3637, PR #3956). * Fixes the two connected-components metrics being transposed in `get_graph_metrics()` on the Neptune graph backends (`GRAPH_DATABASE_PROVIDER=neptune` and Neptune Analytics). `NeptuneGraphDB.get_graph_metrics` unpacked its internal `_get_connected_components_stat()` helper as `num_cluster, list_clsuter_size`, but that helper returns `(sizes, count)` — so `num_connected_components` came back as the descending list of per-component sizes and `sizes_of_connected_components` as the integer number of components, the exact inverse of what the keys name. The unpack is corrected, so `num_connected_components` is now an `int` and `sizes_of_connected_components` a `list[int]` of per-component sizes in descending order, matching the `int`/`list[int]` shape the Ladybug, Neo4j, Postgres, and Turso adapters already returned — Neptune was the only backend that disagreed. **Impact is limited to Neptune deployments:** anything reading these two keys off a Neptune graph received the wrong type for each, so dashboards, alerts, and scripts written against the old shape — for example treating `num_connected_components` as a list, or `sizes_of_connected_components` as a scalar — must be switched back to the documented types, and a consumer that assumed the correct types was getting a type error or nonsense value rather than a wrong-but-plausible number. The same values feed the `graph_metrics` row written by `get_pipeline_run_metrics`, whose columns are `Integer` for `num_connected_components` and `JSON` for `sizes_of_connected_components`, so rows recorded from a Neptune graph before this fix cannot be trusted for these two fields; re-run metrics collection if you rely on their history. No public API signature, configuration option, environment variable, or migration changed — deploy the fix to pick it up (PR #3171). * Separates ontology-aware from ontology-free graph construction in the Cognify extraction path, and bases persisted entity identity on entity names instead of the graph-local ids the LLM assigns. `cognee.modules.graph.utils` no longer exports `expand_with_nodes_and_edges` or `retrieve_existing_edges`: the first is replaced by `construct_data_points_and_edges` plus `attach_new_edges_to_data_points`, and the second by `find_existing_edge_identities`, which takes a collection of `EdgeIdentity` values and returns the subset already in graph storage rather than the previous `{edge_key: True}` mapping. Ontology enrichment moves out to `cognee/modules/ontology/construct_data_points_and_edges_with_ontology.py` and now runs as a canonicalize-first pre-pass that rewrites the extracted graph before any nodes are constructed from it, rather than validating nodes one by one as they are built: nodes matching the same ontology individual collapse into a single node with the collapsed nodes' edges rewired onto the survivor, and an edge from a matched ontology subgraph is attached only when **both** of its endpoints are part of that subgraph instead of minting a node for the missing endpoint. A run with no ontology configured no longer passes through the ontology code path at all — the new `get_configured_ontology_resolver(config)`, which `cognify()` and `get_default_tasks()` both now call in place of their duplicated branching, returns `None` when neither an explicit `config["ontology_config"]["ontology_resolver"]` nor an `ONTOLOGY_FILE_PATH` environment setting is present, where the previous code fell back to instantiating an empty `RDFLibOntologyResolver`. **Entity ids change:** they are now derived from `Entity.id_for(node.name)` rather than `Entity.id_for(node.id)`, so two chunks mentioning the same entity name converge on one node, while several distinct nodes sharing a name inside a single extracted graph keep deterministic chunk-scoped ids instead of collapsing together; an extracted edge whose endpoint is not among that chunk's extracted nodes is now dropped rather than pointing at an id no node was created for. Entity nodes written by earlier runs keep their old id-derived ids, so re-cognifying data that is already in the graph can create a second node for an entity that is already there — re-run the affected datasets from scratch if you need ids to line up. **Edge properties change:** persisted edges no longer carry an `ontology_valid` property — grounding was never applied to relationship names, so the flag is now node-only; filter on the endpoints instead. The `cognify()` signature, configuration options, environment variables, and migrations are unchanged (SDK-160, PR #4262). * Makes enola code-graph ingestion incremental, so re-running it against an unchanged repository is near-free and a changed repository no longer accumulates facts that were deleted upstream. Previously every run re-loaded every fact and deleted nothing, so nodes for removed classes, files, and routes and edges for removed dependencies stayed in the graph indefinitely; the pipeline's generic incremental mode could not help, because it keys on the data item's content hash and the code-graph data item is a repository *path*, which does not change when the repository does. `extract_code_graph` now derives a snapshot identity for each run through the new `snapshot_identity()` helper — enola's `receipt.json` `snapshot_id` when present, otherwise a `sha256:` digest of `facts.jsonl`, and no identity at all when neither is readable, in which case the run always loads fully. That identity is compared against `last_snapshot_id`, a new field on the `CodeRepository` node; the marker is stored on the node in the graph rather than in the relational metastore because this pipeline persists no `Data` row to key relational state on, which also means it cannot outlive the graph it describes. On a match, `extract_code_graph` returns an empty list and both `add_code_graph_data_points` and `add_code_graph_edges` short-circuit, so an unchanged repository costs only the enola scan. When the identity differs, the load becomes a delta write followed by a sweep: `CodeGraphEntity` gained a `fact_hash` field fingerprinting each fact's derived fields, only facts whose hash is new or changed are written, only edges not already present are added, and then nodes and edges that earlier ingestions derived but the current snapshot no longer does are removed via `delete_nodes` and `delete_edge_triples` (chunked, so no single statement outruns the engine's per-call deadline). The sweep is deliberately narrow: it considers only code-graph node types belonging to repositories the current snapshot covers, and removes edges only between surviving code nodes, so other datasets, other repositories in the same graph, and edges to non-code nodes such as `belongs_to_set` → `NodeSet` are never touched. `last_snapshot_id` and a new `last_delta` record — added/updated/unchanged/removed counts, capped name samples, the snapshot id, and a load timestamp — are stamped on the repository node only after the load and the sweep both succeed, so a crashed run cannot later be mistaken for an up-to-date one. `SearchType.CODE`'s `code_query` gains a matching `delta` operation alongside `query_facts`, `explore`, `traverse`, `find_path`, and `impact_analysis`; it reads those records back and reports what the last ingestion changed per repository, returning `delta: null` for repositories loaded before this change. Same-named facts of the same kind now collapse to the first occurrence rather than the last, so a node's stored content and `fact_hash` stay stable across ingestions instead of flip-flopping and reading as "updated" on every run. **Operator impact:** repeated code-graph runs over an unchanged repository drop from minutes to roughly the \~3s enola scan, and orphans left behind by earlier ingestions are cleared on the first changed run after upgrading, so expect a one-off drop in code-graph node and edge counts. Because the marker is a property of the `CodeRepository` node, anything that drops the graph itself — prune, or deleting the graph database files — drops the marker with it, and the next run rebuilds from scratch. `forget(memory_only=True)` does not: it deletes by provenance, and the code-graph pipeline's payload is a repository path with no `Data` row to record provenance against, so its nodes and their marker survive. One known gap: on `index_vectors=True` runs, vector index entries for swept nodes are not yet removed; the default graph-only path (`index_vectors=False`) is fully handled. No environment variable, configuration option, or public function signature changed, and no migration is required (COG-6115, PR #4407). * Fixes `POST /api/v1/cognify` and `POST /api/v1/memify` hiding the real failure inside their `500` response body. Both routers built the body's `detail` as `getattr(run, "error", None) or str(run)`, but `PipelineRunErrored` has no `error` attribute — the failing task's error is recorded on `payload`, which the pipeline runner sets to `repr(error)` — so the `getattr` was always `None` and `detail` fell back to the model's repr, e.g. `status='PipelineRunErrored' pipeline_run_id=UUID('...') dataset_name='ds' payload="ValueError('LLM_API_KEY is missing')" ...` instead of the actionable message it wraps. Both routers now read `payload` when it is a string — the pattern `POST /api/v1/add` already used — so `detail` is the task's own error, e.g. `ValueError('LLM_API_KEY is missing')`, and falls back to the run repr only when the run carries no error string. The `500` status code, the `{"error": "Pipeline run errored", "detail": ...}` body shape, and the separate `{"error": "Internal server error", "detail": ...}` body those endpoints return for unexpected exceptions are all unchanged. **Client impact:** only the text inside `detail` changed, so log parsing or alerting rules that pattern-matched the old `status='PipelineRunErrored' ...` repr no longer match and should read the plain message instead. No public API signature, configuration option, environment variable, or migration changed (PR #3265). * Replaces the non-standard `418 I'm a teapot` status code with `500 Internal Server Error` in the four places it was used. `GET /api/v1/datasets` and `POST /api/v1/datasets` wrap any unexpected failure in an `HTTPException` — retrieving datasets and creating a dataset respectively — and both now raise `500`; their endpoint docstrings and the generated HTTP API reference list `500 Internal Server Error` accordingly. The global `CogneeApiError` handler in `cognee/api/client.py` also returns `500` for its fallback branch, which fires when a raised Cognee exception is missing a message, name, or status code and the handler substitutes `{"detail": "An unexpected error occurred."}`. The `CogneeApiError` base class default `status_code` moves from `418` to `500` as well; every direct subclass sets its own status code (for example `422` for `CogneeValidationError`, `503` for `CogneeTransientError`), so this default applies only to a direct `CogneeApiError(...)` raise that omits `status_code`. **Client impact:** integrations that branch on `418` for these responses must switch to `500` — response bodies, status codes for every other error, endpoint paths, and request shapes are unchanged. No configuration option or environment variable changed (issue #3742, PR #3860). * Removes the stale `tree-sitter` and `tree-sitter-python` dependencies from the `cognee[codegraph]` extra, which now installs only `fastembed` and `transformers`. Neither package was imported anywhere in Cognee: they were left behind by the removed Python AST-based code-graph parser, and the current code-graph implementation delegates extraction to the external `enola` binary (`cognee/tasks/code_graph/extract_code_graph.py`). Because `codegraph` pinned `tree-sitter>=0.24.0,<0.25` while `docling-full` needs `tree-sitter>=0.25` through `docling-core[chunking]`, `pyproject.toml` also declared the two extras mutually exclusive under `[tool.uv] conflicts`, so the contributor setup command documented in `AGENTS.md` — `uv sync --dev --all-extras --reinstall` — failed to resolve at all with `error: Extras codegraph and docling-full are incompatible with the declared conflicts`. With the unused pins gone the conflict declaration is dropped too, so all extras install together and both `codegraph` and `docling-full` can be used in the same environment; `uv.lock` was regenerated to drop the obsolete conflict markers. **Impact:** an `ImportError` mentioning `tree_sitter` is no longer resolved by installing `cognee[codegraph]` — no Cognee code path imports it, so the error comes from something else in your environment. Code-graph extraction behavior is unchanged (it needs the `enola` binary, not a Python parser), and no public API signature, configuration option, or environment variable changed; no migration is required (PR #4432). * **Security: restricts `POST /api/v1/settings` to superusers.** Writing the system LLM configuration (provider, model, API key) and vector-database configuration (provider, URL, API key) over HTTP only required an authenticated user, so any account that could log in — an ordinary tenant member, an integration or agent account, a signed-in UI session — could repoint the deployment's LLM or vector store and read back a masked preview of the stored keys (first ten characters) through `GET /api/v1/settings`. The `save_settings` handler now checks `user.is_superuser` before touching `save_llm_config` / `save_vector_db_config` and answers a non-superuser with `403 Forbidden` and the body `{"error": "Superuser privileges required to modify settings"}`; the request payload, the empty `200` on success, and the `400` / `500` error codes are unchanged. This re-adds a guard that shipped in #3115 and was later reverted, and it fixes **CVE-2026-58473 / GHSA-49f7-whx5-4256**; a regression test (`cognee/tests/api/test_settings_authorization.py`) now pins both the allowed and the forbidden case. **`GET /api/v1/settings` is unchanged** — reading the settings still requires only an authenticated user, so this is not a lockdown of the read path. **Who is affected:** only callers that are not superusers. `create_user(...)` defaults to `is_superuser=False`, so integrations, automation scripts, and UI flows that let non-admin accounts adjust LLM or vector settings now break with `403` and must either authenticate as a superuser or drop the write. The auto-created default user (`default_user@example.com`) *is* a superuser, and with authentication disabled every request falls back to that user, so single-user deployments, local development, and the `cognee.start_ui()` / `cognee-cli -ui` "Add your API key" modal keep working exactly as before. No configuration option, environment variable, or migration ships with the fix — deploy it to pick it up (PR #4434, mirroring contributor PR #4252). * Fixes the eval-framework HTML dashboards interpolating arbitrary benchmark text into their markup without escaping it. Both dashboard modules — `cognee/eval_framework/metrics_dashboard.py`, the one the eval runner uses, and its analysis twin `cognee/eval_framework/analysis/dashboard_generator.py` behind `create_dashboard()` — built the report with f-strings: `generate_details_html` emitted each per-item field straight into a ``, and `get_dashboard_html_template` dropped the `benchmark` name into `` and `<h1>`. Any `<`, `&`, or tag-like content in that text corrupted the rendered page (a `</td>` or `<script>` in a golden answer escaped its cell and broke the table), and made the generated report an HTML-injection sink when opened in a browser, since the text originates from benchmark data and model output rather than from Cognee. Both modules now apply `html.escape` at every interpolation point: the details-table cell values, the derived column headers (`metrics_dashboard.py`, which titles them from the item keys), the per-metric `<h3>` section heading, and the `benchmark` name in the page template. The fields this covers are the ones each module already rendered — `question`, `answer`, `golden_answer`, `reason`, and `score` in `dashboard_generator.py`, and whatever keys the metric items carry in `metrics_dashboard.py` (`question`, `answer`, `golden_answer` by default, `question` and `retrieval_context` for `contextual_relevancy`, and those two plus `golden_context` for `context_coverage` — each with `reason` and `score`). Plotly figure HTML from `create_distribution_plots()` / `create_ci_plot()` and the already-assembled `details_html` list are deliberately left unescaped as trusted, pre-rendered markup, so charts render exactly as before. **Impact:** benchmark fields that contained raw HTML previously rendered as markup and now display as literal text — there is no flag to restore the old behavior. No public API signature, configuration option, or environment variable changed, and no migration is required; regenerate a dashboard to pick up the fix (PR #4429). * **Breaking: `cognee.validate()` now rejects a dataset it cannot read instead of falling back to the default stores.** `validate()` resolved its `dataset` argument through `get_authorized_existing_datasets(..., "read", user)` and read the first result, but when *none* of the requested names resolved it left the resolved dataset as `None` and entered the database context unscoped — so with backend access control disabled a typo, an unknown name, or a dataset the caller had no `read` permission on produced a `ValidationReport` about the single shared graph and vector stores, indistinguishable from a real report about the requested dataset. Every requested name must now resolve to a dataset the caller can read; otherwise `DatasetNotFoundError` (`"Dataset not found or not readable."`, from `cognee.modules.data.exceptions`, `status_code=404`) is raised *before* any graph or vector adapter is opened, so no store is touched on the rejected path. The check compares counts, so a partially authorized list is rejected as a whole — `dataset=["mine", "not-mine"]` raises rather than silently validating `mine` — and it does not branch on `ENABLE_BACKEND_ACCESS_CONTROL`, so the rejection is identical with access control on or off. Selection is otherwise unchanged: when all names are authorized, the first still determines which graph and vector store are read, and one graph is still checked per call. **Caller impact:** the default `dataset="main_dataset"` is subject to the same rule, so a bare `await cognee.validate()` against an installation where `main_dataset` does not exist yet now raises `DatasetNotFoundError` — where previously it reported on the shared stores with access control disabled, and failed with a generic `CogneeValidationError` ("A dataset must be provided...") with it enabled; and because dataset names resolve only within datasets the caller *owns*, a dataset merely shared with the caller is rejected when requested by name. Passing `dataset=None` or an empty list still skips resolution entirely, which remains the only path that enters no dataset context — with access control disabled that reads the unscoped shared stores, while with it enabled the call still fails, since a dataset is required to resolve the per-dataset databases. **Over HTTP the rejection surfaces as a `500`, not a `404`:** `GET /api/v1/validate` wraps every exception from the route in `{"status": "error", "reason": "validation failed: ..."}` with status `500`, so the `DatasetNotFoundError` never reaches the global `CogneeApiError` handler that would map it to `404` — expect additional `500`s whose `reason` is `validation failed: DatasetNotFoundError: Dataset not found or not readable. (Status code: 404)` — the `404` in the string is the exception's own string form, the HTTP status stays `500` — rather than a rise in `4xx`, and match the `reason` for `Dataset not found or not readable.` if you alert on this. Remediation is to request a dataset the caller is authorized to read, or to catch `DatasetNotFoundError`. No configuration option, environment variable, or migration ships with the fix, and `validate()`'s signature is unchanged (fixes #4414, PR #4455, mirroring contributor PR #4416). * Adds `TELEMETRY_ORIGIN`, an environment variable that labels where a telemetry event comes from, so events can be segmented by origin. `send_telemetry` reads it with `os.getenv("TELEMETRY_ORIGIN", "sdk")` and stamps the value onto every event's `properties` payload as `telemetry_origin`; the Cognee-managed cloud sets `TELEMETRY_ORIGIN=cloud`, and everything else — local SDK usage, self-hosted servers — reports the default `sdk`. It is read directly from the environment on each event, matching how `send_telemetry` already reads `TELEMETRY_DISABLED` and `ENV`, so there is no config-schema change and it cannot be set through `cognee.config.set(...)`. The property is written before the `**additional_properties` spread, so a caller that passes its own `telemetry_origin` in `additional_properties` still overrides it per call. **Impact:** non-breaking and no action is required — telemetry payloads gain one property, and `TELEMETRY_DISABLED=true` still suppresses events entirely. No public API signature or migration changed (PR #4433). * Fixes the Redis session-cache adapter (`CACHE_BACKEND="redis"`) returning `None` instead of `[]` from `get_latest_qa_entries` for an empty or unknown session. `RedisAdapter.get_latest_qa_entries` takes a `lindex` fast path when `last_n == 1` — the hot path, since loading the previous turn of a session reads with `last_n=1` — and that branch ended in `if data else None`, contradicting the method's declared `-> list[SessionQAEntry]` return type and diverging from the SQL and FS adapters; the general `lrange` path used for every other `last_n` already returned `[]`. It now returns `[]` on the fast path as well, and the `lindex` optimization is kept. **Impact:** `cognee.session.get_session()` was not affected, because `SessionManager.get_session` carries an `if entries is None` guard that returns `""` or `[]`; the `TypeError: 'NoneType' object is not iterable` this could raise on Redis (while the same call worked on SQLite and FS) was reachable only from code calling the cache adapter directly with `last_n=1`, including through the backward-compatible `get_latest_qa` shim. No public API signature, configuration option, environment variable, or migration changed, and no action is required on upgrade (fixes #3930, PR #3931). * Fixes the Ladybug and Amazon Neptune graph adapters reporting a *failed* batch edge-existence check as an *empty* one, which could let a cognify run finish reporting success while persisting nothing. `has_edges(edges)` takes a list of `(source_id, target_id, relationship_name)` triples and returns the subset that already exists in the graph; `LadybugAdapter.has_edges` (`GRAPH_DATABASE_PROVIDER="ladybug"`, the default backend) and `NeptuneGraphDB.has_edges` (`GRAPH_DATABASE_PROVIDER="neptune"`) both wrapped the query in `except Exception`, logged the error, and returned `[]`. That value is indistinguishable from a genuine "none of these edges exist yet" answer, and the adapter method's only production caller is the cognify dedup step — `find_existing_edge_identities` in `cognee/modules/graph/utils/retrieve_existing_edges.py`, called from `extract_graph_from_data` to subtract already-stored edges before writing the rest. So when the store was unavailable or corrupt — a corrupt write-ahead log after an unclean shutdown, for example — the check answered "nothing exists", every extracted edge was treated as new and written, those writes failed against the same broken store, and the run completed without raising while the graph stayed empty. Both adapters now log and then re-raise, which is what the other backends already did: `neo4j` logs and re-raises `Neo4jError`, and `postgres` and `turso` never caught the error in the first place. Ladybug's empty-**input** short-circuit is unchanged — `has_edges([])` still returns `[]` without touching the store — and Neptune, which has no such short-circuit, is unchanged in that respect too. Only the batch `has_edges` path was touched; the single-edge `has_edge` and every other adapter method behave as before. **Impact:** no public API signature, configuration option, environment variable, or migration changed, and a run against a healthy graph store behaves exactly as it did. What changes is the failure mode: an ingest that previously "succeeded" against a broken store now fails with the backend error propagated to the caller, so a job that was silently a no-op may begin surfacing as a failure in logs and CI after upgrading. That failure reports a problem that was already present rather than introducing a new one, so the remedy is to fix or restore the graph store, not to suppress the error. This covers the silent-data-loss existence check in the linked issue; the startup write-ahead-log recovery and `SIGTERM` handling also requested there are a separate change and are not included. Unit tests in `cognee/tests/unit/infrastructure/databases/test_has_edges_error_propagation.py` cover the three cases — a successful check returning the existing-edge tuples, a query failure raising instead of returning `[]`, and empty input short-circuiting without reaching the store (fixes #4348, PR #4430). * Gives each relational [dlt](/integrations/dlt-integration) source a stable identity, so re-ingesting one updates it in place instead of piling up copies. A source is now stored as a single record keyed on `dlt_source:{dataset_name}:{source_name}` — an identity that does not include the data — with change detection carried separately by a content hash over the source's tables and rows. A plain re-run of `add()` / `remember()` on an already-ingested source is idempotent and skips it without reprocessing, whether or not the data changed; to pick up upstream changes, re-ingest explicitly with `add(..., incremental_loading=False, data_cache=False)` (the completed-skip runs whenever either flag is on) or `update()` with the record's UUID. The record then updates in place under the same identity (so the source is never missing from the store between runs), and the next `cognify()` purges that source's previously derived graph nodes, edges, and vectors before re-emitting the current rows, so upstream deletions and edits no longer leave stale rows behind. Two further changes ship alongside it: dlt row embeddings move out of the shared `DocumentChunk_text` collection into their own `DltRow_text` collection — graph-completion retrieval reads both, but chunk search is now documents-only — and the `DLT_MAX_ROWS_PER_TABLE` default changes from `50` to `0`, which means no cap, so a source that was previously truncated to 50 rows per table is now ingested in full unless you set a positive value (via the env var or the `max_rows_per_table` kwarg). **Impact:** three things to plan for on upgrade. First, reprocessing a *changed* dlt source now requires `delete` permission on the dataset, because the purge is re-authorized as a delete; a run without it fails loudly rather than quietly serving stale rows, so check the permissions of any automation that re-syncs dlt sources. Second, dlt data ingested before this change was stored as pre-manifest per-row records, which are no longer supported — `cognify()` raises on them with a message naming the record, and re-adding the source ingests it once as a manifest (and sweeps the legacy records away), so expect a one-off re-ingest of existing dlt datasets. Third, renaming a dlt source or its dataset changes the identity and is therefore a remove + add: the new name ingests fresh and the old name's records remain until you delete them. No public API signature changed, and `primary_key`, `write_disposition`, `query`, and `max_rows_per_table` are accepted exactly as before (COG-2222, PR #4278). * Fixes background `remember()` work being destroyed by Python's garbage collector mid-flight, so the call reported success while the knowledge graph was never updated. Both fire-and-forget paths were affected: the `run_in_background=True` run that performs `add()` followed by `cognify()`, and the session-to-graph bridge that `remember(session_id=...)` starts when `self_improvement` is on — it defaults to `True`, so every `session_id` call took that path. Each created its task with a bare `asyncio.create_task(...)` and kept the only application-level reference on the returned `RememberResult`. The event loop holds just a weak reference to a running task, so once the caller dropped that result — as `POST /api/v1/remember` does, serializing the result and discarding the object — the surviving task/closure/result references formed a closed cycle with nothing outside it, and the cycle collector could collect the pending task, leaving `Task was destroyed but it is pending!` in the logs as the only trace. Both tasks are now held in a module-level anchor set for their whole lifetime and remove themselves on completion via `add_done_callback`, the same pattern already used for background sync (`cognee/api/v1/sync/sync.py`) and for background pipeline runs (`cognee/modules/pipelines/layers/pipeline_execution_mode.py`). **Who is affected:** callers that start background remember work and do not keep the returned `RememberResult` alive — the HTTP endpoint, and SDK code that calls `remember(..., run_in_background=True)` or `remember(session_id=...)` without holding the result. A caller that kept the result and awaited it was never at risk, and blocking `remember()` calls were never affected. The loss depended on a collection cycle landing at the wrong moment, so an affected deployment saw it intermittently rather than on every request; runs that vanished this way were never recorded as failures, so re-run any background ingestion whose data is missing from the graph. The documented behavior of `remember()` is unchanged — background mode still returns immediately and the result can still be awaited later — it is simply now reliable, and no public API signature, configuration option, environment variable, or migration changed, so deploying the fix is the whole action (fixes #4312, PR #4456). * Fixes `GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw` rejecting a caller who holds a dataset-level `read` grant but does not own the data item. The handler already authorized the dataset with `get_authorized_existing_datasets([dataset_id], "read", user)`, then resolved the item with `get_data(user.id, data_id, dataset[0].id)` — and `get_data` re-checks `data.owner_id == user_id`, raising `UnauthorizedDataAccessError` (HTTP **401**) when they differ. Since a document's `owner_id` is the principal that ingested it, anyone reading a dataset shared with them through an ACL could list its items but got a 401 on every raw download, even though the dataset permission check had already passed. The handler now resolves the item entirely within the authorized dataset: `resolve_data_id(dataset[0].id, data_id)` maps the caller-supplied id to the canonical row id (falling back to the recorded pre-fork `legacy_id`, so ids issued before the dataset-scoping upgrade keep resolving, exactly as before), and the row is then taken from `get_dataset_data(dataset[0].id)`, which is scoped by dataset membership and does not consult `owner_id`. **Impact:** dataset `read` is now sufficient to download raw files — a request that previously returned 401 for a non-owning reader returns the file. Nothing else about the endpoint changed: it is still gated on `read` for the containing dataset, an id that is unknown in that dataset still returns 404, and the owner's own downloads, the S3 streaming path, the local-file path, and the 501 response for unsupported storage schemes all behave as before. One adjacent change: when the dataset itself is not found or not readable, the 404 body's key is now `message` instead of `detail`, so a client reading `detail` off that specific response needs updating (fixes #4162, COG-5923, PR #4468, mirroring contributor PR #4200). * Fixes file-extension detection being case-sensitive, so files named `data.CSV`, `NOTES.MD`, `config.YAML`, `payload.JSON`, or `feed.XML` were ingested as plain text. `guess_file_type` (`cognee/infrastructure/files/utils/guess_file_type.py`) took the extension from the file name and compared it against literal lowercase lists — `.txt`/`.text`, `.csv`, `.md`/`.markdown`, `.json`, `.xml`, `.yaml`/`.yml` — but `Path("data.CSV").suffix` is `".CSV"`, which matches none of them. Those are exactly the formats that carry no magic-number signature, so an unmatched uppercase extension fell through to `filetype.guess`, which returned `None`, and the file was labeled `text/plain` with extension `txt`. The extension is now lowercased before the lookup, so case no longer affects the result. **Impact:** two things downstream change for files whose extension was not already lowercase. `get_file_metadata` stores the guessed mime type and extension, and `classify_documents` selects the document class from that extension — so a `.CSV` file is now a `CsvDocument`, whose reader expands each row into `key: value` text, instead of a plain `TextDocument` chunked as prose; `.MD`, `.JSON`, `.XML`, and `.YAML` already mapped to `TextDocument` and keep that class, but the `mime_type` and `extension` recorded for them are now the real ones (`text/markdown`/`md` rather than `text/plain`/`txt`). Loader selection also shifts: `LoaderEngine.get_loader` tries the raw path extension first and does *not* lowercase it, so the corrected content-detected extension is what takes effect on the fallback branch, and `data.CSV` now reaches the CSV loader path — `CsvLoader`, or the DLT CSV loader when the `dlt` extra is installed — rather than `TextLoader`. Magic-number formats (PDF, images, audio, video) were already case-independent and are unaffected, and an uppercase `.TXT` still resolves to `text/plain` exactly as before. No public API signature, configuration option, environment variable, or migration changed. Data already ingested keeps the type it was stored with — re-ingest affected files to pick up the corrected type. Unit tests in `cognee/tests/unit/infrastructure/files/utils/test_guess_file_type.py` assert correct detection for lower- and uppercase csv/md/json extensions and uppercase XML/YAML, plus the unchanged `.TXT` case (PR #4453). * Adds an opt-in, audit-grade provenance ledger: an append-only, tamper-evident record of the document → chunk → entity → relationship lineage each ingestion produced. This is the migration `b8c1d3e5f7a9` listed above — it creates the single `provenance_entries` table (SQLite and Postgres) that backs the feature, and it is idempotent, so it is a no-op where the table already exists. **Default off** — with `PROVENANCE_TRACKING` unset, the cognify task list and behavior are identical to before, nothing is written to the new table, and no public API signature changed. Enabling it (`PROVENANCE_TRACKING=true` on `CognifyConfig`, `cognee/modules/cognify/config.py`) splices a `record_provenance` task (`cognee/tasks/provenance/record_provenance.py`) into the default pipeline right after `add_data_points` — where node ids are persisted and stable — and before the contradiction-detection spread, committing all of one invocation's entries as a single chained transaction. The task returns its input unchanged and swallows all of its own errors, logging a warning rather than failing the run, so provenance can never break ingestion; DataPoints from custom `graph_model` schemas that lack the default `made_from`/`is_part_of`/`contains` shape are covered by a generic walk using the same traversal `add_data_points` uses. Every entry carries a SHA-256 checksum over its canonical JSON plus the previous entry's checksum, linked by a unique `sequence_id`, so tampering — deletion, reordering, or a single-field edit — is detectable. Ledger keys are prefixed with the dataset id, because cognee entity ids are deterministic and the ledger lives in the shared relational database, so two datasets mentioning the same entity name keep separate version chains. A new `ProvenanceManager` (`cognee.modules.provenance.get_provenance_manager()`) is the programmatic surface: `track_entity`, `track_chunk`, `track_relationship`, `get_provenance`, `get_lineage`, `trace_lineage`, `revision_history`, `invalidate`, `verify_chain`, `check`, and `get_statistics`. All entries share one `sequence_id` sequence and each commit takes a ledger-wide write lock (a Postgres advisory lock, so it serializes across processes), which is why one task invocation commits as one transaction — concurrent runs queue behind each other for their ledger writes. A batch that fails rolls back entirely, so its entries never claim sequence numbers and the chain still verifies as intact: `verify_chain()` proves the stored entries were not tampered with, not that everything an ingestion produced was recorded. **Operator note:** enabling this adds relational-DB writes to every `cognify()` run, so plan for disk and backup sizing and for the cost of periodic verification; `verify_chain()` streams the ledger by keyset pagination instead of materializing it client-side, and `get_statistics()` aggregates DB-side, so both are safe to run against a large table — `check()`, the referential-integrity pass, also streams its rows but first loads every entry id and activity id into memory, so its footprint grows with the ledger. See [Provenance ledger](/python-api/cognify#provenance-ledger) (COG-6172, PR #4476). *** ## v1.4.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.2)** Release that bumps the package version from `1.4.1` to `1.4.2`. No new Alembic revision ships in this cut, so no migration is required on upgrade. The entries below are the work promoted from the development branch into this release; work logged under the v1.4.1.dev0 and v1.4.1.dev1 pre-release sections below — including the JSON visualization endpoints (PR #4331) and the cancelled-request session cleanup (PR #4250) — also ships in this release. ### Highlights * Names raw-text uploads made over a `serve()` connection by content hash. The remote client sent every raw string passed to `remember()` or `add()` — including each string inside a list — under the fixed filename `data.txt`, so all text uploads for a tenant collided on a single remote object and concurrent adds raced the server's content-hash read-back, failing with `FileContentHashingError` 409s. The client now derives the name from the text itself as `text_<md5_hash>.txt`, reusing the same `TextData` namer local ingestion applies to nameless text, so a given string gets the same object name whether it is ingested locally or remotely. File-like uploads are unchanged and still use their own `name` (falling back to `upload`), and the uploaded text content is unchanged. **Operator impact:** tooling or tests that assert the uploaded basename is `data.txt` need updating to expect `text_<md5_hash>.txt`; the client exposes no way to supply your own filename. No public API signature, configuration option, or environment variable changed (PR #4366). * Clarifies that the Postgres graph store is a demo feature and states where to reach us about production use. The `postgres` graph backend's module and adapter docstrings, and the `graph_database_provider == "postgres"` branch in `get_graph_engine.py`, now say explicitly that it is not production-ready and that production workloads should use a graph-native backend such as Kuzu or Neo4j; the production-ready adapter is a licensed product, and the contact route is now `social@cognee.ai` (or a call with the sales team via [cognee.ai](https://www.cognee.ai)) rather than the previously documented `social@topoteretes.com`. This is a documentation and positioning change only — the adapter's behavior, its `GRAPH_DATABASE_*` configuration and fallback to the relational `DB_*` settings, and its existing limitation that `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` raise `SearchTypeNotSupported` are all unchanged (PR #4366). * Fixes a connection-pool deadlock in the API-key authentication path that could take the whole API down under concurrent load — surfacing first as intermittent `401`s and gateway timeouts, then as a full outage, with Postgres backends stranded `idle in transaction`. `UserManager.get_by_token()` in `cognee/modules/users/get_user_manager.py` looked up the `UserApiKey` row in its own session and then resolved the user with `self.get()` **while that session was still open**; `self.get()` borrows the request-scoped session behind the FastAPI `get_user_db` dependency, which is held until the response is finished, so it checked out a *second* pooled connection and kept it open for the entire request — across the slow completion call. Every in-flight authenticated request therefore pinned two pooled connections instead of one: a pool of N slots could safely serve only about N/2 concurrent authenticated requests, and at N concurrent requests every slot was held by an API-key lookup waiting for a second connection that could never free — a circular wait that deadlocked the pool (with the relational defaults of `pool_size=5` plus `max_overflow=35`, that ceiling is 40). `get_by_token()` now resolves both rows with two `select()` calls inside a single short-lived session and returns from inside it, never calling `self.get()`, so authentication holds one connection and releases it as soon as the lookup finishes. This is a **separate cause from the cancelled-request cleanup fix logged under v1.4.1.dev0** (PR #4250): the deadlock here reproduces with plain concurrency and no cancellation involved, which is why that earlier change did not resolve #4197. Two workarounds operators may have reached for did not address it and can be reconsidered — enlarging `POOL_ARGS` only raised the concurrency at which the deadlock hit, and `idle_in_transaction_session_timeout` never reclaimed these slots, because they belong to live, in-flight requests rather than abandoned ones. No public API signature, configuration option, or environment variable changed and no migration is required; redeploy to apply (fixes #4197, PR #4354). * Keeps idle subprocess-backed database engines (embedded Ladybug/Kuzu graph, LanceDB vector) alive for a configurable idle TTL instead of closing them every time a dataset context exits. Previously, the last holder of a dataset's queue slot evicted and force-closed the cached engine on release, so a follow-up request for the same dataset paid for a worker close plus a fresh spawn. The release path now refreshes the engine's idle timestamp, and a background daemon thread (`subprocess-idle-reaper`, started lazily on the first kept-alive release) sweeps the engine caches every `max(5, min(60, TTL / 4))` seconds and force-closes only the engines that have gone a full TTL without use. The sweep skips datasets holding an active queue slot — the same pin that protects them from capacity eviction — so an operation running longer than the TTL cannot lose its engine mid-flight, and it skips engines that are not subprocess-backed, so remote stores (Neo4j, Postgres, PGVector) are unaffected and keep plain LRU behavior. The TTL is set with the new `SUBPROCESS_IDLE_TTL_SECONDS` environment variable, default `600` seconds; negative values are clamped to `0` and fractional values are accepted. **Operator impact:** a kept-warm worker holds its PID, its memory, and — for the file-based graph store — its database file lock for up to the TTL after its last use, so worker PIDs are now stable across requests and a data directory can stay locked longer than before (relevant when another tool or backup job needs to open the same files). The number of retained idle workers is still bounded by the engine cache capacity (`DATABASE_MAX_LRU_CACHE_SIZE`). Set `SUBPROCESS_IDLE_TTL_SECONDS=0` to restore the previous close-at-release behavior; the keep-alive also has no effect when the dataset queue is disabled (`DATASET_QUEUE_ENABLED=false`), because the release path never runs (PR #4384). * Fixes the remaining code paths that held one pooled relational connection while acquiring a second, which could deadlock a bounded Postgres or PGVector pool. This is a different mechanism from the cancellation leak fixed in #4250: nothing was abandoned mid-transaction, the connections simply *overlapped* — a method opened a session (or an `engine.begin()` connection) and then, inside it, called a helper that checks out a connection of its own, so each such call pinned two at once. Once concurrency reached the pool's ceiling, every waiting task held one connection and waited for another that could not come free, and the stuck backends showed up as `idle in transaction`. The default embedded SQLite store has no bounded shared pool, so this only affected Postgres/PGVector deployments. All affected paths were reordered to resolve the lookup first and open the working session second: in `SQLAlchemyAdapter` (`cognee/infrastructure/databases/relational/sqlalchemy/SqlAlchemyAdapter.py`), `insert_data`, `delete_entity_by_id`, `get_all_data_from_table`, `get_table_names`, and `extract_schema` now resolve `get_table()` / `get_table_names()` / `get_schema_list()` before opening their own connection, since each of those helpers checks one out; `create_role`, `create_tenant`, `add_user_to_tenant`, `select_tenant`, `get_default_user`, `get_deletion_counts`, and `get_document_ids_for_user` resolve their user, tenant, and per-dataset lookups outside the session that does the work; and `get_pipeline_run_metrics` computes its token count on the session it already has open instead of calling `fetch_token_count`, which would have opened a second one. `delete_role`, `remove_user_from_role`, and `get_users_in_role` — where the permission check depends on data read from the first session — split into two sequential sessions with the check running on its own connection in between; the entity lookup still precedes the permission check, so `EntityNotFoundError` / `UserNotFoundError` / `RoleNotFoundError` still take precedence over a permission failure exactly as before, and `get_default_user`'s deferred `create_default_user()` call stays inside the same error handling, so a missing schema still surfaces as `DatabaseNotCreatedError`. Every method keeps its arguments, return value, and exceptions. No public API signature, configuration option, environment variable, or migration changed, and no pool resizing is needed — deploying the release is the whole action (PR #4392, follow-up to #4197). * Fixes the SQL session-cache engine crashing at startup when `POOL_ARGS` sets `"poolclass": "nullpool"`. `POOL_ARGS` has two consumers: the relational adapter already normalized the string `"nullpool"` to SQLAlchemy's `NullPool` class before building its engine, while `SqlCacheAdapter` read the same relational pool arguments and passed them to `create_async_engine` unchanged — SQLAlchemy inspects `poolclass` as a class, so the string raised `CacheConnectionError: Failed to initialize SQL cache engine for …: 'str' object has no attribute '__dict__'` for both `CACHE_BACKEND="sqlite"` and `CACHE_BACKEND="postgres"`. The cache adapter now applies the same normalization, so one `POOL_ARGS` value works for both consumers — which matters behind an external pooler (for example Neon's `-pooler` pgbouncer endpoints), where disabling client-side pooling is the deliberate choice and the cache engine connects to that same endpoint. **Operator impact:** none beyond upgrading — no environment variable was added or renamed, no configuration change is required, and no migration ships with the fix; deployments that leave `POOL_ARGS` unset or omit `poolclass` are unaffected, and a deployment that had to drop `poolclass` to work around the crash can restore it (PR #4376). * Fixes an explicitly configured relational `POOL_ARGS` being ignored for per-dataset PGVector engines under `ENABLE_BACKEND_ACCESS_CONTROL="true"`. `PGVectorAdapter` resolved pool arguments as `VECTOR_POOL_ARGS` → built-in access-control default → relational `POOL_ARGS`, so the built-in default (`{"pool_size": 2, "max_overflow": 20}`, which exists to curb connection fan-out when every dataset gets its own engine) beat the operator's own sizing and `POOL_ARGS` silently did nothing in multi-user mode. Precedence is now `VECTOR_POOL_ARGS` → relational `POOL_ARGS` → the access-control default (used only when neither is set, and only while access control is on; an empty pool config otherwise), so explicit configuration outranks the built-in default. **Operator impact:** a deployment that sets `POOL_ARGS` while running with backend access control enabled will see its per-dataset PGVector pools resized to that value after upgrading — larger or smaller than the previous 2/20 depending on what is configured, and multiplied by the number of active datasets — so check the total against the server's `max_connections` and set `VECTOR_POOL_ARGS` to keep PGVector on its own sizing if you want the two to differ. No environment variable was renamed or added and no migration is required; restart or redeploy to pick up the fix (PR #4351). * Scopes role visibility to membership, so a tenant member can see the roles they belong to and who else is in them without holding tenant-wide user-management permission. `GET /api/v1/permissions/tenants/{tenant_id}/roles` no longer raises `PermissionDeniedError` for callers without that permission: when the tenant exists it now returns `200` (a nonexistent `tenant_id` still returns `404`), with owners and user-management holders (for example, an Admin role) seeing every role in the tenant and everyone else seeing only the roles they are a member of. **Client impact:** the `403` is gone from this endpoint — a caller who previously got `403` now gets a filtered list, and a caller who belongs to no role in the requested tenant (including a `tenant_id` for a tenant they are not part of) gets an empty list, so clients that branched on `403` to detect "not allowed" need to branch on the returned list instead. `GET /api/v1/permissions/tenants/{tenant_id}/roles/{role_id}/users` now returns `200` to members of the role itself; non-members without user-management permission still receive `403`. That endpoint also fixes a cross-tenant scoping hole: the role is now resolved by `(role_id, tenant_id)` rather than by id alone, so a role id from another tenant returns `404` instead of that tenant's member list. No new endpoint, request or response field, permission type, environment variable, or migration ships with this change (COG-6064, PR #4336). * Fixes two code paths that held one pooled relational connection open while checking out a second, which could deadlock a bounded Postgres/PGVector pool under concurrency. Each in-flight call pinned two connections at once, so once concurrent calls reached the pool's ceiling the waiters formed a circular wait — requests hung rather than erroring, and the stranded backends showed up as `idle in transaction` and then as pool exhaustion. Two sites are fixed: `get_or_create_dataset_database` — the function that creates per-dataset databases under `ENABLE_BACKEND_ACCESS_CONTROL="true"` — wrapped `create_authorized_dataset(...)` in an `async with db_engine.get_async_session()` block that never used `session`, holding that connection idle while the callee opened its own session plus another for the permission grant; the dead wrapper is removed, so the call now runs with no outer session. That branch fires only when a dataset is passed to `get_or_create_dataset_database` by name rather than id — a live path from `add()` and `cognify()` in v1.4.0 and earlier, while since v1.4.1 the database context resolves dataset names to ids before entry, making the removed wrapper a latent hazard rather than a reachable deadlock on current code. And `PGVectorAdapter.delete_data_points` called `await self.get_table(collection_name)` *inside* its write session, but `get_table()` opens its own `engine.begin()` connection, so the table lookup is now resolved before `get_async_session()` — exactly as the sibling `retrieve()` and `search()` methods already did. When access control is off, PGVector borrows the relational engine, so the `delete_data_points` overlap contended the pool shared with every relational query; under access control it runs on the per-dataset PGVector engines, which are smaller (`pool_size=2`, `max_overflow=20` by default) and reach the two-connections-per-call ceiling soonest. This produces the same `idle in transaction` signature as the cancellation leak fixed in #4250 and the auth-path deadlock of #4197 (fixed in #4354) but by a different mechanism; the defect class is the same one swept across the relational adapter in #4392. No public API signature, configuration option, environment variable, or database migration changed, and no pool resizing is needed — restart or redeploy to pick up the fix (PR #4389). * Fixes extraction schemas derived from a custom `graph_model` dropping domain fields that the model inherits from your own `DataPoint` subclasses. `datapoint_model_to_basemodel` — the conversion behind `graph_model_to_graph_schema` and the structured-output schema Cognee hands the LLM — selected fields by reading each class's own `__annotations__`, so only the fields annotated on the leaf class survived: given `Animal(DataPoint)` with `species: str` and `Dog(Animal)` adding `breed: str`, the schema for `Dog` contained `breed` alone and the LLM was never asked to extract `species`. Fields are now taken from the model's merged `model_fields` minus the names defined on `DataPoint` itself, so inherited domain fields are preserved — including required fields, fields with defaults or default factories, nested `DataPoint` fields (converted through the same shared cache, which also terminates cyclic `A → B → A` model graphs), and fields inherited through several levels of subclassing. `DataPoint`'s own infrastructure fields (`id`, `version`, `type`, `created_at`, `metadata`, and the rest) stay excluded, and because the exclusion is by field name a subclass that overrides `metadata` keeps it out of the schema too. **Impact:** the JSON schema returned by `graph_model_to_graph_schema` gains properties for inheritance-based custom models, and extraction for those models now populates the inherited fields, so downstream validators or snapshot tests that pinned the previous leaf-only schema need updating. Flat custom models — every field annotated on the class you pass, as in the guide examples — produce the same extraction schema as before; the one visible difference is in `graph_model_to_graph_schema` output, where a `metadata` override annotated on the class previously leaked into the schema and is now excluded like the rest of the infrastructure fields. No public API signature, configuration option, or environment variable changed, and no migration is required (SDK-161, PR #4373). * Takes subprocess-engine teardown off the response path for datasets served by subprocess-mode graph or vector databases with the dataset queue enabled. Previously, the last task to release a dataset's queue slot fetched the cached engine and awaited its `close()` inline, so an interactive search or recall waited for the worker process to shut down and drop its file lock before returning. Teardown now routes through the engine cache: the eviction still runs synchronously while the slot is held, so no caller can fetch a dying engine, but the adapter's `close()` runs on the cache's dedicated close threads. File-lock safety is preserved by the cache's pending-close latch instead of by making the caller wait — the next creation of the same engine waits until the previous worker has exited and released its lock, bounded, after which it falls back to the existing `SUBPROCESS_OPEN_LOCK_RETRIES` / `SUBPROCESS_OPEN_LOCK_BACKOFF` open retries. **Operator impact:** lower interactive search and recall latency for subprocess-mode datasets, with unchanged lock safety; a failing engine `close()` now surfaces in the logs with its traceback rather than propagating out of the dataset-context exit, so monitor logs for teardown warnings instead of relying on request errors to reveal them. No configuration option or environment variable changed and no migration is required (PR #4358). *** ## v1.4.1.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.1.dev1)** Development pre-release that bumps the package version from `1.4.1.dev0` to `1.4.1.dev1` and updates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself. No new Alembic revision ships in this cut; the entries below are the work merged to the development branch since v1.4.1.dev0. ### Highlights * Adds JSON siblings to the visualization endpoints, so an external dashboard can consume the same graph the built-in HTML page renders. `GET /api/v1/visualize/json` returns the full preprocessed payload — nodes, links, color maps, schema graph and schema data, pipeline stages, edge classes, bundles, provenance index, and memory map — plus `search_events`. `GET /api/v1/visualize/semantic` returns semantic positions and clusters for the same subgraph as a separate call, so a client that never opens the semantic view never pays for the embedding fetch and PCA behind it. `GET /api/v1/visualize/brains` returns every dataset the caller may read as a small `{dataset_id: {"name", "nodes", "links", "node_set_colors"}}` preview, with `max_nodes` (default `500`, maximum `5000`) applied independently to each dataset rather than as one larger shared cap. `GET /api/v1/visualize/live-events` returns search and improve events newer than a `since` cursor, for polling a timeline without re-fetching the whole payload; the filter is strict, so passing the previous response's `cursor` straight back never delivers an event twice. `GET /api/v1/schema/provenance/json` does the same for the memory-provenance graph. The matching Python entry points — `visualize_graph_json`, `visualize_semantic_json`, `build_brains_payload`, `get_live_events`, and `get_memory_provenance_payload` — are exported from `cognee.api.v1.visualize`. Authorization is unchanged and shared: every JSON route runs the same read-permission check as its HTML counterpart, and the HTML and JSON paths were refactored onto one `fetch_visualization_data` / `fetch_dataset_graph_data` pair so the two cannot drift on which subgraph they return. One scoping caveat on `live-events`: `dataset_id` gates who may call the endpoint, but the events themselves are collected per user rather than filtered to that dataset — the same events `/visualize/json` already embeds for that dataset. The existing `GET /api/v1/visualize` and `POST /api/v1/visualize/multi` endpoints are unchanged, and no configuration option or environment variable was added (CLO-401, PR #4331). * Fixes an explicit relational `POOL_ARGS` being silently ignored by PGVector when backend access control is enabled. `PGVectorAdapter` resolved pool arguments as `VECTOR_POOL_ARGS` → built-in access-control default → relational `POOL_ARGS`, so in multi-user mode the built-in `{"pool_size": 2, "max_overflow": 20}` — a deliberately small default, because each dataset gets its own engine and a large pool fans out as N datasets × `pool_size` — outranked an operator's explicit sizing, which therefore had no effect. Precedence is now `VECTOR_POOL_ARGS` → relational `POOL_ARGS` → the access-control default, with that default applying only when neither is set and multi-user mode is on (and an empty dict otherwise), so explicit configuration beats the built-in. If you set `POOL_ARGS` while running with access control enabled, PGVector now honors it — check that the resulting size is what you want before deploying, since per-dataset connection use can rise above the previous fixed default. No public API signature, configuration option, or environment variable changed (PR #4351). *** ## v1.4.1.dev0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.1.dev0)** Development pre-release that bumps the package version from `1.4.0.dev4` to `1.4.1.dev0` and updates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself. The release cut introduces no functional code, public API, configuration, or environment-variable changes; the entries below are the work merged to the development branch since v1.4.0.dev4. Most of them require no action, but two entries add Alembic revisions that existing deployments must run when upgrading — `c3d5e7f9a1b2` from the session-context deduplication fix and `d4e6f8a0b2c3` from the connector-credential `workspace_id` addition — see those entries for details. ### Highlights * Fixes batched triplet reads skipping triplets on the Neo4j and Ladybug graph backends, which could leave Memify's triplet-embedding pass with incomplete coverage. `get_triplets_batch(offset, limit)` is the paginated read behind `create_triplet_embeddings`, and `get_triplet_datapoints` walks the entire graph with one offset loop, advancing the offset by each batch's size until a batch comes back short or empty — which is only exhaustive if every call slices the same ordering. Both Cypher adapters placed `SKIP $offset LIMIT $limit` after `RETURN` on a `MATCH` with no `ORDER BY`, so consecutive pages were cut from an undefined result order and a multi-batch run could miss triplets (and index others twice), leaving the `Triplet_text` collection incomplete while still reporting success — most likely on large graphs, where the default `triplets_batch_size=100` means many pages. Both adapters now sort before paginating, with `ORDER BY` moved into a `WITH` clause ahead of `SKIP`/`LIMIT` so the skip applies to an already-ordered stream: `start_node.id, end_node.id, type(relationship)` on Neo4j and `start_node.id, end_node.id, relationship.relationship_name` on Ladybug. This aligns them with the SQL-backed Postgres and Turso adapters, which already ordered by `(source_id, target_id, relationship_name)` and are unchanged. No migration or user action is required beyond re-running `create_triplet_embeddings` to pick up triplets a previous run missed; the contents of any single batch are unchanged, but which rows land in which page now differs, so callers that paginate `get_triplets_batch` directly and relied on the previous (undefined) order should not assume the old grouping. No public API signature, configuration option, or environment variable changed (PR #4334). * Adds `opencode` as an agent connection type, accepted by the Python `register()` API and the `POST /api/v1/agents/register` endpoint's `type` field alongside the existing `sdk`/`api`/`mcp`/`claude_code`/`workflow`/`unknown` values, with idempotent registration, listing, and unregistration. The Cognee Cloud dashboard's **Get started** section gains an **OpenCode** connection card that walks through installing the Cognee plugin with `npx @cognee/cognee-opencode setup`; like the Claude Code and Codex cards, it auto-detects a new session — OpenCode sessions are recognized by the `opencode_` prefix the plugin puts on the session id. No existing connection type, API signature, or configuration option changed (PR #4333). * Extends the opt-in OpenTelemetry layer from spans-only to spans, metrics, and logs for the core memory operations, aligned to memory-semconv v0.1.0. `COGNEE_TRACING_ENABLED=true` (or `enable_tracing()`) now also configures a `MeterProvider` and attaches an OTel log bridge, reusing an externally configured provider when one exists; failures setting up either are swallowed, so a partial OpenTelemetry install degrades to traces-only. `add()`, `cognify()`, `search()`, and `forget()` emit spans named `memory.store`, `memory.process`, `memory.retrieve`, and `memory.delete` carrying `memory.system`, `memory.operation`, and — depending on the operation — `memory.collection`, `memory.query.text` (first 500 characters only, to cap cardinality and bound how much user input reaches your backend), `memory.query.type`, and `memory.result.count`; `add()` is instrumented for the first time. **This renames the existing `cognee.api.search`, `cognee.api.cognify`, and `cognee.api.forget` spans, so dashboards, saved queries, and alerts matching the old names must be updated** — span attributes stay backward compatible, since the `cognee.*` keys are still set alongside the new `memory.*` keys. Six metric instruments are recorded: `memory.operation.duration` (histogram, `ms`; on `forget()` only the delete-everything path records metrics), `memory.items.stored`, `memory.items.retrieved`, `memory.items.deleted`, `memory.query.result.count`, and `memory.vector.searches`; four more (`memory.data.bytes.stored`, `memory.graph.nodes.added`, `memory.graph.edges.added`, `memory.operation.errors`) are registered for custom instrumentation but not yet recorded by Cognee. Metrics and logs are only collected when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (or `console_output=True`) — unlike spans, they have no in-memory buffer — and their endpoints are derived from the traces endpoint by rewriting `/v1/traces` to `/v1/metrics` and `/v1/logs`, overridable with the standard `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. Separately, OTLP span export moves from `SimpleSpanProcessor` to `BatchSpanProcessor` (call `disable_tracing()` before exit to flush), and the HTTP-only exporter special case — previously just Langfuse's `/api/public/otel` — now also covers Dynatrace (`dynatrace.com`, `/api/v2/otlp`), port `:4318`, and endpoints with an explicit `:443` port followed by a path (`:443/`), which fixes Dynatrace export silently dropping traces under the gRPC exporter; note that an endpoint spelling out `:443` and including a path now exports over HTTP where it previously used gRPC. Everything remains fully opt-in and a no-op when disabled or when OpenTelemetry is absent, and no new environment variable was introduced (PR #4323). * Lets Cognee's own errors reach HTTP callers on `POST /api/v1/search`, `POST /api/v1/recall`, `POST /api/v1/remember` (including its COGX archive-import and `/entry` variants), and `POST /api/v1/improve` with their real status code and message. The four routers previously caught Cognee errors per type and rewrapped them into ad-hoc bodies such as `{"error": "...", "detail": "..."}` or `{"error": "...", "hint": "..."}`, sometimes replacing the actual message with unrelated advice — a recall permission failure, for example, returned `403` with a "Recall prerequisites not met" hint telling the caller to ingest and cognify first. The routers now re-raise every `CogneeApiError` subclass so the global handler in `cognee/api/client.py` answers with the error's own status code and the body `{"detail": "<message> [<ErrorName>]"}`. **Status codes are unchanged** (`402`/`403`/`404`/`409`/`422` all come from the exceptions themselves); only the response body shape changed, so clients parsing the legacy `error`/`hint` fields on these endpoints should switch to reading the HTTP status code and `detail` — see [Error Handling](/api-reference/introduction#error-handling). Two adjacent gaps were fixed in the same pass: recall's unreachable `except PermissionDeniedError: return []` arm was removed as dead code (it was always shadowed by an earlier catch, so the `200`-with-empty-list behavior it suggested never shipped), and a permission denial during a remember COGX archive import no longer collapses into a generic `409` "error occurred during COGX archive import". The generic fallbacks for unexpected, non-Cognee errors are unchanged (`500` with `{"error": "Internal server error", "detail": "..."}` for search; `409` with `{"error": "..."}` for recall, remember, and improve), and `POST /api/v1/cognify` and the LLM endpoints still return the legacy `{"error": "Token budget exhausted", "detail": "..."}` body on `402`. No public API signature, configuration option, or environment variable changed (SDK-255, PR #4285). * Shrinks the `cognee-mcp` server's advertised tool surface so agents no longer load the whole catalog into context. By default, `tools/list` now advertises 8 of the 10 registered tools — the memory API (`remember`, `recall`, `forget`), the three workspace UI entry points (`visualize_graph_ui`, `upload_file_ui`, `open_cognee_workspace`), plus FastMCP's synthetic `search_tools` and `call_tool` — while the structured-JSON workspace helpers are found by calling `search_tools(query=...)` (BM25 ranking, up to 10 results) and invoked by name or through the `call_tool` proxy. Hiding a tool never makes it unreachable: unadvertised tools stay directly callable, which is what keeps the workspace UI (which calls its internals by name) working. The new `COGNEE_MCP_TOOL_MODE` environment variable — also settable per-process with `--tool-mode` — selects `default` (the surface above), `minimal` (only the three memory tools plus the search pair), or `all` (the previous flat catalog with no search transform); unknown values log a warning and fall back to `default`. One sharp edge: `search_tools` matching is purely lexical with no stemming, and zero-scoring tools are dropped, so a terse query like `dataset` misses `list_datasets_json` (token `datasets`) — multi-word natural-language queries are the reliable form. In the same rework, the `cognify_file` tool was removed and folded into `remember`, which now accepts `filename` + `content_base64` (up to 10 MB) to ingest an uploaded file alongside the existing `data` text form; session-cache writes (with `session_id`) remain text-only (PRs #4283 and #4258). * Makes `cognee-cli config` a working, persistent interface. `cognee-cli config get <key>` and bare `config get` previously printed "not implemented" — they were gated on `hasattr` checks for methods that never existed — and now return real values through the new `cognee.config.get(key)` and `cognee.config.get_all()` APIs, masking secret values (`llm_api_key`, `embedding_api_key`, `vector_db_key`) by default; pass `--show-secrets` on the CLI or `reveal_secrets=True` in Python to print them in plaintext. `config set` and `config unset` previously mutated only the in-process `@lru_cache`d config singletons, so a value vanished the moment the CLI process exited; the CLI now calls `cognee.config.set(key, value, persist=True)`, which additionally writes the value under its resolved environment-variable name to the `.env` file in the current working directory (creating the file if needed) — the same file every config class already reads — so values survive across CLI invocations and are picked up by the next process started from that directory. `unset` resets a key to its default through the same persisted path. The `persist` parameter is new and defaults to `False`, so SDK callers of `config.set()` keep the previous in-memory-only behavior; unknown keys now raise `InvalidConfigAttributeError` on both paths (COG-5970, PR #4287). * Gives graph nodes a notion of fact validity over time. Every `DataPoint` gains a `valid_to` field (milliseconds-epoch integer, default `None` = still current) — distinct from the temporal stack's Event/Interval `time_to`, which records when an event happened, not whether the fact still holds. The new `close_node(node_id, at_ms=None)` helper (`cognee/tasks/storage/close_node.py`) stamps `valid_to` on an existing node when a fact is superseded — mark "Alice works at X" closed and write the replacement, instead of deleting the old node — and `is_valid(node, at_ms=None)` reports whether a node is still current (`valid_to` is `None` or in the future), accepting both `DataPoint` instances and plain graph-node dicts. Persistence goes through a new optional `update_node(node_id, values)` extension on `GraphDBInterface`, currently implemented only on the default Ladybug (Kuzu) adapter with a read-modify-write that leaves fields the caller did not name untouched; on backends without it, `close_node` logs a warning and returns `False` rather than failing silently (`True` means the node existed and was updated). Closing is last-write-wins — re-closing overwrites the earlier stamp, so guard with `is_valid` first where idempotence matters. No existing API signature, configuration option, or environment variable changed (SDK-200, PR #4105). * Turns the client-side LLM RPM limiter on automatically when the provider shows evidence it cannot keep up. Previously the limiter was purely opt-in: unless you set `LLM_RATE_LIMIT_ENABLED=true`, Cognee dispatched unbounded and a busy `cognify` run could keep hammering a provider that was already rejecting requests. A new overload policy (`cognee/infrastructure/llm/overload_policy.py`) watches the dispatch seam in `cognee/shared/rate_limiting.py`; when a failed dispatch carries overload evidence anywhere in its exception cause chain — a rate-limit error, a timeout (how overwhelmed local servers surface, since they never send rate limits), or HTTP `429`, `503`, or `529` — it logs one warning naming the cause and the budget, and paces every subsequent dispatch with `LLM_RATE_LIMIT_REQUESTS` / `LLM_RATE_LIMIT_INTERVAL` for a 900-second cooldown. Further evidence inside that window extends it silently; once the window lapses quietly, behavior returns to the configured state, and a fresh episode warns again. Because adapters enter the limiter inside their retry loop, retried attempts are paced too. The new `AUTO_RATE_LIMIT` environment variable (default `true`) controls this; set it to `false` to keep the old fully unbounded behavior, or keep using `LLM_RATE_LIMIT_ENABLED=true` to pace from the first request regardless. Response latency alone is not treated as overload evidence. Relatedly, when `LLM_RATE_LIMIT_REQUESTS` is not set explicitly, serial local inference servers (Ollama, llama.cpp by provider; LM Studio by the `lm_studio/` model prefix) now default to `10` requests per interval instead of `60`, since the cloud default would still flood them; vLLM is deliberately classed as a regular provider and keeps `60`. An explicitly configured `LLM_RATE_LIMIT_REQUESTS` always wins. User impact: runs against a strained provider now slow down instead of failing, at the cost of taking longer; requests already dispatched when the limiter engages cannot be un-queued (PR #4240). * Raises the default `chunks_per_batch` for the standard Cognify pipeline from `100` to `2000`. `get_default_tasks` in `cognee/api/v1/cognify/cognify.py` uses this value as the `batch_size` for the graph extraction/summarization task and for `add_data_points`, so larger batches mean fewer, bigger chunk-level task batches per run. An explicit `chunks_per_batch` argument and the `chunks_per_batch` value from `CognifyConfig` both still take precedence, and the temporal Cognify pipeline default is unchanged at `10` (PR #4240). * Scopes the default session to the dataset, so omitting `session_id` no longer funnels every dataset's turns into one shared conversation. `SessionManager.resolve_session_id` (now public, pure, and synchronous — it performs no database lookups) returns an explicit `session_id` unchanged, but when none is given and the manager knows its dataset — from the `dataset_id` constructor argument or the `current_dataset_id` context variable that dataset-scoped `recall()`/`search()` calls enter — it derives `default_session_<dataset_id>` instead of the single global `"default_session"`. With no dataset known the plain global default is still used, exactly as before. Both the read and write sides route through the same function, so an omitted `session_id` resolves identically in either direction within a dataset context. **Behavior change:** history that previously accumulated under the shared `default_session` is now written per dataset; existing `default_session` entries are not migrated and stay readable by passing the literal `session_id="default_session"`. `cognee.session.get_session`'s `session_id` default changed from `"default_session"` to `None` to defer resolution to the manager; a bare call outside any dataset context rebuilds its manager scoped to the caller's existing `main_dataset` (a read-only lookup — no dataset is created) so dataset-scoped writes are readable back without stating a dataset, and raises a `SessionPreconditionError` (a `CogneeValidationError`) when no `main_dataset` exists rather than silently returning the unscoped global session. Relatedly, `set_database_global_context_variables` now accepts exactly `Optional[UUID]` and rejects dataset names and UUID *strings* with a `CogneeValidationError` (`SessionManager` likewise rejects a non-UUID `dataset_id` with `SessionParameterValidationError` instead of degrading to an unscoped session); every production call site already passed a UUID object, but automation or tests that entered the context by dataset name must now resolve the dataset and pass its `.id`. In the CLI, the per-user `scoped_session_id` scheme (`"<user_id>:default"`) is removed from `search`, `recall`, `feedback`, and `sessions` — the dataset-scoped default replaces it, so `cognee search` now passes `session_id=None` and the explicit `--session-id` values you pass to the other commands are used verbatim rather than being prefixed with your user id. No configuration option or environment variable changed (SDK-255, PR #4286). * Fixes duplicate session-context rows in the SQL session-cache backends (`CACHE_BACKEND=sqlite`, the default, or `postgres`) permanently breaking a session with a 503. `create_session_context_entry` appended a row on every call, so racing update-then-create writers (for example the session persist watermark) accumulated several rows for the same `(user_id, session_id, entry_id)`; `update_session_context_entry` then resolved that key with `scalar_one_or_none()` and raised `MultipleResultsFound`, which surfaced as a per-session 503 that recurred on every subsequent turn of that session. Creates now issue a dialect-specific `ON CONFLICT (user_id, session_id, entry_id) DO UPDATE` upsert (last writer wins, refreshing `payload` and `expires_at`), so those flows converge on a single row, and the update path resolves to the newest row (`ORDER BY seq DESC LIMIT 1`) instead of raising while its `UPDATE` still rewrites every straggler duplicate with the merged payload. The `cache_session_context` table gains a `uq_cache_session_context_entry` unique index on that key — the index the upsert targets. Cache tables are created on init rather than managed by Alembic, so **fresh** databases get the index from the table definition and need no action, but an **existing** deployment must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) before the upsert has an index to conflict on: new Alembic revision `c3d5e7f9a1b2` deletes all but the newest (`MAX(seq)`) row per key and creates the index, covering both the Alembic-connected database and the standalone SQLite `cache.db` that the default backend keeps next to the relational database. Because the migration mutates data and builds a unique index, it can take time and hold locks on a large `cache_session_context` table — back up the database and run it in a maintenance window. The migration has deliberately no runtime fallback: if `CACHE_DB_URL` points at a separate non-SQLite database it cannot reach, it raises `RuntimeError` naming that URL and printing the `DELETE` / `CREATE UNIQUE INDEX` statements to apply there manually before re-running, rather than letting a deployment proceed against a cache database the upsert would break on. No public API signature, configuration option, or environment variable changed (fixes #4226, PR #4232). * Fixes cancelled requests leaking Postgres connections stuck in `idle in transaction`, which could exhaust the connection pool until every query failed. `SQLAlchemyAdapter.get_async_session()` in `cognee/infrastructure/databases/relational/sqlalchemy/SqlAlchemyAdapter.py` ended in `finally: await session.close()`, which runs before the `async_sessionmaker` context manager's own exit; when the surrounding task was being cancelled — a client disconnect, a timeout, or any `asyncio` cancellation — that `await` was interrupted before the `ROLLBACK` reached Postgres, so the connection went back to the pool still inside an open transaction. Enough of those piled up and the pool hit its ceiling, at which point requests began failing across the board (authentication included, since the API-key lookup is itself a DB query). The explicit `close()` is removed and replaced with an `except asyncio.CancelledError` branch that calls `await asyncio.shield(self._discard_cancelled_session(session))` and then re-raises; the new `_discard_cancelled_session()` static helper calls `await session.invalidate()`, which drops the DBAPI connection from the pool without needing the `ROLLBACK` round-trip that cancellation keeps interrupting, and logs (rather than raises) if the invalidation itself fails. The `asyncio.shield` keeps the cleanup from being cut short by the same cancellation. Successful requests and ordinary exceptions are unaffected — the sessionmaker's own context manager already closes and rolls back on those paths. No public API signature, configuration option, environment variable, or database migration changed; no pool-sizing changes are needed beyond deploying the fix (fixes #4197, PR #4250). * Fixes Zep/Graphiti imports silently dropping scope from every entity node and every fact when the export spells the scope key `session_id` rather than `group_id`. In `ZepSource` (and its `GraphitiSource` alias), the episode branch already resolved the record's scope as `group_id or session_id`, but the entity and fact branches built `COGXScope(session_id=...)` from `group_id` alone — so a single export using the `session_id` spelling imported with scope intact on its episodes and empty on all of its nodes and edges, with nothing raised. Both branches now apply the same fallback (`node.get("group_id") or node.get("session_id")`, `edge.get("group_id") or edge.get("session_id")`), and `group_id` keeps precedence where a record carries both, matching the episode branch. User impact: exports already keyed on `group_id` are unaffected and import exactly as before; if you imported a `session_id`-keyed Zep or Graphiti export — most consequentially into a multi-tenant graph, where entities and facts lost their tenant attribution — re-run that import to restore scope on the affected nodes and edges. No public API signature, configuration option, environment variable, or migration changed (PR #4293). * Fixes Mem0 and LangMem imports landing zero memories when the payload carries an accepted wrapper alias that is present but empty ahead of a populated one. `Mem0Source._load_raw` and `LangMemSource._load_raw` unwrapped on the *first* accepted alias that happened to be a list — `results`, `memories`, `items` for Mem0, and `memories`, `results`, `items`, `data` for LangMem — so a payload like `{"results": [], "memories": [...]}` unwrapped to the empty `results` list and never read the populated alias, and the import still reported success with nothing imported. Both sources now scan their alias order and return the records under the first alias that actually carries any, filtering to dict items before deciding an alias is populated — the same rule `ZepSource`'s `_first_list` helper already applied. Error behavior is preserved: a dict where none of the accepted aliases maps to a list, and a payload that is not a list, both still raise `ValueError`, while a recognized wrapper whose aliases are all empty still yields zero records without raising. User impact: payloads whose first accepted alias was already the populated one are unaffected and import exactly as before; if a Mem0 or LangMem import reported success but landed zero memories, re-run that import. No public API signature, configuration option, environment variable, or migration changed (PR #4314). * Fixes `improve()` silently retargeting the caller's default dataset when the `dataset` argument could not be resolved. Previously an internal `_resolve_dataset_name` helper returned `"main_dataset"` for any unresolved reference, so a mistyped, nonexistent, or unauthorized dataset UUID quietly enriched the caller's own default dataset instead of failing, and individual stages re-resolved the dataset independently. The target dataset is now resolved and authorized once, up front, through the same `resolve_authorized_user_datasets` layer that `remember()` and `memify()` use: a UUID the caller does not hold `write` permission on raises `PermissionDeniedError` — read permission is not enough, since every improvement stage writes — and a UUID that does not exist raises the identical error, so dataset existence is never confirmed to an unauthorized caller. Dataset **names** keep their owner-scoped semantics: a name is resolved — or created — in the caller's own scope, so improving a name another user happens to own creates the caller's own dataset with that name rather than touching theirs; a dataset owned by someone else must be targeted by UUID. The session-bridging stages now receive the resolved dataset UUID instead of re-resolving a name. User impact: if your integration passed `improve()` a bad or unauthorized dataset reference and relied on the silent `main_dataset` fallback, the call now raises `PermissionDeniedError` — catch it and pass a UUID you hold `write` permission on. The `improve()` signature, configuration options, environment variables, and migrations are unchanged (SDK-255, PR #4294). * Adds an optional `workspace_id` owner dimension to the third-party connector credential store, so several users can share one connection instead of each connection belonging to exactly one user. `upsert_credential` in `cognee/modules/integrations/credentials.py` gains a keyword-only `workspace_id: Optional[UUID] = None`; because the `integration_credentials` table is keyed on the external account (`UNIQUE(provider, provider_account_id)`), a reconnect by a *different* owner while the current connection is still active raises `CrossUserConflictError` rather than silently taking the connection over, and `workspace_id` now decides who that owner is. When it is passed, the conflict comparison runs on `workspace_id` — two different users in the same workspace can reconnect the same external account, while a different workspace is still refused — and `user_id` continues to record which user actually connected it. Omitted (the default), the ownership and conflict rules are exactly the previous single-user contract, compared on `user_id`. A companion `get_active_credential_for_workspace(workspace_id, provider)` mirrors the existing `get_active_credential_for_user(user_id, provider)` for the new dimension, filtering on `status == "active"` and ordering newest-first; use it for connections upserted with a `workspace_id`, since `user_id` on those rows is only who connected them, not the owner. The `IntegrationCredential` model gains a matching `workspace_id` column — `nullable`, indexed, and deliberately **not** a foreign key (a plain opaque id, same as `user_id`, so there are no cascade semantics) — backed by new Alembic revision `d4e6f8a0b2c3` (on top of `c3d5e7f9a1b2`), which adds the column and the `ix_integration_credentials_workspace_id` index. An **existing** deployment must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) to pick up the column; the revision is additive and both its `upgrade()` and `downgrade()` inspect the live schema first, so re-running either is a no-op. Existing rows keep `workspace_id` as `NULL` and behave as before. This module is internal plumbing rather than part of the public SDK: no public API signature, configuration option, or environment variable changed (CLO-406, PR #4318). *** ## v1.4.0.dev4 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev4)** Development pre-release that bumps the package version from `1.4.0.dev3` to `1.4.0.dev4`; the accompanying `uv.lock` change records the new version only. No new Alembic revision ships in this cut. Relative to v1.4.0.dev3 this build also restores the v1.4.0.dev2 changes (including the OAuth integrations framework and its `b2c4d6e8f0a1` migration) that the dev3 release branch did not carry. ### Highlights * Adds an `embedding_max_concurrent_data_points` setting (default `150`, env var `EMBEDDING_MAX_CONCURRENT_DATA_POINTS`, also settable via `cognee.config.set_embedding_config({"embedding_max_concurrent_data_points": ...})`) controlling how many data points may be in flight to the embedding engine during indexing. `index_data_points` in `cognee/tasks/storage/index_data_points.py` previously capped concurrency with a hardcoded `asyncio.Semaphore(4)` — four concurrent batches regardless of batch size; the semaphore is now sized `max(1, embedding_max_concurrent_data_points // batch_size)`. The default changes effective throughput: with the LiteLLM engine's default `batch_size=100`, indexing now runs one batch at a time (about 100 points in flight) instead of four (about 400) — raise `EMBEDDING_MAX_CONCURRENT_DATA_POINTS` to restore or exceed the old parallelism, or lower it to stay under provider rate limits. `embedding_batch_size` and the embedding rate-limit settings are unchanged, and no public API signature changed (SDK-285, PR #4144). * Fixes the `dry_run=True` cost estimator applying the `ACCEPT_LOCAL_FILE_PATH` gate inconsistently across platforms. In `cognee/modules/cognify/estimator.py`, `_path_candidate` decided whether an existing file path was *absolute* — and therefore rejected when the flag is disabled — with a bare `value.startswith("/")` check. That check does not describe absolute paths on Windows, so a genuine Windows path like `C:\data\notes.txt` was not recognized as one: with `ACCEPT_LOCAL_FILE_PATH=False` it slipped past the gate and the path string itself was priced as raw text instead of raising, while conversely a `/`-prefixed string — which Windows resolves to a *drive-relative*, unusable path — was rejected even though a real run ingests it as text. Detection now matches `save_data_item_to_storage` exactly: the string must look absolute for the current platform (`/`-prefixed, or drive-lettered when `os.name == "nt"`) **and** satisfy `Path(os.path.normpath(value)).is_absolute()`. On Windows, `C:\...` paths that exist now raise `Local files are not accepted` when the flag is disabled, and drive-relative `/`-prefixed strings fall through to raw text; POSIX behavior is unchanged. Existing relative paths and strings that are not existing files remain raw text on every platform, and enabling the flag (the default) is unaffected. The rest of the PR is unit-test repair with no runtime effect — a missing `SAMPLE_ARGUMENTS` entry for `ProviderNotDeducibleError` in the error-contract test, and an autouse fixture that snapshots and restores patched module attributes so mocks stop leaking between test modules. No public API signature, configuration option, environment variable, or migration changed (PR #4257). * Fixes Letta imports silently dropping every message — and the whole conversation episode — when a message serializes `content` as `null` and carries its text under the `text` alias. In `LettaSource`, `_message_text` resolved the alias with `message.get("content", message.get("text"))`, but `dict.get`'s default only fires on a *missing* key: a Letta serializer that writes unset fields as `null` instead of omitting them produced `content: None`, so the `text` fallback never ran and the message resolved to `""`. The caller skips any message with no text, and because every message in one agent file shares the same serialization, the agent's turn list ended up empty and the `if turns:` gate emitted no `COGXEpisode` at all — the import still reported success. The fallback is now explicit (`content = message.get("content")`, then `if content is None: content = message.get("text")`), so those messages and their episode are imported. Files that were already importing correctly are unaffected, and an explicitly empty `content` (`""`) is still treated as a message with no text rather than falling through to `text`. User impact: if you imported Letta agent files and found conversation history missing, re-run those imports to recover the episodes. No public API signature, configuration option, environment variable, or migration changed (PR #4261). * Fixes `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` failing at query time on installations that do not have the optional `postgres` extra. Both retrievers began by checking whether the active graph engine is one of the Postgres graph backends, which they reject with `SearchTypeNotSupported` because those backends cannot run Cypher. That check imported the Postgres adapters (`PostgresAdapter`, `PostgresHybridAdapter`), and those adapter modules import `asyncpg` at module scope — a dependency that ships only with `cognee[postgres]` — so on an installation without the extra the check itself raised `ImportError`/`ModuleNotFoundError` rather than the backend it was trying to detect. In `NaturalLanguageRetriever.get_retrieved_objects` the import was unguarded, so the error propagated to the caller; in `CypherSearchRetriever.get_retrieved_objects` it sat inside the block whose handler wraps unexpected exceptions, so it surfaced as a misleading `CypherSearchError`. The check no longer imports backend adapters at all: Cypher capability is now declared on the adapter class itself — `GraphDBInterface` defines `supports_cypher_queries = True`, the Postgres, Postgres hybrid, and Turso adapters override it to `False` — and both retrievers read the flag off the active engine, raising `SearchTypeNotSupported` (naming the rejected adapter class) when it is false. User impact: on a Cypher-capable backend such as Kuzu or Neo4j, both search types now work without installing `cognee[postgres]`; installations that do have the extra are unaffected, and Postgres graph backends still raise `SearchTypeNotSupported` as before. The Turso graph backend — whose `query()` also runs SQL rather than Cypher — is newly covered by the same check, so it now raises `SearchTypeNotSupported` up front instead of failing when the generated Cypher reaches it. No public API signature, configuration option, environment variable, or migration changed (fixes #4123, PRs #4124 and #4274). *** ## v1.4.0.dev3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev3)** Development pre-release that bumps the package version from `1.4.0.dev2` to `1.4.0.dev3` and re-resolves `uv.lock`. Unlike the other pre-releases in this line it was cut from a release branch rather than the tip of the development branch, so it carries the deployment and Ollama-validation work below while omitting some changes already shipped in v1.4.0.dev2 — most notably the OAuth integrations framework and its `b2c4d6e8f0a1` migration are absent from this build; both are included again in v1.4.0.dev4. ### Highlights * Adds an advisory Ollama model-support warning and ships S3 support in the default Docker image. When `LLM_PROVIDER="ollama"`, a new `cognee/infrastructure/llm/ollama_support.py` helper classifies the configured `LLM_MODEL` against a built-in support matrix as the LLM configuration is validated and logs a warning for models Cognee has not validated for structured graph extraction: `llama3` through `llama3.3` and `qwen2.5` tags of 14B or larger are recommended (nothing logged); `mistral`, `phi3`, `phi3.5`, and `qwen2.5` below 14B or with no parseable size warn about known limitations (schema validation errors, silent drops); every other model warns that it is unvalidated and extraction quality may vary. The check is advisory only — nothing is blocked, no exception is raised, and each distinct `LLM_MODEL` value warns at most once per process. Matching ignores an `ollama/` prefix and everything after the `:` in a tag (except `qwen2.5`, whose tag is parsed for a parameter count), and the message points at the new `docs/ollama_models.md` in the repo; see [Model Support Warning](/setup-configuration/llm-providers#model-support-warning) on the LLM Providers page for the full matrix. Separately, the root `Dockerfile` now includes the `aws` extra (`s3fs[boto3]`) in both `uv sync` steps, so the source-built API image supports [S3 file storage](/guides/s3-storage) out of the box without passing `COGNEE_EXTRAS="aws"`. The access-control handler compatibility check also now accepts the `ladybug`/`kuzu` provider aliases interchangeably — both handler names register the same embedded Ladybug (formerly Kuzu) handler and both provider names satisfy its check — so mixing the current and legacy names no longer raises `EnvironmentError` at startup under backend access control. The branch also adds runnable examples: `examples/demos/local_ollama_example.py` (fully local add → cognify → search with Ollama; since moved to `examples/guides/local_ollama_example.py`) and `examples/tutorials/migrate_from_mem0_tutorial.py` with a bundled sample export (since moved to `examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py`). No public API signature, configuration option, or environment variable changed (PR #4272). *** ## v1.4.0.dev2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev2)** Development pre-release that bumps the package version from `1.4.0.dev1` to `1.4.0.dev2`; the `uv.lock` change records the new version only. One entry requires action: the new OAuth integrations framework ships Alembic revision `b2c4d6e8f0a1`, which existing deployments must run when upgrading (`cognee.run_migrations()`, or `alembic upgrade head`) — see that entry for details. ### Highlights * Adds an opt-in `memify` pipeline, `consolidate_entities`, that merges near-duplicate `Entity` nodes left behind by repeated `cognify` runs or multi-source ingestion (e.g. "New York City" vs "NYC", whose name-derived ids never collapse on their own). `consolidate_entities_pipeline` (in `cognee/memify_pipelines/consolidate_entities.py`) runs two new tasks exported from `cognee.tasks.memify`: `detect_entity_duplicates` embeds each entity name and clusters candidates by cosine similarity (`similarity_threshold=0.85` default, `top_k=10` neighbors) plus normalized-name equality (`name_match=True`), and `merge_entity_duplicates` picks the canonical node (most connected, ties broken by oldest then name), re-points every edge onto it with direction preserved, unions descriptions, records a `merged_from` report, then deletes the duplicate nodes and their `Entity_name` vector embeddings. Merging stays within one `EntityType` unless `allow_cross_type=True`; `protect_node_types` excludes types entirely, and `dry_run=True` logs the plan without mutating anything. The pipeline only runs when invoked — existing pipelines, public APIs, configuration options, and environment variables are unchanged (PR #3534). * Fixes `cache_root_directory` in `BaseConfig` (`cognee/base_config.py`) silently accepting relative paths. The sibling root directories — `data_root_directory`, `system_root_directory`, and `logs_root_directory` — were already normalized through `ensure_absolute_path()`, which resolves absolute paths, passes `s3://` URLs through untouched, and raises `ValueError` on a relative path; `cache_root_directory` alone skipped that check, so a relative cache path would resolve differently depending on the current working directory, a hard-to-debug misconfiguration. The one-line change adds the same `ensure_absolute_path()` call for `cache_root_directory`, placed after the S3 auto-config logic so S3 cache paths keep working. Action needed only if you currently set a relative `cache_root_directory`: it now raises `ValueError` at configuration time and must be an absolute path (or an `s3://` URL). No new configuration option, environment variable, public API signature, or migration is involved (PR #3581). * Adds a `COGNEE_PROVENANCE_MODE` environment variable (also readable from `.env`) controlling provenance stamping on DataPoints during pipeline runs, backed by the new `cognee/modules/pipelines/provenance_config.py` (`ProvenanceConfig`, cached via `get_provenance_config()`). Valid values are `lightweight` (default), `deep`, and `disabled`; values are lowercased, and an unknown value logs a warning and falls back to `lightweight`. In the shipped gating, `disabled` skips the `_stamp_provenance` call in `run_tasks_base.py` entirely as a zero-overhead escape hatch, while `lightweight` and `deep` both run the existing stamping (the config exposes `is_lightweight()`/`is_deep()` helpers, but no code path yet branches on them). PR #3775 as merged left `run_tasks_base.py` half-edited — a broken two-argument `_stamp_provenance(result_data, pipe_name)` call raised `TypeError: missing 1 required positional argument: 'task_name'` on every pipeline run alongside the surviving ungated call — and PR #4215 (SDK-317) repaired this before release with a single full-argument call gated on `get_provenance_config().is_disabled()`. Default behavior is unchanged; no action needed unless you want `disabled` (PRs #3775 and #4215). * Fixes `LangchainChunker` (`cognee/modules/chunking/LangchainChunker.py`), which could not be instantiated at all after the chunker API unification even though it is user-facing — documented in the `cognify()` docstring and offered among the CLI chunker choices — so selecting it crashed mid-pipeline. Three stacked breaks are repaired, re-landing the alignment from PR #2966 that was reverted in #3167: the constructor parameter is renamed from `max_chunk_tokens` to `max_chunk_size` (every `Document.read()` call passes `max_chunk_size=`, which previously raised `TypeError`), `super().__init__` now passes the three arguments the `Chunker` base actually accepts, and `read()` checks `self.max_chunk_size` and emits `DocumentChunk` with `chunk_size=token_count` plus `importance_weight` propagation — matching `TextChunker` — instead of the undefined `word_count`/`token_count` fields while omitting the required `chunk_size`. New regression tests are guarded with `pytest.importorskip("langchain_text_splitters")` so environments without the `langchain` extra skip cleanly. `TextChunker` behavior, configuration options, and environment variables are unchanged (fixes #3888, PR #3893). * Changes LLM configuration to reduce required setup: `LLM_PROVIDER` is now optional and inferred from the `llm_model` prefix (e.g. `anthropic/claude-...` implies `anthropic`) via a new validator in `cognee/infrastructure/llm/config.py`; an explicit provider — kwarg or env var — always wins, and a prefix outside `KNOWN_LLM_PROVIDERS` raises `ProviderNotDeducibleError` with a message naming the supported providers and pointing litellm-routed prefixes (e.g. `openrouter/`, `groq/`, `deepseek/`) at `LLM_PROVIDER="custom"`. Behavioral change requiring action: such prefixes previously defaulted to `openai` passthrough and now fail fast until you set `LLM_PROVIDER="custom"`. Additionally, a central `instructor_modes.py` table replaces the `default_instructor_mode` values duplicated across nine adapters; the `embedding_rate_limit_*` fields move from `LLMConfig` to `EmbeddingConfig` (field and env-var names unchanged); quote-stripping now covers all declared string fields instead of a 14-field allow-list; and the Ollama validator no longer checks embedding env vars, leaving that to `EmbeddingConfig` (SDK-142, part of #3382, PR #3994). * Adds the catalog contract underpinning the planned Integrations Hub and Use-Case Gallery: a new top-level `catalog/` directory in the cognee repo with a draft-07 JSON Schema (`catalog/schema.json`), a validating loader (`catalog/loader.py`, runnable as `python -m catalog.loader`) that applies three passes — schema validation, naming rules, local path resolution — with aggregated errors, ten seed YAML entries under `catalog/entries/` spanning all three kinds (integrations such as `claude-code` and `langgraph`, packages such as `qdrant` and `weaviate`, and use-cases such as `agent-memory` and `temporal-reasoning`), an advisory cross-repo drift check (`catalog/inventory_sync.py`) against the `cognee-integrations` repository's `inventory.yml`, a `Catalog` CI workflow (`.github/workflows/catalog.yml`), and a contributor guide (`docs/contributing/add-catalog-entry.md`). This is chunk 1 of issue #3603; Hub/Gallery rendering and full cross-repo aggregation are deferred to follow-ups. The catalog tooling is not shipped in the wheel, so the installed `cognee` package, public APIs, configuration options, and environment variables are unchanged (SDK-151, PR #4023). * Adds opt-in temporal contradiction resolution for functional (single-valued) relationships to the cognify pipeline. When a relationship like `ceo_of` should hold only one target per subject but the graph accumulates several — e.g. one document names Alice as CEO and a later one names Bob — cognee previously kept both as current facts. Passing the new keyword `cognify(functional_relationships={"ceo_of"})` appends a `resolve_temporal_contradictions` task (in `cognee/tasks/graph/resolve_temporal_contradictions.py`) that runs last, after the graph is written: for each declared relationship it groups stored edges by subject, keeps the most recent assertion (by `updated_at`) as current, and tags the older ones with `superseded=True`, `superseded_by` (the winner's `edge_object_id`), and `supersession_reason` — nothing is deleted and provenance is preserved. The core, `tag_superseded_edges` in `cognee/modules/graph/utils/temporal_conflict_resolver.py`, is deterministic with no LLM call, resolution runs against the stored graph so a fact ingested today supersedes one from last month, and re-runs are idempotent. The parameter defaults to `None`, so the default pipeline, `add_data_points`, and all existing behavior are unchanged; no environment variable or migration is involved (SDK-184, PR #4084). * Adds a Graph Insight Report that describes a built knowledge graph instead of only drawing it: a Markdown report covering hub nodes ("god nodes", ranked by degree plus `networkx.pagerank` over the entity/entity-type layer, with a degree-only fallback), surprising cross-set connections (entity pairs whose endpoints belong to different `node_set`s, resolved through `belongs_to_set` edges), edge provenance (EXTRACTED entity-to-entity relationships vs DERIVED chunk/document scaffolding), and LLM-suggested follow-up questions ready to pipe into `search()` — the one section that costs an LLM call. Exposed three ways: a new public `cognee.report(datasets="main_dataset", output_path="graph_report.md", top_n=10)` API in `cognee/api/v1/report/report.py`, a new `SearchType.GRAPH_REPORT` backed by `GraphReportRetriever` in `cognee/modules/retrieval/graph_report_retriever.py`, and a `cognee-cli report` command. The feature is additive-only: it reads solely via `get_graph_data()`, so there are no schema, storage, or migration changes, and no existing API signature, configuration option, or environment variable changed (SDK-188, PR #4101). * Adds opt-in contradiction detection between newly ingested facts and facts already stored in the graph. When enabled, cognify appends a `detect_contradictions` task (in `cognee/tasks/graph/detect_contradictions.py`) that runs last: it collects the entities the current ingestion touched, fetches their 1-hop stored neighbourhood via `get_neighborhood`, asks the LLM which fact pairs conflict, and records each contradiction as a queryable `contradicts` edge carrying both facts, the reason, and a confidence score, alongside a logged warning — nothing is deleted or overwritten, and the task swallows its own errors so detection can never break ingestion. It is enabled through configuration rather than a new function argument: `CONTRADICTION_DETECTION=true` on `CognifyConfig` (`cognee/modules/cognify/config.py`), tuned by `CONTRADICTION_CONFIDENCE_THRESHOLD` (default 0.5) and `CONTRADICTION_MAX_FACTS` (default 500); it also applies to `remember()`, which builds its graph through `cognify()`. Default off — with the flag unset the pipeline task list and behavior are identical to before, and no public API signature changed (SDK-199, PR #4104). * Fixes three `CogneeError` subclasses that overrode `__init__` without calling `super().__init__()`, leaving `Exception.args` empty — which broke `repr()`, exception chaining (`raise ... from cause`), and the centralized logging in `CogneeApiError.__init__`. The offenders were `EntityNotFoundError` and `NodesetFilterNotSupportedError` in `cognee/infrastructure/databases/exceptions/exceptions.py` and `WrongTaskTypeError` in `cognee/modules/pipelines/exceptions/tasks.py`; an AST sweep confirmed these were the only three in the repo. All now call `super().__init__(message, name, status_code)`; the two database exceptions pass `log=False` because they are raised in routine control flow (user resolution, migrations, pruning, graph projection), so `args` and chaining are restored without flooding logs with ERROR lines for expected conditions, while `WrongTaskTypeError` — a genuine programming error — keeps default logging. A new regression guard, `cognee/tests/unit/exceptions/test_cognee_error_contract.py`, enforces the contract via both a static AST sweep and a runtime check of every importable subclass. Constructor signatures, exception types, and raising sites are unchanged, so no caller needs action (SDK-240, fixes #3749, PR #4151). * Fixes ingestion turning a `/`-prefixed string that is not an actual file — e.g. a text note like `/remember to call Bob about the meeting` — into a broken `file://` URI pointing at a non-existent path, which the loader later failed on. The absolute-path branch of `save_data_item_to_storage` (in `cognee/tasks/ingestion/save_data_item_to_storage.py`) converted unconditionally; it now also requires `abs_path.is_file()`, mirroring the relative-path branch, so an absolute-looking string becomes a `file://` URI only when it points at an existing file and otherwise falls through to text ingestion on every platform. Existing absolute file paths still convert as before, and `ACCEPT_LOCAL_FILE_PATH=false` still rejects existing local files with `IngestionError`. The dry-run estimator's `_path_candidate` (`cognee/modules/cognify/estimator.py`), which mirrors this routing, is updated to price a missing absolute path as text instead of raising "file does not exist". Side effect of keying on "existing file": directory paths, broken symlinks, and pathological paths also route to text rather than a broken URI. Follow-up to #3892; no public API signature, configuration default, or migration changed (SDK-234, fixes #3887, PR #4155). * Adds a reusable OAuth integrations framework to the cognee server, with Slack as the first provider. The framework (`OAuthIntegration` ABC and registry in `cognee/modules/integrations/`, a generic per-provider connect/callback/disconnect router in `cognee/api/v1/integrations/routers/get_integrations_router.py`) stores third-party credentials AES-256-GCM-encrypted in a new `integration_credentials` table, with key-rotation support via the `INTEGRATION_CREDENTIALS_KEYS` / `INTEGRATION_CREDENTIALS_ACTIVE_KEY_ID` env vars (legacy single-key `INTEGRATION_CREDENTIALS_KEY` still works). The Slack integration adds workspace connect/disconnect with token refresh and revoke, a `/cognee-ask` slash command answering via `HYBRID_COMPLETION` search through Slack's `response_url`, a per-channel allowlist, an App Home tab, a "Remember this" message shortcut that saves messages via `cognee.remember()`, request verification via `SLACK_SIGNING_SECRET` (plus `SLACK_`-prefixed app credentials), and a matching frontend Integrations page. The table is created by new Alembic revision `b2c4d6e8f0a1`, so existing deployments must run the migration (`cognee.run_migrations()`, or `alembic upgrade head`) to create it; the revision is guarded and skips if the table already exists. The core SDK pipeline (`add`/`cognify`/`search`) is unchanged (PR #4175). * Fixes `cognee-cli --api-url <cloud-url>` commands failing against reachable cloud tenants with a bogus "Cannot connect to Cognee API… Is the server running?" error. `api_dispatch.dispatch()` ran a fatal pre-flight `client.health()` probe before every command, using an unauthenticated one-off `httpx` client against the DB-backed `/health` endpoint and mapping any exception — including 503, 404, and 401 responses — to that message. The probe is removed: the real command runs immediately, and only genuine transport failures (classified by the new `is_connection_error()` helper, matching `httpx.TransportError`) are translated into a friendly message that includes the attempted URL; real HTTP errors such as 401 now surface their actual detail. Dataset collection calls (`datasets_list`, `datasets_create`, `datasets_delete_all`) now use the canonical trailing slash on `/api/v1/datasets/` and the shared client sets `follow_redirects=True`, so both cloud and local OSS deployments work; `health()` remains available as an explicit check, reuses the pooled authed client, and returns the body on 503 instead of raising. No CLI flags, config options, or environment variables changed (CLO-321, PR #4189). * Adds an opt-in, server-side default synthesis prompt for the MCP `recall` tool. Previously the per-call `system_prompt` argument (added in PR #4122) was the only way to override the terse `RecallPayloadDTO` default, so a shared cognee deployment could not define a default synthesis policy — every MCP client had to resend the full prompt on each call. `CogneeClient.recall` in `cognee-mcp/src/cognee_client.py` now resolves a default when the caller omits `system_prompt`: `COGNEE_MCP_RECALL_SYSTEM_PROMPT` supplies inline prompt text, and `COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE` points to a file holding it (inline takes priority; an unreadable file logs a warning and is skipped). Precedence is explicit caller argument, then env default, then backend default, and the resolution happens before the API/SDK branches so both modes are covered. With neither variable set, behavior is unchanged — no `system_prompt` is added to the payload — and the `search` tool is deliberately unaffected (PR #4205). * Fixes `forget(everything=True)` appearing to hang for 15+ minutes when deleting thousands of vector rows from LanceDB, the default vector backend. `LanceDBAdapter.delete_data_points` issued one `collection.delete` per id; each delete is a LanceDB commit that appends a table version, and manifest listing degrades as versions accumulate, so a 13,207-id wipe became 13k increasingly slow commits (\~7 deletes/sec and falling) that no per-call timeout or worker-death check could catch. Deletes now run as `id IN (...)` predicates in batches of 1,000, controlled by the new `DELETE_PREDICATE_BATCH_SIZE` class attribute; per-id single-quote escaping is preserved, sequential batches from one caller cannot commit-conflict, and missing ids or collections remain no-ops. The measured 13,207-id delete drops from 30+ minutes to about 0.1 seconds. Other providers (PGVector, Turso, hybrid Postgres, Neptune) already batched and are unchanged; no public API signature, configuration option, or environment variable changed (PR #4235). * Fixes Ollama model name handling that persisted after PR #3994's provider inference: when `LLM_MODEL` is set to a litellm-style prefixed name such as `ollama/llama3.1:8b`, the provider was correctly inferred as Ollama, but `OllamaAPIAdapter` stored the model name verbatim and sent the full prefixed string to Ollama's OpenAI-compatible endpoint, where no model by that name exists. The adapter's `__init__` (in `cognee/infrastructure/llm/structured_output_framework/litellm_instructor/llm/ollama/adapter.py`) now strips the prefix with `model.removeprefix("ollama/")` when the name starts with `ollama/`, so both `llama3.1:8b` and `ollama/llama3.1:8b` resolve to the bare model name `llama3.1:8b`. Unprefixed model names behave exactly as before, and no public API signature, configuration option, or environment variable changed; new unit tests cover both spellings and the inferred-provider path end to end (PR #4243). * Fixes a `TypeError` when the shared storage `JSONEncoder` encountered a plain `datetime.date` value. JSON Schema fields declared with `format: "date"` are materialized as `datetime.date` objects, but `JSONEncoder.default` in `cognee/modules/storage/utils/__init__.py` only handled `datetime`, `UUID`, and `Decimal`, so dates fell through to the standard library encoder and raised. A new `isinstance(obj, date)` branch now serializes plain dates with `date.isoformat()` as ISO-8601 strings; it is placed after the existing `datetime` check (which matches first, since `datetime` is a `date` subclass), so datetime serialization is byte-for-byte unchanged, as are UUID and Decimal handling, and unsupported objects still raise `TypeError`. No public API signature, configuration option, or environment variable changed (fixes #4239, PR #4244). * Fixes `parse_timestamp` in `cognee/modules/migration/cogx.py` — the helper feeding all five migration source adapters (Mem0, Zep, Graphiti, Letta, LangMem) and `export.py` — returning a mix of timezone-aware and naive datetimes: epoch values and ISO strings with a `Z` or offset came back UTC-aware, while offset-less ISO strings, bare dates, and naive `datetime` passthrough stayed naive. `LettaSource` can see both shapes in one conversation (Letta serializes `created_at` with or without an offset depending on version), which made `render_episode`'s `.timestamp()` sort apply the importing machine's local timezone — reordering transcript turns on any machine not set to UTC — and made direct naive/aware comparisons raise `TypeError`. `parse_timestamp` now always returns timezone-aware UTC datetimes (or `None`): offset-less inputs get `tzinfo=timezone.utc`, matching what the exporting systems store, while values with an explicit offset are preserved. `external_created_at`/`external_updated_at` metadata and rendered episode timestamps now carry `+00:00` where they were previously naive; no function signature or configuration changed (PR #4248). * Fixes the Letta and Zep migration adapters silently importing nothing when an export carries an empty key alias ahead of the populated one. The `_first_list(container, *keys)` helper, duplicated in `cognee/modules/migration/sources/letta.py` and `zep.py`, resolves the different spellings exports use for the same collection (e.g. `messages`/`in_context_messages`/`message_history` in Letta, `facts`/`edges`/`entity_edges` in Zep), but it returned on the first key whose value was a list — and since `[]` is a list, an alias that was present but empty short-circuited the scan. A Letta file with `messages: []` and the real history under `in_context_messages` therefore imported nothing, with `records()` yielding an empty stream and the import reporting success; six call sites across the two adapters were affected. The helper now keeps scanning until an alias actually yields dict records. Behavior changes only for inputs that previously imported nothing: a populated alias still wins in the same priority order, and all-empty inputs still yield nothing (PR #4253). * Fixes `recall()` auto-routing sending natural-language questions to coding-rules retrieval because they happened to contain a code token. In the rule-based router (`cognee/api/v1/recall/query_router.py`), the incidental code-token pattern (`def `, `return `, `async `, `await `, `import `, `class X(`, `.py`, `function x(`) scored `3.0` toward `CODING_RULES` — enough to clear the router's `2.0` default threshold and win outright, so questions like "Describe the import process for customer records" or "What does the return policy say?" were routed to coding rules, and the token could also outrank stronger intent cues in the same query. That weight is now `1.0`, which sits below the default threshold: an incidental code token can no longer select a search mode by itself, and it loses to any stronger cue. Explicit coding vocabulary (`coding rules`, `code review`, `best practice`, `lint`, `refactor`) keeps its `5.0` weight and still routes to `CODING_RULES`. User impact: affected queries now resolve to `GRAPH_COMPLETION` or to whichever cue actually dominates, so the `search_type`/`kind` reported on their results can differ from before; pass `query_type` explicitly to bypass the router entirely. No public API signature, configuration option, environment variable, or migration changed (PR #4207). * Fixes edge-filtered neighborhood queries crashing on the default Ladybug (Kuzu) graph backend. `LadybugAdapter.get_neighborhood(edge_types=[...])` built its Cypher with an `ALL(rel IN r WHERE rel.relationship_name IN $edge_types)` predicate over the variable-length relationship binding `r` from `-[r*1..depth]-`; in Kuzu `r` is a `RECURSIVE_REL` rather than a `LIST`, so `ALL(...)` is a binder-type mismatch that, combined with the parameter reference, drove the engine into a failed internal assertion. Because Ladybug is the default backend, any edge-filtered neighborhood query failed out of the box (Neo4j and Postgres were unaffected); the built-in search and visualization flows never pass `edge_types`, so the crash surfaced for programmatic callers of the public `get_neighborhood` primitive — custom retrievers and SDK code supplying an edge-type allow-list. The `ALL()` predicate is now dropped; when `edge_types` is supplied the adapter fetches the paths unfiltered and post-filters in Python, keeping a neighbor iff some path reaching it within `1..depth` hops has every edge type in the allowed set (undirected) — the same semantic Neo4j and Postgres already enforce. The behavioral tradeoff: the filtered branch enumerates every path up to `depth` before post-filtering, so its cost grows combinatorially with node degree × depth on dense graphs; it is best suited to shallow, targeted neighborhoods. The fast `edge_types=None`/`[]` path is unchanged. No public API signature, configuration option, or environment variable changed (fixes #3585, carries community fix #3591 by @ly-wang19, PR #4156). * Surfaces dataset and data-item **names** in the text channel of the `cognee-mcp` JSON tools `list_datasets_json` and `list_dataset_data_json`. Previously both tools put names only in `structuredContent` and emitted just a count in `content[0].text` (e.g. `8 dataset(s).`), so text-only MCP clients — such as agents in Cursor that never see `structuredContent` — could not tell what existed and fell back to raw HTTP. A new shared `_format_named_items` helper in `cognee-mcp/src/server.py` now renders one `- name (id)` line per item into the text content (falling back to `- name` when an id is absent and `(unnamed)` when a name is absent), prefixed by a header line (`8 datasets:`, or `1 dataset:` in the singular). The text is capped at 50 items, with a trailing `… and N more (see structuredContent).` note when the list is longer; an empty list reads `No datasets found.` / `No data items found.`. The change is backward-compatible and text-only: `structuredContent` (`{datasets: [...]}` / `{data: [...]}`), both tool schemas, and the Cognee workspace UI are unchanged, and no configuration option or environment variable was added (CLO-319, PR #4186). * Fixes the `cognee-mcp` client failing or hanging when listing datasets or checking status against Cognee Cloud, and makes API-mode `cognify` submit a background run instead of blocking. In API mode the client's `list_datasets` requested `/api/v1/datasets` (no trailing slash) and relied on the server's 307 redirect to the canonical `/api/v1/datasets/`; the client does not follow redirects, so the call failed with an HTTP error on the redirect response, and the redirect `Location` could additionally downgrade to `http://` against the HTTPS-only edge (see the server-side fix, CLO-320). `list_datasets` now calls the canonical trailing-slash route `/api/v1/datasets/` directly, so no redirect is involved. Separately, the client applied its single client-wide `timeout=300.0` to every request, so a hung or black-holed read-only GET froze the caller for a full five minutes; a per-request `READ_TIMEOUT_SECONDS = 30.0` is now applied to the dataset **list** and **status** GETs, while the 300s client-wide default remains for other requests. `READ_TIMEOUT_SECONDS` is a hardcoded module constant in `cognee-mcp/src/cognee_client.py`, not an environment variable or documented setting; the tradeoff is that a valid but very slow (>30s) GET will now time out. Finally, the client's `cognify` POST now sends `run_in_background: true`, so the request submits the pipeline run on the server and returns immediately instead of holding the HTTP request open for the whole run; the MCP `cognify` tool already returned immediately and directs callers to poll dataset status, which now reflects the server-side background run. No public MCP tool signature, configuration option, or environment variable changed (CLO-322, PR #4184). * Fixes OpenRouter embedding requests failing with a `400 invalid_value` error on `encoding_format`. Older LiteLLM releases serialize an *omitted* `encoding_format` as JSON `null`; OpenAI tolerates it, but OpenRouter rejects it (it accepts only `"float"` or `"base64"`), so an OpenRouter embedding config (`EMBEDDING_PROVIDER="custom"`, `EMBEDDING_MODEL="openrouter/openai/text-embedding-3-small"`) could 400 on every embedding call. `LiteLLMEmbeddingEngine.embed_text` now sets `encoding_format="float"` whenever it detects an OpenRouter route — a model id beginning with `openrouter/`, an explicit `openrouter` provider, or an `openrouter.ai` endpoint host (all matched case-insensitively). Cognee always consumes float vectors, so the value is safe to make explicit. The guard is scoped narrowly to OpenRouter on purpose: Cognee does not enable `litellm.drop_params`, and `encoding_format="float"` would raise `UnsupportedParamsError` for providers such as gemini/bedrock/vertex\_ai, so it is not applied unconditionally. On current LiteLLM versions the guard is a no-op for `openrouter/`-prefixed models (litellm's dedicated OpenRouter branch drops the omitted parameter), but endpoint-based configs — an unprefixed model pointed at an `openrouter.ai` endpoint — route through litellm's OpenAI handler, which injects the `null` even on current versions, so the guard is what fixes those. The change requires no user action, and no public API signature, configuration option, or environment variable changed (SDK-311, fixes #3660, PR #4195). * Adds a `COGNEE_EXTRAS` Docker build argument to the root `Dockerfile`, so optional-dependency groups can be baked into a source-built API image without editing the `Dockerfile`. The argument takes a space-separated list of extra names — `docker build --build-arg COGNEE_EXTRAS="docs langchain" -t cognee-custom .` — which the build expands into `--extra <name>` flags on top of the existing default set (which has since grown to include the `aws` extra, so `aws` no longer needs to be passed). It is applied to **both** `uv sync` steps: the second sync is exact and would otherwise drop extras installed only in the dependency-cache layer, so applying it twice is what makes the packages reach the final runtime stage. The argument defaults to an empty string, making it a no-op, so existing builds are unaffected. Both `uv sync` invocations keep `--frozen`, so the argument only selects extras already resolved in `uv.lock` rather than resolving new dependencies — builds stay deterministic and no lockfile change is required. Note that the `cognee` service's `build:` block in `docker-compose.yml` has no `args:` entry, so `docker compose up --build cognee` does not forward the value until you add one. The argument is declared only in the root API `Dockerfile`; the MCP and frontend images do not accept it. No public API signature, configuration option, or environment variable changed (SDK-314, PR #4211). * Stamps messages remembered through the Slack integration with a `slack` node set, so Slack-sourced data carries a structured origin marker into the graph. Slack's "Remember this" shortcut called `remember()` with only `dataset_name="slack"` and no `node_set`, so the resulting document reached the graph with no Slack marker at all — no `NodeSet` node, no `belongs_to_set` edge, no `source_node_set` property — and the only trace of where the content came from was the relational dataset name and the English prefix baked into the message text (`In #channel, <@user> said: …`). `remember_message` now passes `node_set=SLACK_NODE_SET` (a new module constant equal to `["slack"]` in `cognee/modules/integrations/slack/remember_message.py`), which materializes a `slack` [NodeSet](/core-concepts/further-concepts/node-sets) node with `belongs_to_set` edges that propagate from the document down to its chunks and extracted entities, and sets the `source_node_set` property the pipeline carries across task boundaries. The practical effect is that Slack items gain the same source dimension every other ingestion path already had: they are grouped and colored by origin in graph visualization, and retrieval can be scoped to them with `recall(..., node_name=["slack"])`. The change is non-breaking and requires no user action — Slack data already in the graph is untouched and is only marked if those messages are remembered again — and no public API signature, configuration option, or environment variable changed (SDK-318, PR #4216). * Reduces CPU spent on openai-python's per-response type introspection in Cognify and other LLM-heavy workloads. On every API response the OpenAI SDK rebuilds its `ChatCompletion` response tree, repeatedly calling `get_origin`, `get_args`, `is_annotated_type`, and `is_literal_type` against the same static types; at cognify scale that introspection dominated CPU. A new internal module `cognee/infrastructure/llm/openai_type_cache.py` wraps each of those four helpers in a `functools.lru_cache(maxsize=4096)` and rebinds the cached versions at every known openai import site — patching the source module alone is insufficient, because most call sites use `from ._compat import get_origin` and capture the name at import time. The install runs once and is idempotent, triggered when `cognee.infrastructure.llm` is imported, which any normal Cognee usage already does before an OpenAI SDK call is made; no user action or configuration is required. In an end-to-end 200-document Cognify run, CPU time dropped from \~93.85s to \~67.70s (≈28%). Failure modes degrade safely rather than break: the rebind targets are openai-python **private** modules (`openai._models`, `openai._utils._compat`, `openai._utils._typing`, `openai._response`, `openai._legacy_response`, `openai._base_client`) that are not covered by its compatibility guarantees, so if a future openai release moves them the import fails, `install()` becomes a no-op, and Cognee keeps running against the uncached originals; likewise an unhashable argument falls back to the original uncached helper, and any call site that is not rebound simply stays uncached. No public API signature, configuration option, or environment variable changed (COG-5963, PR #2876). * Adds `LangMemSource`, a memory-migration source for importing LangMem memories, so LangMem joins Mem0, Letta, Zep/Graphiti, and COGX archives as a system `cognee.remember()` can import from. Construct it with `LangMemSource(data, mode="re-derive")` — exported from `cognee.modules.migration.sources` — where `data` is a path to a LangMem JSON export, an already-parsed list (for live-API use, e.g. a response fetched with the LangMem client), or a dict wrapping the list under `memories`, `results`, `items`, or `data`; any other shape raises `ValueError`. Each item becomes a COGX **memory** record: content is the first string present among `content`, `text`, `memory`, `data`, and `message` (items with none of those are skipped), `user_id` falls back to `namespace` for the record's scope (which also carries `agent_id`, `session_id`, and `run_id` when present), `categories` accepts a single string or a list, `created_at`/`createdAt`/`timestamp` and `updated_at`/`updatedAt` supply timestamps, `id` falls back to a positional `langmem-<index>` identifier, and any `metadata` is carried over nested under `langmem_metadata`. The mode default is the base-class `re-derive`, which suits LangMem's free-form text (it carries no derived graph of its own to preserve). The change is additive: no existing source, `remember()` signature, configuration option, or environment variable changed. See [Migrate Memory Systems with COGX](/examples/migrate-memory-systems) (PR #4208). * Fixes ingested document names being recorded percent-encoded, and platform-foreign paths being recorded whole. `get_file_metadata` derived the document name (`FileMetadata["name"]`, persisted as `Data.name`) with `Path(file_path).stem`, where `file_path` is the opened stream's `.name`. In the ingestion pipeline that value is a percent-encoded `file://` URI — `LocalFileStorage` wraps every opened file in a `FileBufferedReader` named `Path(full_path).as_uri()` — so escapes leaked into the stored name on every platform: a file named `Annual Report.pdf` was recorded as `Annual%20Report`. `Path` is also OS-specific, so a raw backslash path from another caller (`C:\Users\me\report.pdf`) yielded the whole path minus its extension as the "stem" on POSIX (the reverse direction was never broken: Windows path handling accepts `/` as a separator). A new `_derive_basename` helper now percent-decodes `file://` URIs (`unquote(urlparse(...).path)`) and resolves the basename with `PureWindowsPath`, which treats both `/` and `\` as separators on every host OS, so those inputs are recorded as `Annual Report` and `report`; a degenerate input that yields an empty stem becomes `None`, letting the caller fall back to an explicitly supplied filename. The prior extension-less stem semantics are preserved: only the last suffix is stripped (`archive.tar.gz` → `archive.tar`), dotfiles such as `.gitignore` stay intact, and the extension continues to be stored separately in `FileMetadata["extension"]` — persisted for the original file as the `Data.original_extension` column, while `Data.extension` describes the stored text representation (typically `txt`). Separately, `classify()`'s fallback for `BufferedReader` / `SpooledTemporaryFile` inputs passed without an explicit `filename` now derives the basename with `str(data.name).replace("\\", "/").split("/")[-1]` instead of splitting on `/` alone (the same normalization as `_normalize_filename` in `cognee/tasks/ingestion/utils.py`), so a Windows-style stream name resolves to `report.pdf` rather than the entire path; this is a defensive path, because in the ingestion pipeline `classify()` only ever sees a `file://` URI. The change is backward-compatible and requires no action: POSIX paths and unencoded names resolve exactly as before, deduplication remains content-hash based so no record changes identity, and no public API signature, configuration option, environment variable, or database migration changed. Names already stored before the fix are not rewritten automatically, though re-adding a file refreshes its stored name through the re-ingest update path (SDK-237, PR #4157). * Fixes `cognify` failing on dlt-backed sources when the schema and foreign-key edges are registered in the relational rollback ledger. `extract_dlt_fk_edges` built its edge tuples with stringified node ids in the source/destination slots (`str(source_table_id)`, `str(relationship.id)`, and the `doc_id`/`target_data_id` strings it carries in its document map), but the same tuple list is handed both to `graph_engine.add_edges` and to the ledger's `upsert_edges`, which declares `List[Tuple[UUID, UUID, str, Dict]]` and binds slots 0/1 straight into the `edges` table's `source_node_id`/`destination_node_id` columns — both typed `UUID(as_uuid=True)`. SQLAlchemy's UUID bind processor reads `.hex` off the bound value, so the ledger insert raised `AttributeError: 'str' object has no attribute 'hex'` and failed the run. The failure was not limited to schemas that declare foreign keys: the per-row `is_row_of` edge linking each dlt row document to its `SchemaTable` node is emitted for every dlt row, so CSV, SQL connection-string, Gmail, and Slack-export ingestion were all affected. Slots 0/1 now carry `UUID` objects (including `UUID(doc_id)` and `UUID(target_data_id)` for the row-level edges), matching the tuple contract `upsert_edges` and the standard `get_graph_from_model` path already rely on, while each edge's JSON attribute dict keeps its string `source_node_id`/`target_node_id` copies, so stored edge properties are unchanged. Runs whose provenance is stamped in-graph skip the ledger write and were never affected. No public API signature, configuration option, environment variable, or database migration changed (SDK-183, PR #4081). * Doubles the remote request timeouts used by the `cognee.serve()` client, so long-running server-side work is no longer cut off mid-flight. `CloudClient` bounds ordinary remote operations (`remember`, `recall`, `improve`, `add`, `cognify`, `search`, `forget`) with a client-wide `aiohttp` total timeout, and applies a separate per-request timeout to archive uploads (`content_type="cogx-archive"`, the `cognee.push()` path) whose total is uncapped in favour of a per-read inactivity bound. Both limits were 300 seconds, which a blocking `cognify()` over a large dataset — or an archive upload plus its synchronous server-side import — could legitimately exceed, aborting the call client-side even though the server was still making progress. `DEFAULT_TIMEOUT` is now `total=600` and `UPLOAD_TIMEOUT` is now `sock_read=600`; the `sock_connect=30` connect bound and the uncapped upload total are unchanged, as is the two-tier structure itself (only the two 300s values moved). Both are hardcoded module constants in `cognee/api/v1/serve/cloud_client.py`, not environment variables or documented settings, so there is no way to tune them per deployment; the tradeoff is that a genuinely hung request now blocks for up to ten minutes rather than five. The change is backward-compatible, and no public API signature, configuration option, or environment variable changed (SDK-284, PR #4145). *** ## v1.4.0.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev1)** Development pre-release that bumps the package version from `1.4.0.dev0` to `1.4.0.dev1` (the tag sits on the SQL session-cache fix merge rather than a separate release-cut commit). No new Alembic revision ships in this cut. ### Highlights * Gates delete operations behind the same per-dataset lock that serializes pipeline runs. The in-process asyncio lock registry moves from `cognee/modules/pipelines/operations/pipeline.py` into the new `cognee/infrastructure/locks/dataset_lock.py`, exporting `dataset_lock`, `get_dataset_lock`, and `held_datasets`; pipeline runs and deletes now acquire from one shared registry. `datasets.delete_dataset` and `datasets.delete_data` in `cognee/api/v1/datasets/datasets.py`, plus `cognee.forget`'s dataset- and data-level memory clearing in `cognee/api/v1/forget/forget.py`, now run inside `async with dataset_lock(dataset_id)`, so a delete waits for any in-flight `add`/`cognify`/`memify` run on the same dataset (and vice versa) and two deletes on one dataset are serialized, while different datasets still proceed in parallel. The lock is re-entrant per execution context via the `held_datasets` `ContextVar` and remains process-local (asyncio) — it does not protect against multiple processes or workers touching the same dataset. No public API signature, configuration option, or environment variable changed (PR #4042). * Changes `cognee.improve()` so session-persistence failures are no longer swallowed. `_bridge_sessions` in `cognee/api/v1/improve/improve.py` wrapped its `persist_sessions_in_knowledge_graph_pipeline(...)` call in a blanket `try/except Exception` that logged `improve: session persistence failed (non-fatal)` as a warning and let the remaining improve stages (trace persistence, agent-context extraction, session distillation) run as if bridging had succeeded. That handler is removed: exceptions now propagate out of `improve()` (a `finally` still releases the per-session improve lock), and error handling is delegated to the pipeline system, which already records pipeline-run failures where suppression is appropriate. Callers that relied on `improve()` never raising during session bridging will now see those exceptions and can react to real failures instead of getting silent partial results. No public API signature, configuration option, or environment variable changed (PR #4056). * Adds a first-class `dataset_id` keyword parameter to `cognee.remember()` and routes it through improve operations. Previously `dataset_id` was only an untyped pass-through kwarg (listed in `RememberKwargs`/`_ADD_ONLY` and forwarded to `add()`), and the `self_improvement=True` paths always invoked `improve()` with `dataset_name` — so improve could not target a dataset addressed by UUID. `remember()` in `cognee/api/v1/remember/remember.py` now declares `dataset_id: Optional[UUID] = None` (takes precedence over `dataset_name`), resolves or creates the dataset before the session branch, and both the permanent and session self-improvement paths call `improve(dataset=dataset_id or dataset_name, ...)`. `RememberResult` fills `dataset_name` from the pipeline run when only an id is supplied, session-mode results now include `dataset_id`, and `CloudClient.remember` forwards the id as a `datasetId` form field. Passing `dataset_id` with `MemorySource` imports or typed `MemoryEntry` payloads raises `ValueError`; those stay dataset-name based. No env vars or migrations involved (PR #4158). * Fixes local-mode tenant context in the Cognee UI and batches related frontend/backend changes. `TenantContext`/`useTenant()` now expose `tenantReady`, `podUnreachable`, `isOwner`, `availableTenants`, and `releaseLoader`, and `LocalProvider` supplies the same shape, giving self-hosted (local mode) sessions the identical context contract as cloud tenants; tenant pod domain resolution is centralized in `getTenantApiDomain.ts` (explicit `NEXT_PUBLIC_TENANT_API_DOMAIN`, else parsed from `NEXT_PUBLIC_MANAGEMENT_API_URL`). Every `createHttpClient()` instance now attaches an `X-Request-Id` correlation header (the opt-in `setup.ts` registration is removed). Uploads in `rememberData.ts` pass `timeoutMs` to the shared HTTP client instead of racing a local `AbortController` against the client's 30-second default, restoring the intended 5-minute upload window, and `DatasetsPage` reports upload failures and knowledge-graph build failures separately while enforcing `MAX_FILES_PER_UPLOAD`. On the backend, a new `GET /api/v1/datasets/graph-summary` endpoint returns per-dataset node/edge counts cached in `GraphMetrics` per latest cognify `pipeline_run_id`, far cheaper for status polling than the full graph endpoint (PR #4179). * Fixes the Mistral transcription adapter sending an entire Windows path as the API `file_name`. `MistralAdapter.create_transcript` derived the file name with `input.split("/")[-1]`; on Windows the audio path uses backslash separators (e.g. `C:\audio\clip.mp3`) and contains no forward slashes, so the split left the value unchanged and the whole path — rather than the basename `clip.mp3` — was sent to the Mistral transcription API. The basename is now derived with `str(input).replace("\\", "/").split("/")[-1]`, normalizing both `\` and `/` separators (the same handling used by `_normalize_filename` in `cognee/tasks/ingestion/utils.py`), so Windows and POSIX paths both send just the file name. The change is backward-compatible: POSIX paths resolve to the same basename as before, and no public API signature, configuration option, or environment variable changed (fixes #3587, PR #3588). * Fixes writes failing on Postgres-backed graph and vector stores when node/edge fields or vector payloads contain NUL bytes (`\u0000`). Postgres text columns and JSONB reject the `\u0000` escape, and while the vector store's `json` column accepts it on insert, the `payload::jsonb` casts used by search/merge queries later reject it — so ingesting content with an embedded NUL byte could error. A shared `sanitize_relational_payload` helper now strips NUL bytes from strings and recurses through nested containers (dicts, lists, tuples), decoding `bytes`/`bytearray` values as UTF-8 with replacement so invalid byte sequences do not break persistence. It is applied in the Postgres graph adapter to the `id`, `name`, `type`, and `properties` of nodes and to the `source_id`, `target_id`, `relationship_name`, and `properties` of edges (ids sanitized identically on both sides so references stay consistent), and in `PGVectorAdapter` to each data point's serialized payload. This is an internal serialization fix for the Postgres graph and PGVector adapters; no public API signature, configuration option, environment variable, or database migration changed (PR #4153). * Fixes ingestion crashing on Windows when a string starts with `/` or is drive-relative. In `save_data_item_to_storage`, the absolute-path branch treated any string beginning with `/` (or, on Windows, one whose second character is `:`) as a local file path and called `Path(...).as_uri()` on it. On Windows, `os.path.normpath("/etc/hosts")` yields a *drive-relative* path and a drive-relative input like `C:notes.txt` stays drive-relative, so `as_uri()` raised `ValueError` (relative paths cannot be expressed as `file:` URIs) — any POSIX-style path string or plain text note starting with `/` crashed `add()`. The branch is now additionally guarded by `Path(os.path.normpath(data_item)).is_absolute()`, so on the current platform only genuinely absolute paths convert to a `file:` URI; non-absolute `/`-prefixed or drive-relative strings fall through to the existing relative-path/text handling and are ingested as text (saved to Cognee's data storage as a text file). POSIX behavior is unchanged (`/...` paths still convert to `file:` URIs) and genuine Windows absolute paths (`C:\...`) still convert as before. No public API signature, configuration option, or environment variable changed, and `accept_local_file_path` continues to govern acceptance of true absolute paths (fixes #3887, PR #3892). * Fixes S3 ingestion failing on Windows with `PermissionError` (WinError 32). In `data_item_to_text_file`, the S3 branch downloaded the object into a `tempfile.NamedTemporaryFile` created with the default `delete=True` and then passed `temp_file.name` to the loader, which reopens the file by name while Cognee's handle is still open. On Windows that reopen raises `PermissionError [WinError 32]`, so every S3 ingestion failed. The temp file is now created with `delete=False`, its handle is flushed and closed before the loader reopens it, and it is removed with `os.unlink` in a `finally` block (guarded against `OSError`) so no temp file is leaked — mirroring the `delete=False` pattern already used by the SQLAlchemy and ladybug S3 temp-file paths. POSIX behavior (Linux/macOS) is unchanged and temporary files are still cleaned up after use. No public API signature, configuration option, or environment variable changed (fixes #3339, PR #3340). * Fixes two failures in the SQL session-cache backend (`CACHE_BACKEND=postgres` / `sqlite`) when ids are UUID-like or when many writers target the same session concurrently. First, cache key columns (`user_id`, `session_id`, `qa_id`, `entry_id`, `log_key`, and the KV `key`) now use a `StringKey` `TypeDecorator` that coerces stringable ids such as `uuid.UUID` to `str` in the bind processor: the asyncpg dialect renders explicit bind casts, so an id bound as a `uuid.UUID` made Postgres parse `text = uuid` and raise `42883` ("operator does not exist"), while SQLite rejected the non-str bind outright — passing a `uuid.UUID` id (rather than a string) to the adapter could therefore fail to read or write. The decorator normalizes every read and write to the same string regardless of the caller's type; because DDL is delegated to the underlying `Text` impl, the emitted column stays plain TEXT and existing tables need no migration. Second, on Postgres each same-session write transaction now takes a transaction-scoped `pg_advisory_xact_lock` keyed by `(table, user_id, session_id)` before writing, so concurrent writers to one session queue instead of deadlocking on the sliding-TTL UPDATE (SQLSTATE `40P01`); the lock auto-releases at COMMIT/ROLLBACK, `delete_session` acquires the per-table locks in a fixed order so a multi-table writer can't cycle with single-table writers, and the whole mechanism is a no-op on SQLite (which serializes writers with its own single-writer lock). The tradeoff is that concurrent writes to the *same* session may serialize slightly; writes across different sessions are unaffected. No public API signature, configuration option, environment variable, or database migration changed (PR #4182). *** ## v1.4.0.dev0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev0)** First development pre-release after v1.4.0, bumping the package version from `1.4.0` to `1.4.0.dev0` and re-resolving `uv.lock`. Note that cognee cuts its `.devN` pre-releases after the corresponding stable release, so despite what PEP 440 version ordering suggests, `1.4.0.dev0` is newer than `1.4.0`. No new Alembic revision ships in this cut. ### Highlights * Fixes two crash/debuggability issues around telemetry and CLI user resolution. `send_telemetry()` in `cognee/shared/utils.py` called `asyncio.get_running_loop()` unconditionally, so in a sync context or after event-loop shutdown it raised `RuntimeError` and crashed the caller mid-pipeline; the `get_running_loop()`/`create_task()` preamble is now wrapped in `try/except RuntimeError`, making telemetry genuinely best-effort — the event is dropped instead of taking down the pipeline. Separately, `resolve_cli_user()` in `cognee/cli/user_resolution.py` caught bare `except Exception:`, so database connectivity or ORM failures were silently treated as "user not found"; the handler is narrowed to `EntityNotFoundError` (what `get_user()` raises for an unknown UUID), so genuine infrastructure errors now propagate with their original traceback. No public API signature, configuration option, or environment variable changed, and no action is needed (PRs #3276 and #3327). * Fixes three gaps in the recall API's result normalization that left agents and MCP clients with incomplete metadata. `AGENTIC_COMPLETION` results, previously normalized as `kind: "unknown"`, now map to `SearchResultKind.GRAPH_COMPLETION` in `cognee/modules/recall/methods/normalize_search_payload.py`. `FEELING_LUCKY` searches are resolved via `select_search_type()` in `cognee/modules/search/methods/get_retriever_output.py` before the retriever is instantiated, so the payload (and tracing span) carries the effective search type — e.g. `CHUNKS` — instead of the unresolved sentinel that also normalized to `"unknown"`. And when a completion entry is a Pydantic `BaseModel` (a `response_model` search), the item now gets `kind: "structured"` with the `structured` field populated from `model_dump()` while `text` stays renderable, where `structured` was previously always `null`. Normalization-layer only: no retriever ranking, public API signature, configuration option, or environment variable changed (fixes #3820, PR #3822). * Adds optional TLS to the Redis cache adapter, which previously had no SSL path and would hang against managed Redis endpoints with in-transit encryption (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis) until `socket_timeout` and then raise `CacheConnectionError`. Two new `CacheConfig` settings — `cache_ssl` (bool) and `cache_ssl_cert_reqs` (`"required"`/`"optional"`/`"none"`), settable via the `CACHE_SSL` and `CACHE_SSL_CERT_REQS` environment variables — are threaded through `get_cache_engine`/`create_cache_engine` into `RedisAdapter`, which forwards `ssl` and `ssl_cert_reqs` to both its sync and async `redis.Redis` clients. This brings the Redis cache to parity with the Postgres and pgvector adapters, which already read SSL settings from `DATABASE_CONNECT_ARGS`. Defaults are off and `"required"`, so existing deployments connect exactly as before; to enable TLS, set `CACHE_SSL=true` and adjust `CACHE_SSL_CERT_REQS` for self-signed certificates (fixes #3850, PR #3851). * Changes the four session-context methods on `SessionManager` (`create_session_context_entry`, `get_session_context_entries`, `update_session_context_entry`, `delete_session_context` in `cognee/infrastructure/session/session_manager.py`) to let `SessionParameterValidationError` propagate on an empty or whitespace `user_id`/`session_id`, matching `add_qa` and every other method in the class. Previously each wrapped `_validate_session_params` in `try/except Exception: return False` (or `[]`), so a caller bug like `user_id=""` was indistinguishable from an unavailable cache — and unlike the cache path, without even a `logger.warning`. The fail-open behavior for infrastructure errors is unchanged: when the cache is unavailable or the cache operation fails at runtime, the methods still return `False`/`[]` with a log message rather than raising, and behavior for valid inputs is identical. Callers passing invalid IDs will now see an exception instead of a silent `False`; no public API signature, configuration option, or environment variable changed (PRs #3881 and #3911). * Adds a pre-flight dry-run token/cost estimator and makes LLM quota/billing exhaustion fail fast instead of retrying. `remember(dry_run=True)`, `cognify(dry_run=True)`, and the CLI `--dry-run` flag on both commands return a stage-level estimate (structured graph extraction plus chunk summarization) of LLM tokens and rough USD cost with no LLM calls, no ingestion, and no graph writes; the estimator (`cognee/modules/cognify/estimator.py`) reuses the real pipeline's document classification, chunker, and prompt templates, resolves datasets read-only, and rejects unsupported inputs (remote URLs, directories, binary formats, session memory, remote `serve()` mode) loudly rather than mis-estimating. Separately, a shared `llm_retry_condition` in `cognee/infrastructure/llm/retry_config.py` now governs every structured-output adapter and BAML, treating quota/billing wordings like `insufficient_quota` as terminal — `LLMGateway.acreate_structured_output` converts them into an actionable `LLMQuotaExceededError` — while transient rate limits (including Gemini free tier's recoverable "exceeded your current quota") still retry. `dry_run` defaults to `False`, so existing calls are unaffected (SDK-136, fixes #3643, PR #3974). * Adds an optional `LLM_PROVIDER=mcp-sampling` backend that lets cognee, when running as an MCP server (`cognee-mcp`) inside a host such as Claude Code or Cursor, delegate all LLM completions to the host's own model via MCP's `sampling/createMessage` — so no `LLM_API_KEY` is needed (the provider is excluded from the API-key-required set; `LLM_MODEL` is only a hint, the host picks the model). The new `MCPSamplingAdapter` reads the host session from the MCP SDK's per-request context via `get_sampling_session()`, so `cognee-mcp/server.py` needed no changes; structured output is produced by embedding the response model's JSON Schema in the prompt with a bounded validate/repair loop, since sampling returns free text only. It fails closed with an actionable `MCPSamplingUnavailableError` when cognee is not running as an MCP server or the host did not grant the `sampling` capability. Completions only — embeddings, audio, and vision are not covered, so configure an embedding provider for vector search; non-MCP usage and all existing providers are unchanged, and `mcp` remains an optional dependency (SDK-139, closes #3644, PR #3982). * Adds video ingestion: a new `video_loader` (`cognee/infrastructure/loaders/core/video_loader.py`) transcribes a video's audio track through the existing `LLMGateway.create_transcript` path and feeds the text into the normal pipeline as a regular `TextDocument` — no new document type. Supported extensions are `mp4`, `m4v`, `mov`, `webm`, `mkv`, and `avi`; on OpenAI/Azure the transcript carries inline `[HH:MM:SS]` segment timestamps so timing survives chunking, while other providers fall back to a plain transcript. ffmpeg is optional: `.mp4`/`.webm` are sent straight to the transcription endpoint, other containers use a system ffmpeg (found via `shutil.which("ffmpeg")`) to extract the audio first and raise an actionable error when it is absent. The PR also fixes a real cross-provider bug: `create_transcript` forwards `**kwargs`, but the mistral/gemini/custom adapters rejected them with a `TypeError` inside the retry loop; the `**kwargs` contract is now uniform across `LLMInterface` and all adapters. No new configuration options, environment variables, or extras were added (SDK-141, related issue #3636, PR #3986). * Adds Turso (libSQL) as a selectable backend for all three cognee stores. Vector: `VECTOR_DB_PROVIDER=turso` routes to the new `TursoVectorAdapter` (collections as libSQL tables with `F32_BLOB` vector columns, cosine similarity via `vector_distance_cos`), working embedded (local file) or against Turso cloud via `VECTOR_DB_URL`/`VECTOR_DB_KEY`, and requires the `cognee[turso]` extra (`libsql-experimental`). Relational: `DB_PROVIDER=turso` selects a `TursoAdapter` that is a thin subclass of `SQLAlchemyAdapter` — a libSQL file is a SQLite file, so it reuses the same `aiosqlite` driver, sqlite dialect, and Alembic migrations; remote mode uses Turso's embedded-replica sync via `DB_TURSO_URL`/`DB_TURSO_AUTH_TOKEN`. Graph: `GRAPH_DATABASE_PROVIDER=turso` stores the knowledge graph as `graph_node`/`graph_edge` tables with recursive-CTE traversals, local/embedded only, needs no extra dependency, and optionally takes a `GRAPH_DATABASE_URL` file path. Vector and graph each ship a per-dataset dataset-database handler, so multi-user isolation under `ENABLE_BACKEND_ACCESS_CONTROL` works unchanged. All Turso backends are opt-in; defaults and existing backends are untouched (SDK-152, SDK-175 and SDK-176; PRs #4027, #4077 and #4080). * Adds deterministic, LLM-free code-graph extraction backed by the external `enola` binary (Apache-2.0, by Enola Labs). The new `cognee/tasks/code_graph` package runs `enola --generate` (discovered on `PATH` or via the `ENOLA_PATH` env var, with an `install_enola` helper pinned to a known version) and maps its `.enola/facts.jsonl` output — module/symbol/route/storage/dependency/service facts — to DataPoint models with deterministic `uuid5` ids plus typed edges (`calls`, `imports`, `implements`, `depends_on`, and others). User-facing entry points: `cognee.remember(repo_path_or_git_url, content_type="code")` indexes one or more local repos or git URLs (optional `index_vectors=True` enables embeddings; the default graph-only run needs no LLM key), a new `SearchType.CODE` queries the result through the new `CodeRetriever`, and `get_code_graph_tasks(repo_path)` plugs into `run_custom_pipeline`. Extraction covers Go, Python, TypeScript, Java, Kotlin, Swift, Ruby, C/C++, PHP, Vue, Svelte, OpenAPI, and gRPC. No enola code is vendored and no new Python dependencies are added; a missing binary raises an actionable `EnolaNotInstalledError` (COG-5837, PR #4037). * Adds Langfuse tracing support by wiring Langfuse into cognee's existing OpenTelemetry pipeline as just another OTLP destination — no separate Langfuse SDK. Setting `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` (optionally `LANGFUSE_HOST`, with `LANGFUSE_BASE_URL` accepted as an alias) makes `base_config.py` derive the OTLP endpoint (`{host}/api/public/otel/v1/traces`) and the `Authorization: Basic` header and turn tracing on; both keys must be set together or a `ValueError` is raised, and an explicit `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` always wins. Generation spans in `get_observe.py` now emit the vendor-neutral `gen_ai.request.model` and `gen_ai.system` attributes plus `langfuse.observation.type` and `SpanKind.CLIENT`, so Langfuse and any other OTLP backend render LLM calls as generations, and `tracing.py` forces the HTTP exporter for Langfuse endpoints (detected by the `/api/public/otel` path) since Langfuse does not support gRPC. Fully opt-in and off by default; requires the `cognee[tracing]` extra; no existing configuration or public API changed (SDK-167, PR #4055). * Adds an opt-in `litellm_native` structured-output framework that returns validated Pydantic objects without the `instructor` library, using LiteLLM's own `response_format`. Set `STRUCTURED_OUTPUT_FRAMEWORK="litellm_native"` and `LLMGateway` routes `acreate_structured_output` to a single universal `NativeLiteLLMAdapter` (`cognee/infrastructure/llm/structured_output_framework/litellm_native/`) covering every provider: when `litellm.supports_response_schema(model)` is true (OpenAI, Azure, Gemini, Mistral, Bedrock, …) the Pydantic model is passed straight through as `response_format` and validated with `model_validate_json`; otherwise (Ollama, llama.cpp, custom endpoints) it falls back to `response_format={"type": "json_object"}`, injects the JSON Schema into the prompt, and on validation failure feeds the error back for up to 3 self-correcting retries. Error handling matches the instructor adapters: auth and budget errors (`LLMPaymentRequiredError`) are terminal, rate limits retry with backoff, and content-policy violations fall back to the configured fallback model. The default remains `instructor` and the `baml` path is untouched; `create_transcript`/`transcribe_image` are unaffected (SDK-172, PR #4066). * Adds a one-command evaluation runner and makes the eval harness a clean optional addon. `run_eval(config) -> EvalResult` (`cognee/eval_framework/runner.py`) chains corpus building, answering, evaluation, and dashboard generation for a single deterministic config and returns artifact paths plus aggregate metrics; it is exposed as both `cognee eval …` and `python -m cognee.eval_framework …`, mapping the same flags onto `EvalConfig`. The evaluator registry now resolves engines lazily by import path, so importing the registry or running the DirectLLM engine no longer pulls in `deepeval`; selecting DeepEval without the dependency raises an actionable error pointing at `pip install "cognee[eval]"`, a new umbrella extra that installs the dashboard (`plotly`), the DeepEval engine, and dataset-download deps — `--engine direct_llm --no-dashboard` runs without it, and the runner preflights dashboard imports before any paid pipeline work. A `seed` is now actually threaded into the benchmark adapters, and artifacts are namespaced under `<output-dir>/<benchmark>_<engine>/` with the resolved config saved alongside (SDK-174, PR #4073). * Adds optional EXIF metadata extraction and perceptual-hash deduplication to `ImageLoader` (`cognee/infrastructure/loaders/core/image_loader.py`). With `IMAGE_EXIF_ENABLED=true`, `_extract_exif_metadata` pulls camera make/model, date taken, exposure, F-number, ISO, focal length, and GPS coordinates from the image's EXIF data and appends them as an `[EXIF Metadata]` block to the vision-LLM transcription. With `IMAGE_PERCEPTUAL_HASH_ENABLED=true`, a 64-bit difference hash (dHash) is computed per image and appended as a `[Perceptual Hash: …]` marker; an in-memory check against hashes seen in the same process flags visually similar re-ingestions with a duplicate note rather than skipping them. Both features are controlled by plain environment variables read at load time, are off by default, and use only PIL, which is already a cognee dependency — no new packages, and existing behavior is byte-identical unless a flag is enabled (partially addresses #3637, PR #4076). * Fixes BAML structured output failing on Pydantic models with PEP 604 optional fields. With `STRUCTURED_OUTPUT_FRAMEWORK=baml`, any response model containing an `X | None` field raised `ValueError: Unsupported type for BAML mapping: str | None` — in practice making `GRAPH_COMPLETION` recall unusable with BAML (common on local/Ollama setups), since the completion response model uses `str | None` fields. The cause: `map_type()` in `create_dynamic_baml_type.py` only matched `origin is Union` (`typing.Union`), while PEP 604 unions report `get_origin() == types.UnionType`, so they fell through to the unsupported-type error even though the equivalent `typing.Optional[str]` worked. The condition is now `if origin is Union or origin is types.UnionType`, handling both spellings on Python >= 3.10 (the project minimum). One-line fix plus a comment; no public API, configuration option, or environment variable changed, and non-BAML frameworks are unaffected (PR #4121). * Adds an optional `system_prompt` parameter to the MCP `recall` tool and forwards it end to end, so MCP clients can override the synthesis prompt for completion searches. Previously the tool signature had no such parameter, so custom recall system prompts were silently impossible over MCP. The parameter now flows through `CogneeClient.recall` (`cognee-mcp/src/cognee_client.py`) in both modes: in API mode it is included in the JSON payload POSTed to `/api/v1/recall`, and in direct/SDK mode it is passed as a kwarg to `cognee.recall`. When omitted, nothing is added to the payload or kwargs, so existing behavior is unchanged; regression tests cover both the API payload forwarding and the MCP tool forwarding. No other tool parameters, defaults, or server endpoints changed (fixes #4120, PR #4122). * Fixes `cognee remember --dry-run` and `cognee cognify --dry-run` silently executing real remote operations when `--api-url` is supplied. The API dispatch path bypassed the local command implementations that honor `dry_run` and simply ignored the flag, so a "dry run" performed an ordinary remote remember/cognify. `dispatch()` in `cognee/cli/api_dispatch.py` now checks `args.dry_run` up front and raises a `RuntimeError` ("--dry-run is not supported in --api-url mode. Run without --api-url to estimate locally without remote side effects.") before constructing the API client, so no health check or operation request is ever sent. Users who want a dry-run estimate must run without `--api-url`; ordinary API-mode behavior without the flag, local dry runs, and server behavior are unchanged, and no API endpoint, dependency, or configuration option changed (closes #4125, PR #4126). * Changes `visualize_graph()` and `GET /api/v1/visualize` to render a bounded, relevant subgraph by default instead of the whole graph. The renderer now selects a small set of seed nodes, expands their *k*-hop neighborhood, and caps the result at `max_nodes`, keeping renders fast and readable on large graphs. Seeds are resolved by priority — explicit `seed_node_ids` > a `recall()` or search result's graph provenance (`recall_result`, via `used_graph_element_ids`) > a `query` string's nearest (distance-ranked) vector hits > the graph's highest-degree nodes as a fallback — so a bare `visualize_graph()` call still shows a representative view, and "show me the subgraph behind this answer" and query-seeded views are deterministic and capped. New **keyword-only** parameters were added to `visualize_graph()`: `full`, `query`, `seed_node_ids`, `recall_result`, `neighborhood_depth` (default `2`), `neighborhood_seed_top_k` (default `10`), and `max_nodes` (default `500`); when a neighborhood exceeds `max_nodes`, nodes are kept by hop distance from the seeds and edges survive only when both endpoints do (no dangling edges). To restore the previous whole-graph render, pass `full=True` (or `?full=true` on the endpoint). `GET /api/v1/visualize` gains matching query params `full`, `query`, `seed_node_ids`, `neighborhood_depth`, `neighborhood_seed_top_k`, and `max_nodes` (`recall_result` is Python-only). The change is backward-compatible: the new parameters are keyword-only, so existing positional callers keep working; the underlying renderer and shared graph primitives are reused. See [Graph Visualization → Bounded subgraph by default](/guides/graph-visualization) (SDK-140, PR #3985). * Fixes the stored file size not refreshing when a file is re-ingested. On the re-ingest update path, `ingest_data` assigned the new size to `data_point.file_size`, but the `Data` model defines the column as `data_size` (the name already used correctly on the new-record branch). SQLAlchemy silently ignored the nonexistent attribute, so the persisted `data_size` stayed at its original value after a file was re-added with new or changed content. The assignment now targets `data_point.data_size`, so re-ingestion records the current size. No public API signature, configuration option, environment variable, or database migration changed (fixes #3160, PR #3578). * Fixes regex entity extraction config files failing to load on platforms whose default locale encoding is not UTF-8 (commonly Windows). `RegexEntityConfig._load_config` previously opened the config JSON with `open(path, "r")`, which relies on the platform's locale encoding; on a non-UTF-8 system a config containing non-ASCII characters (for example Unicode entity names, descriptions, or regex patterns) could raise a decode error and fail to load. The file is now opened with an explicit `encoding="utf-8"`, so configs load consistently across platforms. The change is backward-compatible: existing UTF-8/ASCII configs load exactly as before, no config edits or migration are required, and no public API signature, configuration option, or environment variable changed (fixes #3316, PR #3337). * Fixes a `TypeError` when instantiating the Amazon Neptune Analytics graph adapter (`NeptuneGraphDB`). `GraphDBInterface` declares `is_empty()` as an abstract method, but the Neptune adapter never implemented it, so the class was abstract and any attempt to construct it (including test collection in `cognee/tests/test_neptune_analytics_graph.py`) raised `TypeError: Can't instantiate abstract class NeptuneGraphDB with abstract method is_empty`. The adapter now implements `async is_empty() -> bool`, which runs a small openCypher node-existence query (`MATCH (n) RETURN true LIMIT 1`) and returns `True` when the graph contains no nodes and `False` otherwise; it relies on Neptune's openCypher support. This is a non-breaking bug fix that only adds the required method — no public SDK function, configuration option, or environment variable changed (closes #3407, PR #3457). * Fixes embedding retries wasting the full back-off window on deterministic "context window too small" failures. When an over-length embedding input is split down to a single string that still exceeds the model's context window but can no longer be divided, the engines now raise a new terminal `EmbeddingContextWindowTooSmallError` (a subclass of `EmbeddingException`, default message `Text is too short to split further but exceeds context window.`) and add it to their `retry_if_not_exception_type` set, so the failure returns immediately instead of consuming the \~128-second retry/back-off window. This applies to `LiteLLMEmbeddingEngine`, `FastembedEmbeddingEngine`, and `OpenAICompatibleEmbeddingEngine`; generic `EmbeddingException` failures remain retryable. The `OpenAICompatibleEmbeddingEngine` also now imports the shared `EmbeddingException`/`EmbeddingContextWindowTooSmallError` from `cognee.infrastructure.databases.exceptions` instead of defining a local `EmbeddingException`. No public API signature, configuration option, or environment variable changed; code that already catches `EmbeddingException` continues to catch the new subclass (fixes #3319, PR #3424). * Fixes `.txt` prompt templates being HTML-escaped on the wire. `render_prompt` configured its Jinja2 environment with `autoescape=select_autoescape(["html", "xml", "txt"])`, and because every prompt template shipped with Cognee is a `.txt` file, every interpolated variable in every rendered LLM prompt was HTML-escaped for all providers — apostrophes became `'`, triplet arrows in retrieval context became `-->`, ampersands became `&`, and angle brackets became `<`/`>`, including the user's own question in completion prompts. Autoescape now covers only markup templates (`["html", "xml"]`), so `.txt` prompts render their content verbatim while `.html`/`.xml` templates remain escaped. The user-visible effect is restored prompt fidelity, reduced token waste, and fewer subtle parsing/extraction issues; the change is internal to prompt rendering and non-breaking — no public API signature, configuration option, or environment variable changed (SDK-203, PR #4115). * Fixes DLT orphan cleanup leaving forgotten rows in the per-dataset graph and vector stores under `ENABLE_BACKEND_ACCESS_CONTROL`. When re-ingesting a DLT source (or a document source such as Notion/Slack/Google Drive) after rows were removed upstream, Cognee reconciles the corpus by deleting rows no longer present. Under access control the graph and vector engines are dataset-scoped, but the cleanup ran outside the dataset DB context — most visibly on the background-ingest path, where `orphan_cleanup` runs before any pipeline establishes that context — so `delete_data_nodes_and_edges` resolved the *default* engines and the graph + vector purge silently targeted the wrong database, leaving the forgotten row's chunks and entities in place and still retrievable (only the relational record was removed). The per-orphan deletion now runs inside `set_database_global_context_variables(dataset.id, dataset.owner_id)`, so the graph, vector, and relational stores are all purged for the correct dataset. Cleanup remains best-effort: partial failures are logged and retried on the next ingest rather than failing the add. No public API signature, configuration option, or environment variable changed (SDK-189, PR #4090). *** ## v1.4.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0)** Release that bumps the package version from `1.3.0` to `1.4.0` and refreshes `uv.lock`. The release cut itself introduces no functional code, public API, configuration, or environment-variable changes; the entries below are the accumulated work promoted from the development branch in this release (SDK-197). No migration or action is required for existing integrations. ### Highlights * Fixes `RAG_COMPLETION` and `TRIPLET_COMPLETION` searches ignoring the `node_name` filter. The public `search()` API already accepted `node_name` (and `node_name_filter_operator`) to restrict results to specific node sets, but for these two search types the argument was silently dropped: `CompletionRetriever` and `TripletRetriever` never received it, so their vector lookups (`DocumentChunk_text` / `Triplet_text`) searched the whole collection and returned chunks/triplets from outside the requested node set(s). Both retrievers now accept `node_name` and `node_name_filter_operator`, the search-type factory (`get_search_type_retriever_instance`) forwards them, and they are passed through to the vector search so results are scoped to the given node set(s) using the chosen `AND`/`OR` operator. The change is backward-compatible: `node_name` defaults to `None` (no filtering), so calls that never set it behave exactly as before, and no public API signature, configuration option, or environment variable changed (COG-5868, PR #4053). * Fixes two `cognee-mcp` bugs that surfaced on the first `remember` of a clean direct-mode (stdio) MCP session. First, the session-backed `remember()` flow now completes the session-to-graph bridge cleanly instead of tripping over dataset setup on the first write. Second, MCP startup migration output no longer pollutes the stdio JSON‑RPC channel, so clients do not misread migration chatter as protocol data. The `remember()` / `cognee.remember()` signatures are unchanged and no configuration option or environment variable changed; the only user-visible tradeoff is a brief one-time delay on the first `remember` while the session is initialized and bridged (SDK-192, PR #4091). * Fixes `improve()` runs failing to persist agent-trace feedback on multi-tenant (`ENABLE_BACKEND_ACCESS_CONTROL=true`) deployments. The agent-trace-feedback persistence path now forwards the authenticated `user` into `cognee.add()` and `cognee.cognify()`: `cognify_agent_trace_feedback` accepts a `user` parameter and passes it to both calls, and `persist_agent_trace_feedbacks_in_knowledge_graph_pipeline` supplies the pipeline's `user` to that enrichment task. Previously these `add`/`cognify` calls ran as the default user, which has no write ACL on multi-tenant deployments, so trace persistence raised a `403 PermissionDeniedError` and the `improve()` run showed errored memify-pipeline stages while feedbacks were silently skipped. The `user` parameter these internal pipelines and tasks already accepted is unchanged, and no public `improve()`/`memify()` API signature, configuration option, or environment variable was modified (COG-5893, PR #4097). *** ## v1.3.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.3.0)** Release that bumps the package version from `1.2.2` to `1.3.0` and regenerates the dependency lockfiles (`poetry.lock`, `uv.lock`, `cognee-mcp/uv.lock`). The release cut itself introduces no functional code changes; the entries below are the accumulated work promoted from the development branch in this release. Deployers upgrading should re-lock and reinstall to pick up the refreshed dependency graph. ### Highlights * Fixes document classification for text-like and unknown file extensions. `classify_documents` previously looked up the document class with `EXTENSION_TO_DOCUMENT_CLASS[data_item.extension]`, which raised `KeyError` during ingestion for extensions the map didn't cover — including common text formats (`md`, `json`, `xml`, `yaml`) and uppercase variants such as `.PDF` or `.CSV`. The extension is now normalized to lowercase before lookup, `md`/`json`/`xml`/`yaml` are mapped to `TextDocument`, and any unrecognized extension falls back to `TextDocument` instead of crashing the pipeline. Files that previously failed to ingest are now classified and processed. The change is backward-compatible: already-mapped extensions classify exactly as before, and no public API signature, configuration option, or environment variable changed (fixes #3657, PR #3662). * Fixes the CLI `cognify` command's `--ontology-file` flag, which previously had no effect. The command passed `ontology_file_path=` to `cognee.cognify()`, but `cognify()` accepts only a `config` object and silently swallowed the unsupported argument through `**kwargs`, so the ontology was never loaded. The command now translates `--ontology-file` into the canonical ontology `config` structure (an `rdflib` resolver with fuzzy matching, built with the same factory `cognify()` uses for its env-based fallback), validates up front that every referenced path exists and otherwise raises a clear `Ontology file not found: <paths>` error, and accepts multiple ontology files as a comma-separated list. Separately, a failed CLI command now always prints its error message: `cognee cognify` failures raise a `CliCommandException` whose `raiseable_exception` field is unset, and the entry point previously printed the message only when that field was set, so the command exited with code `1` but no explanation. Exit codes are unchanged (still `1` on failure), and no new flag, public API signature, or environment variable was introduced (PR #3997). * Improves concurrency and throughput of LanceDB subprocess mode (`VECTOR_DB_SUBPROCESS_ENABLED=true`) by replacing the session-wide RPC lock with id-based routing. Each async RPC now carries a per-request id and a main-process reader thread routes responses to per-call futures, so concurrent `call_async` operations run in parallel instead of serializing behind a single lock. A new `SUBPROCESS_WORKER_MAX_INFLIGHT` environment variable (default `16`) bounds how many async operations a worker runs at once; it must be `> 0` or worker initialization raises `ValueError` rather than silently degrading. Failure semantics also change: a per-call timeout or cancellation now resolves only that call and no longer tears down the entire subprocess session — the session ends only on genuine crash/shutdown/respawn events, which propagate a `SubprocessTransportError` to any still-pending calls. Synchronous calls (such as the Kuzu graph backend) continue to run serially and are unchanged. In internal Locust benchmarks, `/api/v1/add` average latency dropped from \~1371ms to \~246ms and p95 from \~9300ms to \~520ms, with overall throughput up \~21% (PR #2826). * Fixes propagation of the authenticated user into the memify session- and feedback-persistence pipelines, correcting multi-tenant attribution. The `persist_sessions_in_knowledge_graph_pipeline` and `persist_agent_trace_feedbacks_in_knowledge_graph_pipeline` functions now set the session user context (`set_session_user_context_variable(user)`) before running memify, so persisted sessions and agent-trace feedbacks are recorded against the intended authenticated user instead of a default/missing user. The `user` parameter these pipelines already accept is unchanged — no public API signature, parameter, or environment variable was modified, and no data migration is required. Logs and stored knowledge-graph entries may now show different (correct) user associations; custom memify pipeline hooks that relied on the previous missing-user behavior should be verified (PR #3950). * Adds optional per-stage LLM model routing so the extraction, summarization, and query stages can each run on a different model or provider. Each stage reads an optional `LLM_<STAGE>_*` environment group — `LLM_EXTRACTION_*`, `LLM_SUMMARIZATION_*`, and `LLM_QUERY_*`, each accepting `MODEL`, `PROVIDER`, `ENDPOINT`, `API_KEY`, and `API_VERSION` — whose set fields override the base `LLM_*` values for that stage while any unset field falls back to `LLM_*`. Because extraction runs once per chunk and dominates token spend, a common setup routes a cheaper or local model for extraction while keeping a stronger model for summarization and query-time reasoning (for example `LLM_EXTRACTION_MODEL="ollama_chat/llama3.1"`, `LLM_EXTRACTION_PROVIDER="ollama"`, `LLM_EXTRACTION_ENDPOINT="http://localhost:11434"`). Under the hood `LLMConfig.stage_config(stage)` returns a copy of the base config with any stage overrides applied, and a `pipeline_stage(stage)` context manager sets the existing `llm_config` ContextVar to that merged config for the duration of the stage; the client cache key is already derived from the context config, so each stage transparently gets its own cached client. This is fully backward-compatible and requires no action for single-model setups: the stage fields default to empty, so with no `LLM_<STAGE>_*` variables set the effective config is identical to today, and no extraction, summarization, retrieval, or SDK call signature changed. See the "Per-Stage Model Routing" section of [LLM Providers](/setup-configuration/llm-providers) for the full field reference and examples (PR #3961). * Fixes the missing exception type in error logs. When an exception is logged, `setup_logging()`'s structlog `exception_handler` processor records the exception class name in the `exception_type` field. That field guarded its assignment with `hasattr(exc_type, __name__)`, which used the module's own `__name__` (the string `"cognee.shared.logging_utils"`) rather than the literal `"__name__"`; since an exception class never has an attribute by that name the check was always false, so `exception_type` was never added to the log event. The guard now checks `hasattr(exc_type, "__name__")`, so logged exceptions record their type (e.g. `ValueError`) alongside the existing `exception_message`, identifying what failed rather than only that something failed. No public API, configuration option, or environment variable changed (PR #3998). * Fixes the `exception_type` field being silently omitted from logged exceptions. The custom `exception_handler` structlog processor in `cognee.shared.logging_utils.setup_logging` checked `hasattr(exc_type, __name__)`, where the unquoted `__name__` resolved to the module's name rather than the literal attribute name — so the check almost never passed and `event_dict["exception_type"]` was never set. The argument is now the string `"__name__"`, so log records for exceptions correctly capture the exception class name (`exception_type = exc_type.__name__`). This only affects the metadata attached to logged exceptions; no public API signature, configuration option, or environment variable changed (fixes #3709, PR #3849). * Restores a synchronous `get_vector_engine()` as a deprecated backward-compatibility shim and establishes `get_vector_engine_async()` as the canonical async accessor for the vector engine. Both are exported from `cognee.infrastructure.databases.vector`. `get_vector_engine()` is safe to call synchronously from any context — it does no async work when constructing the engine handle — but it now emits a `DeprecationWarning`, and the returned adapter's methods (`embed_text`, `search`, `get_connection`, ...) remain coroutines that must be awaited inside a running event loop. Released users who called `get_vector_engine()` without `await` are unaffected. Dev users who adopted the unreleased async form `await get_vector_engine()` should switch to `await get_vector_engine_async()`, which keeps a uniform "await the engine getter" contract alongside `await get_graph_engine()` (PR #3967). * Speeds up graph extraction for inputs with many chunks by removing a quadratic scan in `extract_graph_from_data`. DLT row chunks (whose graph is built deterministically from schema metadata rather than by the LLM) are excluded from the extraction path; previously each chunk was matched against the DLT set with a repeated list-membership check that triggered a full Pydantic `__eq__` comparison per pair, so the filter cost scaled with `len(data_chunks) × len(dlt_chunks)`. The function now partitions `data_chunks` into DLT and non-DLT lists in a single pass and returns `integrated + dlt_chunks`, making the extraction hot path linear in the number of chunks (a micro-benchmark reports roughly a 9,000× speedup at 4,000 chunks). Outputs are identical and the change is internal to `extract_graph_from_data` — no public API signature, configuration option, or environment variable changed (fixes #4015, PR #4017). * Preserves external ontology IRIs end-to-end and adds an RDF/SPARQL read surface plus RDF ingestion. `DataPoint` gains an optional `ontology_uri` field (defaults to `None`) that carries the external IRI a node is grounded in, threaded through `expand_with_nodes_and_edges` so persisted nodes keep their identifier instead of collapsing it to a local label. A new read surface (`cognee.modules.graph.rdf`) exposes the memory graph as RDF: `graph_data_to_rdf`, `export_memory_graph_to_rdf`, `serialize_memory_graph`, and `query_memory_graph_sparql`. Ungrounded nodes receive minted IRIs under `https://cognee.ai/graph/...` so the RDF is well-formed, `is_a` maps to `rdf:type` (individual→class) or `rdfs:subClassOf` (class→class), and other relations become predicate IRIs. RDF ingestion (`cognee.modules.ontology.rdf_xml.rdf_ingest` — `ingest_rdf`, `load_rdf_graph`, `build_datapoints_from_rdf`) parses TBox/ABox into `EntityType`/`Entity` datapoints that preserve verbatim IRIs, with identity derived from the IRI so re-ingesting the same RDF is idempotent. The change is backward-compatible: `ontology_uri` defaults to `None`, no DB migration is required, and the RDF surface rides on the existing `rdflib` ontology dependency (ensure `rdflib` and any parser backends are available in your runtime to use RDF export/ingest). See [Ontologies → RDF read/write surface](/core-concepts/further-concepts/ontologies#rdf-readwrite-surface) (PR #3928). *** ## v1.2.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.2)** Patch release that bumps the package version from `1.2.1` to `1.2.2` and refreshes `uv.lock`. This release introduces truth-subspace retrieval improvements, opt-in feedback weighting, and reliability fixes for S3-backed LanceDB setups. ### Highlights * Adds the truth subspace builder, which compiles accepted session learnings into centroids and slots that can be used to align and rerank retrieval results. * Adds opt-in truth-subspace reranking and learned feedback weighting for graph search. The default influence remains `0.0`; enable it with `DEFAULT_FEEDBACK_INFLUENCE` or per-call `feedback_influence` values. * Adds `build_truth_subspace` to the Improve API so truth-subspace indexes can be rebuilt as part of the enrichment flow. * Tracks the active dataset through request-local context so retrieval and background tasks can keep dataset-scoped truth state aligned. * Fixes LanceDB dataset provisioning for S3-backed system roots by avoiding direct local directory creation for S3 paths. * Adds demos and tests for truth-subspace building, reranking, feedback influence, and graph truth-state persistence. ### Other fixes * Removes the Sentry and Langfuse third-party observability integrations while keeping the OpenTelemetry tracing layer intact. The `Observer.LANGFUSE` enum value, the Langfuse branch in `get_observe()`, and the `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_HOST` configuration fields and environment variables are gone, and Sentry initialization is dropped from the API. The `@observe` decorator now maps only to OpenTelemetry — it emits an OTEL span when tracing is enabled (`COGNEE_TRACING_ENABLED=true`) and is a no-op otherwise — so existing `@observe` usage keeps working unchanged with no call-site edits. **Packaging (breaking):** the `monitoring` extra is removed in favor of the existing `tracing` extra, which installs only the OpenTelemetry API/SDK and OTLP exporters. Migrate with `pip install cognee[tracing]` in place of `pip install cognee[monitoring]`. Users who relied on Sentry or Langfuse should switch to an OTLP-compatible backend; configure it via `OTEL_EXPORTER_OTLP_ENDPOINT` and related `OTEL_*` variables (see [OpenTelemetry Tracing](/integrations/opentelemetry-tracing)). Lockfiles (`uv.lock` / `poetry.lock`) that still reference the removed `sentry-sdk` / `langfuse` packages should be regenerated. * Fixes a crash when running a pipeline in the background (`run_in_background=True`) with no explicit `datasets`. The background runner (`run_pipeline_as_background_process`) now reads the effective `user` from the run's `params` first and only falls back to the default user when none was supplied, then resolves the run across all datasets that user has write access to. Previously `user` was bound only on the fallback path, so the usual case — a user passed in `params` — left `user` unassigned and raised `UnboundLocalError: cannot access local variable 'user'` before the run started. No API or CLI changes are required. * Fixes a lock-starvation bug in single-session `improve()`. When `improve()` is called with one `session_id`, it holds a per-session lock so concurrent auto-improve, idle-watcher, and `SessionEnd` runs serialize instead of duplicating work. Previously the lock was released only after a successful run (and on early stage 1–2 failures), so an exception in a later stage — default enrichment (`memify`), the global context index, or the graph-to-session sync — left the lock held permanently, and every subsequent `improve()` for that session silently returned `{}` until the process restarted. All stages are now wrapped in a single `try/finally`, so the session lock is always released on exit regardless of which stage fails. No public signature, parameter, return type, configuration option, or environment variable changed. The fix prevents the issue from recurring; sessions already stuck from before the upgrade still need a process restart to clear the held lock (closes #3313, PR #3317). * Fixes a CLI startup crash on a fresh, uninitialized database (for example a new Postgres) when no `--user-id` is passed. When `resolve_cli_user()` resolves the default user, it now catches `DatabaseNotCreatedError`, runs the database migrations to create the schema, and retries — so the command proceeds with the default user instead of failing on first run. The recovery is automatic when resolving the default user, including omitted `--user-id` and non-strict fallback-to-default paths; normal calls against an already-initialized database are unaffected and incur no extra overhead. No new flag, configuration option, or environment variable is introduced (fixes #3267, PR #3308). * Allows one automatic retry on structured-output (instructor) calls in the generic LLM API adapter (`LLM_PROVIDER="custom"` and other generic OpenAI-compatible providers). The adapter's `acreate_structured_output` now passes `max_retries=2` to instructor on both the primary and the content-policy fallback request, where it previously allowed no retry. When the model returns output that fails instructor's schema parsing or validation, instructor reissues the request once before surfacing an `InstructorRetryException`. The user-visible effect is fewer transient structured-output failures, at the cost of a slight latency increase on the rare request that is retried. No public API signature, configuration option, or environment variable changed, and the existing retry-warning logs are unchanged (PR #3413). *** ## v1.2.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.1)** Patch release that bumps the package version from `1.2.0` to `1.2.1` and refreshes `uv.lock`. This release follows `v1.2.0` with targeted reliability fixes for dataset-scoped ingestion, background task lifetime, and dataset helper authorization. ### Highlights * Fixes `remember(..., dataset_id=...)` so it now forwards `dataset_id` to `add()`. Previously `dataset_id` was used only to build the `cognify()` target while `add()` silently ingested raw data into the default `main_dataset`, so `cognify()` ran on the intended (but empty) dataset and produced no new graph. Ingestion and graph building now target the same dataset. No API or migration changes are required; callers who passed `dataset_id` and saw missing results should upgrade. * Anchors fire-and-forget background tasks so Python's garbage collector can no longer abort them mid-run. Background syncs (`cognee.api.v1.sync.sync.sync`) and background pipeline runs (`run_pipeline_as_background_process`) now hold a strong reference to their in-flight `asyncio.Task` in a module-level set (`_BACKGROUND_SYNC_TASKS` / `_BACKGROUND_PIPELINE_TASKS`) until the task finishes, with a done-callback that discards the reference on completion. Previously the event loop kept only a weak reference, so the GC could collect a still-running task and silently abort a background sync or pipeline run. This fixes those intermittent silent aborts; the only side effect is a small, transient increase in retained memory while tasks run (released as each task completes). No public API signature, request/response schema, configuration option, or environment variable changed. * Fixes `cognee.datasets.has_data()` raising `AttributeError`. The method now forwards the full `User` object to its internal authorization helper instead of `user.id`, so calls succeed and return the expected `bool`. No signature, parameter, or behavioral change beyond the method no longer crashing. *** ## v1.2.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.0)** Release that promotes accumulated `dev` work after the `v1.2.0` development builds, bumping the package version to `1.2.0` and refreshing the lockfile (`uv.lock`). Highlights include ChromaDB search enhancements, BM25 lexical chunk search, search-answer reference evidence, session-context guidance enabled by default, and a range of Postgres/Neptune adapter, visualization, logging, and memory-stability fixes. ### Highlights * Adds ChromaDB vector search support for `include_payload=False`, so callers can omit metadata payloads from returned `ScoredResult` values when they only need ids and scores. * Adds ChromaDB `node_name` filtering for `search()` and `batch_search()`, including `OR` and `AND` semantics through `node_name_filter_operator`. * Prevents Entity and EntityType node id collisions by namespacing generated ids by node category. * Excludes internal `EntityType` taxonomy nodes and their `is_a` edges from the schema inventory output (`get_schema_inventory` and the visualize schema inventory endpoint). Consumers no longer receive a separate `EntityType` type group or `is_a` relationship aggregates; entity instances are still grouped under their resolved semantic type. * Improves ontology parsing for file-like inputs with filename/content-type detection, RDFLib fallback formats, and clearer initialization errors when parsing fails. * Reaps subprocess database workers deterministically at interpreter exit. The `cognee_db_workers` harness now registers an `atexit` handler that force-terminates any still-live LanceDB/Kuzu worker processes on shutdown, instead of relying on garbage-collector and `__del__` ordering that is not guaranteed to run at interpreter exit (notably for Windows `spawn` daemon workers). This helps avoid leftover worker processes and shutdown hangs when running with `graph_database_subprocess_enabled=true` or `vector_db_subprocess_enabled=true`. * Offloads the Ollama adapter's blocking client calls off the asyncio event loop. The `LLM_PROVIDER="ollama"` adapter wraps a synchronous OpenAI-compatible client, so its chat-completion, audio-transcription, and image/vision calls previously ran inline and blocked the running event loop for the full duration of each Ollama request, serializing concurrent async callers (for example the per-chunk extraction that `cognify()` fans out). These calls are now dispatched through `asyncio.to_thread`, so they execute in worker threads and no longer stall the loop. Public async signatures are unchanged and no configuration or migration changes are required; because requests now run in worker threads, any objects shared with the Ollama client should be thread-safe. * Serializes concurrent decodes on the shared llama.cpp local in-process model. The `LLM_PROVIDER="llama_cpp"` local (in-process) adapter now guards calls into its single `llama_cpp.Llama` instance with a lock, so the per-chunk extraction that `cognify()` fans out via `asyncio.gather`/`asyncio.to_thread` no longer decodes on the same non-thread-safe instance concurrently. This helps avoid native `GGML_ASSERT` crashes from corrupted KV-cache/logits state during local llama.cpp runs; in-process requests are now processed one at a time (use server mode for parallel decoding). * Simplifies the structured-output schema sent to the LLM during graph extraction when a custom `graph_model` (a `DataPoint` subclass) is used. `extract_content_graph` now converts the model to a plain `BaseModel` that keeps only the fields you declare on each subclass — DataPoint infrastructure fields (such as `id`, `created_at`, `version`, `type`, `belongs_to_set`) and the `metadata` field are dropped from the schema the LLM is asked to fill — and then rehydrates the LLM result back into your original `DataPoint` model via `model_validate`. The LLM extracts only your domain fields, while declared `metadata` defaults (for example `{"index_fields": ["name"]}`) are preserved on the rehydrated objects, so indexing behavior is unchanged. This is a no-action change for callers of the high-level extraction, `cognify`, and `remember` APIs. * Fixes Neptune (`GRAPH_DATABASE_PROVIDER="neptune"`) edge writes for relationship types that contain spaces, hyphens, or openCypher reserved words. The adapter now backtick-quotes (and escapes embedded backticks in) the relationship type when interpolating it into the generated openCypher `MERGE` statements for both single-edge and batched (`UNWIND`) edge upserts, preventing query syntax errors and unsafe interpolation. Also fixes the batched-edge fallback path so that when a batch insert fails, the per-edge retry iterates the edges for that relationship instead of the relationship grouping map. No configuration or migration changes are required, but generated/logged openCypher will now show backtick-quoted relationship-type names. * Reuses a single `aiohttp.ClientSession` across anonymous telemetry requests instead of opening a new session per call. This avoids a repeated DNS + TCP + TLS handshake to the telemetry endpoint on every event, helping lower latency and connection churn for telemetry. The shared session is created lazily inside the running event loop and rebuilt transparently when the loop changes (for example across tests or `asyncio.run` boundaries) or after it is closed; telemetry stays best-effort and never raises. No new configuration is required, and telemetry can still be turned off with `TELEMETRY_DISABLED=true`. * Tolerates a missing `dataset_database` table on PostgreSQL during startup migrations and pruning. The `run_startup_migrations()` vector step and the graph/vector prune routines now also catch the asyncpg `ProgrammingError` / `UndefinedTableError`, in addition to the SQLite `OperationalError` already handled. Running against a fresh PostgreSQL/pgvector database (for example the pgvector example) now skips the step with a warning instead of crashing with an undefined-table error. * Reduces peak memory use of the Postgres graph adapter (`GRAPH_DATABASE_PROVIDER="postgres"`) for graph node and edge relational upserts. `add_nodes`/`add_edges` now stream each batch to Postgres in fixed-size chunks (1000 rows per `INSERT ... ON CONFLICT` statement) instead of compiling one large multi-thousand-row statement, and JSONB property columns are serialized once at execute time via an engine-level `json_serializer` (the UUID/datetime-aware `JSONEncoder`) rather than a per-row `json.loads(json.dumps(...))` round-trip. This helps avoid the transient allocation churn and memory spikes seen on large single-batch writes (the commit reports roughly a 20x reduction). The number of rows written, the upsert/conflict semantics, and the data stored are unchanged; no configuration or migration changes are required. * Switches `CHUNKS_LEXICAL` search to BM25 ranking. Lexical chunk searches now rank exact-term matches with BM25 instead of the previous Jaccard-style scorer, and the retriever filters default stop words unless explicitly configured otherwise. API signatures stay the same, but result ordering can change for `SearchType.CHUNKS_LEXICAL`. * Adds lightweight references (Evidence) to completion-style search answers via a new `include_references` flag (default `true`) on `search()`, `recall()`, and the `POST /api/v1/search` and `POST /api/v1/recall` request bodies. When enabled, a deterministic `Evidence:` block is appended to the answer text, assembled in-process (no extra LLM call) from the retrieved chunk payloads, falling back to entity → chunk → document graph traversal when chunk metadata is missing. The response schema and return types are unchanged — Evidence is added to the answer text only. Because this changes default answer text, snapshot and evaluation baselines will diff; set `include_references=False` to restore the exact prior output. Older indexes lacking the new `document_id`/`document_name` chunk fields use the graph fallback where available or omit the Evidence block silently. * Disables local-variable rendering in logged exception tracebacks. `setup_logging()` now configures the console renderer with `RichTracebackFormatter(show_locals=False)`, so when an exception is logged the traceback no longer expands each frame's local variables. In the retrieval/search path those locals can hold graph objects carrying embedding vectors and deep node/edge references, and rendering them recursively spiked memory to multiple GB and OOM-killed the process (notably in CI) whenever an exception was logged mid-search. Tracebacks themselves are still logged; only the per-frame locals dump is omitted. No configuration changes are required. * Restores Cognee's safe uncaught-exception hook. `setup_logging()` now installs `sys.excepthook` so non-`KeyboardInterrupt` exceptions are logged through structlog before Python's default traceback is printed, and falls back to plain traceback output if rich rendering itself fails. No configuration changes are required. * Propagates relational `DATABASE_CONNECT_ARGS` SSL settings to the Postgres maintenance, PGVector, and graph Postgres engines, so connections to managed Postgres that enforce SSL (for example Neon, RDS/Aurora, Azure Database for PostgreSQL) succeed. Previously only the main relational engine received these args, so CREATE/DROP DATABASE maintenance, per-dataset PGVector engines, and the `GRAPH_DATABASE_PROVIDER="postgres"` graph engine could fail with missing-SSL errors. The maintenance engine also maps the libpq `sslmode` key to the asyncpg `ssl` key, and rewrites a Neon `-pooler.` host to its direct endpoint because CREATE/DROP DATABASE cannot run through Neon's PgBouncer pooler. No configuration schema change is required — supply asyncpg SSL options via the existing `DATABASE_CONNECT_ARGS` and they are now honored across Cognee's Postgres engines; the env unset stays a no-op for in-cluster Postgres. Deployments on managed Postgres with enforced SSL should upgrade. * Fixes two interaction glitches in the graph visualization (`visualize_graph`) story view. Clicking a node no longer displaces it: the click-vs-drag threshold is raised from 3px to 6px so trackpad jitter on a plain click is no longer treated as a drag that reheated the force simulation and sent the clicked node (and, in the **Force** and **Flow** layouts, the whole layout) flying off the canvas. In the **Story** layout, dragging now drives the node position directly instead of reheating the pinned grid, and a released node snaps back cleanly to its lane. Separately, the pipeline stage-header pills (shown in the **Story** and **Flow** layouts) are now drawn in a final pass after edges, nodes, and labels, so a dense graph panned toward the top of the viewport can no longer paint over them. Generated visualization HTML changes only; no API, configuration, or migration changes are required. * Retries the Kuzu/Ladybug JSON extension load on the live connection when it is missing at runtime. In the subprocess graph worker (`graph_database_subprocess_enabled=true`), if `LOAD EXTENSION` fails with a "not been installed" error, the worker now runs `INSTALL` on the active connection and retries the load once; if that `INSTALL` fails it raises with the real underlying cause instead of the generic load error. This recovers from cases where the best-effort warm-up install on the throwaway database did not complete (for example a transient network error while downloading the extension on a fresh machine). The warm-up install path also now logs its failure cause to stderr (`[ladybug worker] warm-up INSTALL JSON failed: ...`) instead of swallowing it silently, so these conditions are diagnosable from worker/CI logs. The retry may perform an extension install on the live connection and add a small startup delay; no configuration or migration changes are required. * Relaxes the bundled Ladybug graph-store dependency from the `ladybug==0.16.0` pin to `ladybug>=0.16.0,<0.18`, so installs can pick up the `0.17.x` line. The database migration worker's storage-version table now maps the `0.17` on-disk format (catalog code `41`) to `0.17.1`, so an existing `0.16.x`/`0.17.x` graph database is recognized as current and is not flagged for legacy migration (migration still targets only pre-`0.15.0` databases). No manual database schema changes are required; deployers upgrading should re-lock dependencies (refresh `uv.lock`) and redeploy database workers to pick up the new range. * Adds a session-context guidance layer and turns it on by default. The cache `AUTO_FEEDBACK` setting now defaults to `true` (previously `false`), so when `CACHING` is enabled, session-capable completion searches run one additional structured-output LLM call per answered turn under the resolved session (`default_session` when `session_id` is omitted) to analyze the current turn against the previous one. The analysis can rewrite the turn into an effective query used for retrieval, accumulate durable per-session guidance grouped into `goals`, `rules`, `preferences`, and `lessons_learned` that can be injected into later answers, and **gate** a follow-up turn — returning a short acknowledgement (the analysis reply, or `"Got it."`) instead of running retrieval and completion. The step fails open to answering the original query when analysis errors or no session is available. Because guidance and the effective query can change retrieval inputs, session answers and turn gating may differ from history-only sessions, and per-turn latency and token usage increase. Set `AUTO_FEEDBACK=false` to disable and restore plain conversation-history replay. See [Sessions and Caching](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback). * Bypasses the Instructor structured-output pipeline when `acreate_structured_output` is called with `response_model=str` on the default OpenAI, generic, and Ollama LLM adapters. Plain-text requests are now sent directly to the provider and the model's raw string content is returned, instead of being wrapped in Instructor's JSON/tool-call schema. This avoids repeated parse failures and retry storms on local llama.cpp-compatible servers that don't honor those schemas, and can lower latency for string completions. Rate limiting still applies to these direct calls. Passing a Pydantic model is unchanged — it still returns a validated model instance — so this is a no-action change for callers. * Fixes the condition that gates name-to-UUID resolution of the `datasets` argument in `search()`. The check was wrapped in a single-element list (`[all(...)]`), which is always truthy, so the name-resolution path ran for any non-`None` `datasets` value. It now runs only when every entry in `datasets` is a string. Passing dataset names (the documented usage) is unaffected; the only behavior change is that non-string entries supplied through `datasets` (for example already-resolved UUIDs) are no longer forced through name-based authorization lookup — pass UUIDs via `dataset_ids` as before. No API signature, default, or migration change is required. * Corrects two `.env.template` knob names that the config loader was ignoring. The template previously listed `LLM_MAX_TOKENS` and `EMBEDDING_MAX_TOKENS`, but Cognee's settings classes read these values from `LLM_MAX_COMPLETION_TOKENS` (default `16384`) and `EMBEDDING_MAX_COMPLETION_TOKENS` (default `8191`). Anyone who copied the old template and set the chunk-sizing limits under the previous names had them silently ignored, so chunk sizing fell back to the defaults. If you relied on those entries, rename them to `LLM_MAX_COMPLETION_TOKENS` / `EMBEDDING_MAX_COMPLETION_TOKENS` in your `.env`. No code or schema changes are required. The same `.env.template` update also documents additional already-supported settings (LLM/embedding tuning and rate limiting, chunking, session cache, graph/vector connection and subprocess tuning, Langfuse monitoring, llama.cpp, and the auth-token secrets) as commented examples with their defaults. * Fixes an `UnboundLocalError` in file metadata extraction (`get_file_metadata`) when the underlying file-like object cannot seek. `content_hash` is now initialized before the seek/hash attempt, so when `file.seek(0)` raises `io.UnsupportedOperation` (the error is still logged), metadata is returned with an empty `content_hash` instead of crashing. This makes ingestion more robust for non-seekable file-like inputs; no configuration or API changes are required. * Preserves structured search completions in the result payload. `SearchResultPayload.completion` now models a single `dict`, a Pydantic `BaseModel`, and a list of models in addition to the previous `str` / list-of-string / list-of-dict shapes, and a custom serializer dumps model instances to their `dict` representation. This fixes searches that pass a non-string `response_model` (typed LLM output via `retriever_specific_config`): the structured object is kept as-is instead of being dropped or coerced into an empty model. The default string-answer path is unchanged; no configuration or migration changes are required. * Caps LLM retries and recovers from over-length embedding input. The structured-output and transcription adapters (anthropic, azure\_openai, gemini, generic\_llm\_api, llama\_cpp, mistral, ollama, openai) now stop on a fixed number of attempts (`stop_after_attempt`) instead of the previous time-based `stop_after_delay(128)` window, and the instructor retry counts were lowered (for example structured-output generation tops out at 3–4 attempts, while Bedrock's and the other adapters' inner instructor `max_retries` drop to 1–2). This makes transient failures fail faster and at lower cost, with a small reduction in resilience to intermittent errors — watch your LLM error/latency/cost metrics after upgrading. Separately, `LiteLLMEmbeddingEngine` now recovers from over-length embedding input: a context-window error or a `400 BadRequestError` matching `maximum input length` triggers recursive split-and-pool (splitting the batch, or splitting a single string into overlapping halves and averaging the resulting vectors) instead of failing, while other 400 errors still fail fast. No API or configuration changes are required. See [Embedding Providers → Timeout and Retry Behavior](/setup-configuration/embedding-providers). * Bounds the input-data preview persisted in the `pipeline_runs.run_info` column so a single run cannot grow the table without limit. On pipeline run start, error, and completion, the audit-only `run_info` data is summarized: a list of `Data` records is still reduced to their IDs and empty input is still recorded as `"None"`, but any other payload is now stringified and truncated to a 512-character preview ending with `... [truncated, <N> chars total]` instead of being stored verbatim. `run_info` is never read back during processing; persist large raw inputs (for example text passed to `add()`/`cognify()`) elsewhere if you need the full payload. No configuration or migration changes are required. * Fixes `forget(everything=True)` under multi-tenant per-dataset database isolation (`ENABLE_BACKEND_ACCESS_CONTROL=true`). The `everything` branch no longer runs inside a single-dataset database context; the per-dataset context is now established per dataset inside the underlying delete-all flow. Previously, entering a single-dataset context with no dataset reference could try to create a `dataset_database` row for a non-existent dataset and fail the operation. Single-dataset, single-item, and `memory_only` modes are unchanged, and the public `forget()` signature, return shapes, and error messages are unchanged. * Forwards `FALLBACK_ENDPOINT` to the OpenAI adapter's content-policy fallback request (`LLM_PROVIDER="openai"`). Previously this `api_base` override was not applied, so the fallback completion always went to the default OpenAI endpoint even when `FALLBACK_ENDPOINT` was set; now the fallback request is routed to the configured base URL. Deployments that set `FALLBACK_ENDPOINT` to an OpenAI-compatible proxy or alternate endpoint will see their fallback traffic go there. `FALLBACK_ENDPOINT` remains optional for `openai` — when unset, the fallback still uses the default OpenAI endpoint. * Caps the `instructor` dependency at `<1.15.3` (previously `<2.0.0`) and lowers the `litellm` minimum to `>=1.83.7` (previously `>=1.84.0`). This pins structured-output extraction to a known-good `instructor` range and widens the compatible `litellm` window; lockfiles (`poetry.lock`, `uv.lock`) are refreshed to match. No API or behavioral changes — callers using the high-level `cognify`/`search` APIs are unaffected. Developers and deployers should re-lock and reinstall dependencies to pick up the new constraints. * Detects Markdown, JSON, XML, and YAML files by extension during file-type guessing. `guess_file_type` now returns deterministic types for `.md`/`.markdown` (`text/markdown`), `.json` (`application/json`), `.xml` (`application/xml`), and `.yaml`/`.yml` (`application/yaml`) instead of relying on content-based detection, which has no magic-number signature for these formats and fell back to `text/plain`/`txt`. The recorded file metadata (`mime_type` and `extension`) for these files now reflects their actual format. Loader selection is unchanged — `TextLoader` already handled these extensions — so no action or migration is required. * Adds a `GET /api/v1/proposals/{proposal_id}` endpoint for reviewing a stored skill-improvement proposal before applying it. The endpoint takes a required `dataset_id` query parameter and returns the proposal's `status` (`proposed`/`applied`), `confidence`, `rationale`, `model_name`, and before/after procedures (`old_procedure`/`proposed_procedure`); it is read-only and never mutates the graph (applying still goes through `POST /api/v1/remember/entry` with `skill_improvement`). It returns `403` when the caller is not authorized for the dataset and `404` when the proposal is not found. * Adds no-code, inline skill ingestion. `POST /api/v1/remember` (with `content_type=skills`) now accepts `skills_text` (a `SKILL.md` markdown body as a string) and `skill_name` (the skill name/slug, defaults to `skill`) form fields, so a skill can be ingested without uploading a file — when `skills_text` is set and no files are uploaded, it is written to a temporary `SKILL.md` and ingested through the existing skills pipeline. A new `POST /api/v1/skills` endpoint exposes the same inline ingestion via a JSON body (`skills_text`, optional `skill_name`, and one of `dataset_name`/`dataset_id`). ### Notes * Includes a behavior-preserving cleanup of the LiteLLM embedding engine (`LiteLLMEmbeddingEngine`): no public `__init__` signature, env-var (`MOCK_EMBEDDING`, `EMBEDDING_ENDPOINT`), or default changes, and embedding behavior is unchanged. * Deployers upgrading should re-lock dependencies (refresh `uv.lock`) and reinstall, then rebuild/redeploy to pick up the updated dependency set. * The ontology parser update improves file-like parsing behavior; upload endpoint format restrictions should be documented separately if they change. *** ## v1.1.3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.3)** Patch release focused on API-mode robustness and dependency safety. It enables remote pipeline status checks for MCP/API deployments, improves vector retrieval behavior for empty input, and tightens the `instructor` dependency range. ### Highlights * Enables `cognify_status` in API mode. The MCP can resolve dataset IDs remotely and read pipeline status from `GET /api/v1/datasets/status`, so self-hosted API deployments can check background pipeline status without local database access. * Adds API-mode support to `CogneeClient.get_pipeline_status`, which now queries the server's `/api/v1/datasets/status` endpoint instead of raising `NotImplementedError`. * Makes LanceDB retrieval return an empty list when called with an empty id list, preventing avoidable errors for callers that sometimes have no vector ids to fetch. * Pins `instructor` below `1.15.3` and refreshes lock metadata. Deployers with exact dependency pins should re-lock or reinstall against the updated constraints. * Refreshes the README with clearer Cognee positioning, branding, and a research paper link. *** ## v1.1.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.2)** Patch release with a refreshed public frontend, improved Cloud UI workflows, and a Postgres graph adapter compatibility fix for asyncpg/PostgreSQL 16. ### Highlights * Syncs the public frontend with the SaaS application, bringing updated dashboard, search, dataset, connection, onboarding, knowledge graph, and graph model editor experiences. * Adds conversation-based search history and refreshed multi-dataset search flows in the frontend. * Improves connection and onboarding flows with a connection modal, step-by-step agent setup guidance, new quickstart assets, and updated loading visuals. * Adds memory customization UI support for datasets, including graph models, custom prompts, and ontology-related configuration. * Fixes Postgres graph neighborhood expansion under asyncpg/PostgreSQL 16 by casting recursive CTE seed parameters to `text[]`. ### Notable Changes * Bumps the package version from `1.1.1` to `1.1.2` and refreshes lockfiles. * Aligns frontend API routes and local development behavior with the OSS backend. * Updates API key, tenant, configuration, dataset, ingestion, ontology, search-history, session, analytics, and user frontend modules. * Adds frontend assets for quickstarts, agent integrations, loading states, and graph previews. * Adds regression coverage for Postgres graph neighborhood seed array typing and retries a flaky usage-logger e2e path in CI. ### Fixes and Improvements * **Postgres neighborhood query parameter typing**: The Postgres graph adapter's `get_neighborhood` query now casts the seed parameter to `text[]` (`unnest(CAST(:seeds AS text[]))`) in its recursive CTE seed row. Deployments using `GRAPH_DATABASE_PROVIDER=postgres` with asyncpg/PostgreSQL 16 should no longer hit parameter type inference errors when expanding neighbors from seed node ids. * **Cloud UI refresh**: Dashboard, dataset, dataset detail, connections, search, onboarding, knowledge graph, and graph model editor screens were refreshed and aligned with current Cloud workflows. * **Search and dataset workflows**: Search now supports conversation history and multi-dataset recall flows, while dataset pages add improved status polling, graph access, and memory customization entry points. * **Connect Agent flow**: The frontend adds clearer connection setup prompts, modal-based setup guidance, and integration visual assets. * **Frontend resilience**: Error handling, loading states, analytics logging, tenant context, user configuration, and local fetch behavior were updated across the public frontend. *** ## v1.1.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.1)** Patch release that promotes accumulated `dev` work after `v1.1.1.dev0`, with agent-management APIs, graph visualization updates, custom graph-model support in `remember`, and backend stability fixes. ### Highlights * Adds agent management and connection endpoints for listing, creating, inspecting, registering, unregistering, and deleting agents and their active connections. * Reworks graph visualization with a pipeline-aware Story layout, Schema view, improved labels, legends, and modular visualization components. * Adds `graph_model` support to the `remember` REST endpoint, letting API callers pass a JSON-serialized graph schema into ingestion. * Expands graph and retrieval behavior with local Neo4j dataset handling, global context graph bucketing, improved edge text, and `node_name` filtering for chunk retrieval. * Improves LLM, PGVector, remember/session, prune, forget, and graph-projection error handling. ### Notable Changes * Bumps the package version from `1.1.0` to `1.1.1` and refreshes the release lockfiles. * Splits agent lifecycle and connection handling into dedicated modules and API routes, including persisted agent connection state and agent-session names. * Adds SDK/API support for retrieving specific agent configuration and for inspecting current agent connections. * Adds local Neo4j dataset database handling and updates graph database selection to recognize that handler. * Reworks global context index internals with graph bucketing, scoring, build, update, load, summarize, and persistence flows. * Improves edge indexing and rendering by preserving natural edge descriptions, generating fallback edge text from metadata, and rendering relationship labels inside edge markup. * Updates CI and test coverage across database adapters, agents, visualization, global context indexing, retrieval filters, and LLM configuration. ### Fixes and Improvements * **Remember custom graph models**: The `remember` REST endpoint now accepts an optional `graph_model` form field, parses the JSON schema into a graph model, and forwards it into the ingestion flow. * **Agent lifecycle and connections**: Agent endpoints now separate agent resources from agent connections, support agent-session names, persist connection metadata, mark unregistering agents inactive, and expose connection detail. * **Graph visualization**: Story view spacing, column pinning, schema rendering, edge-label rendering, and fallback labeling were improved so generated graph views are easier to inspect. * **Graph ingestion and retrieval**: Edges with unprojectable endpoints are skipped instead of failing graph projection, `KnowledgeGraph` subclasses follow the knowledge-graph integration path, chunk retrieval receives `node_name` filters, and `forget` can handle dataset values that are string UUIDs. * **PGVector metadata consistency**: `create_collection` now reflects SQLAlchemy metadata only after the table-creation transaction commits, avoiding stale metadata entries when table creation rolls back. * **LLM adapters**: Generic LLM API transcription and Ollama image transcription now raise clear `ValueError` messages for empty responses, Mistral guards against `None` messages before reading content, and OpenAI instructor mode is honored. * **Session remember routing**: `remember(session_id=...)` now routes through the JSON `/entry` endpoint in API mode, and using `custom_prompt` with `session_id` raises a clear `ValueError`. * **Operational stability**: Prune errors and dataset lookup issues are handled more defensively, brittle batch-query test settings were adjusted, and optional LLM configuration can be passed through CI. *** ## v1.1.0.dev1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.0.dev1)** Developer preview release on the way to `v1.1.0dev1`. This release includes API, retrieval, permissions, storage-runtime, and backend consistency changes. ### Highlights * Adds database subprocess workers for LanceDB and Kuzu so native database work can run outside the main Cognee process. The wheel now includes the `cognee_db_workers` package. * Exposes more ingestion controls through the public API and remote client paths, including chunk sizing and background execution options for `remember()` and `cognify()`. * Adds `dataset_ids` support to `recall()`, making shared-dataset retrieval more reliable when dataset names are not owned by the calling user. * Expands permission management with DELETE endpoints for dataset permissions, roles, and user-role membership. * Improves session visibility so parent users can see sessions created by child-agent users where appropriate. ### Notable Changes * Adds `graph_database_subprocess_enabled` and `vector_db_subprocess_enabled` configuration, plus Kuzu tuning variables for threads, buffer pool size, and max DB size. * Keeps `belongs_to_set` metadata consistent across dataset deletion and shared-node/vector upserts in LanceDB, PGVector, and Neo4j paths. * Adds `include_payload` behavior to Neptune Analytics vector search. * Improves Postgres hybrid batching by respecting embedding-engine batch size. * Improves infer-schema text sampling and prompting. * Rewrites the examples README into a fuller index and adds performance-testing support with Locust. * Deprecates `.env.example` as the canonical template in favor of `.env.template`. * Bumps the package version from `1.0.9` to `1.1.0.dev1` and refreshes lockfiles. *** ## v1.0.3 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.3)** Patch release with bug fixes and stability improvements on top of v1.0.2. ### Highlights * Promotes accumulated `dev` work to `main` for the `v1.0.3` release * Adds session lifecycle APIs, unified memory/session handling, and dashboard support * Introduces dataset queueing for async context management and ingestion flows * Ships new relational migrations, including session lifecycle tables and `parent_user_id` * Expands recall/remember and cloud routing behavior, plus frontend onboarding and Connect Agent updates ### Notable Changes * Added session endpoints, metrics, and supporting persistence work * Added dataset queue infrastructure and follow-up fixes for background processing * Added database migrations for new tables and user/dataset ownership handling * Updated recall, remember, improve, and search-related API behavior * Added frontend work for Connect Agent, dashboard/activity views, API keys, and onboarding * Included guide updates, workflow/tooling changes, and dependency updates such as `litellm` and `onnxruntime` ### Bug Fixes * **PostgreSQL null-byte compatibility**: Embedded null bytes (`\x00`) in node or edge string fields no longer cause errors when using PostgreSQL as the relational backend. Null bytes are now automatically stripped from all string values (including nested attributes) before writes to the relational store. This sanitization is transparent — affected strings are silently cleaned rather than rejected. * Fixed duplicate graph nodes caused by `DataPoint.id` being regenerated during graph construction. The original `id` is now preserved when converting DataPoint instances into graph nodes, ensuring node identity is stable across graph extraction passes. *** ## v1.0.2 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.2)** Patch release with bug fixes and stability improvements on top of v1.0.1. ### Bug Fixes * **LanceDB schema migration**: "contained null values" errors (raised when old rows lack a field required by a newer DataPoint schema) are now treated as recoverable schema drift. The affected table is automatically rebuilt from the current schema instead of raising a hard failure. * **cognee-mcp Docker image build**: Added missing `build-essential` and `libpq-dev` system packages to the builder stage so that `cognee[postgres]` can compile `psycopg2` from source on Linux. ### Dependency Updates * Bumped `llama-index-core` requirement from `>=0.13.0,<0.14` to `>=0.14.20,<0.15` for the `llama-index` extra. * Pinned `nltk>=3.9.3,<4` explicitly in the `docs` extra to satisfy `unstructured`'s dependency until `unstructured` v0.21.0. *** ## v1.0.1 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.1)** Patch release with bug fixes on top of v1.0.0. *** ## v1.0.0 **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.0)** ### Highlights * **New high-level API**: `remember`, `recall`, `improve`, and `forget` cover the full memory lifecycle in four operations * Session-aware memory via `session_id` — short-term context that can be promoted into the permanent graph * Unified `recall` replaces the previous `search` call with automatic retrieval strategy selection * Legacy operations (`add`, `cognify`, `search`, `memify`) remain available as lower-level building blocks ### New Features * `cognee.remember(data, session_id=...)` — ingest and graph in one call; supports permanent or session memory * `cognee.recall(query, session_id=...)` — query across both the permanent graph and session cache * `cognee.improve(...)` — enrich an existing graph with feedback-based weighting and session promotion * `cognee.forget(dataset=..., session_id=...)` — delete data, datasets, or full session memory *** ## v0.5.4.dev1 **Released:** March 5, 2026\ **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.4.dev1)** ### Highlights * Developer preview release focused on quality, performance, and developer ergonomics * Faster ingestion and sync * Improved search relevance and new filtering options * Stability fixes for memory creation, deletion, and CLI workflows * Internal refactoring and dependency upgrades ### New Features * Bulk import CLI for faster batched ingestion * Search filters for tags and date ranges * Optional per-collection ingestion throttling ### Improvements * Lower latency for ingestion and sync * Better search ranking * More robust deletion and duplicate handling * Clearer CLI messages and debug logs ### Bug Fixes * Fixed duplicate memories under concurrent ingestion * Fixed partial state after deletion * Fixed CLI export formatting issues * Fixed intermittent retrieval failures under load *** ## v0.5.3 **Released:** February 27, 2026\ **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.3)** ### Highlights * New graph visualization improvements * Expanded permissions and user management work * SessionManager and cache/session persistence work * Search and graph retrieval improvements * Multiple stability and CI/CD fixes ### Notable Changes * Added role-based permission checks and permission endpoints * Added graph visualization updates, including note set coloring * Added return type hints to API functions * Added chunk associations for the memify pipeline * Added vector filtering based on node sets * Fixed delete flow bugs, health check issues, MCP issues, and several config/integration issues *** ## v0.5.3.dev1 **Released:** February 20, 2026\ **[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.3.dev1)** ### Highlights * Added vector filtering based on node sets * Added principal Cognee configuration * Fixed health check issues * Fixed FalkorDB adapter port bug * Fixed Ollama image ingestion argument issue ### Notes * Includes a small set of targeted fixes and feature work on top of `v0.5.3.dev0` * Introduced one new contributor in this release