`BeautifulSoupLoader` parses HTML files using CSS selectors. By default it applies a comprehensive set of extraction rules covering common HTML content areas (headings, paragraphs, articles, tables, code blocks, etc.). You can pass your own `extraction_rules` dict to target specific elements.
Each rule is a dict with the following optional keys:
| Key | Type | Description |
| ----------- | ------ | -------------------------------------------------------------- |
| `selector` | `str` | CSS selector to match elements |
| `xpath` | `str` | XPath expression (requires `lxml`) |
| `attr` | `str` | HTML attribute to extract instead of text content |
| `all` | `bool` | If `True`, extract all matches; otherwise only the first |
| `join_with` | `str` | String used to join multiple extracted values (default: `" "`) |
```python theme={null}
import cognee
await cognee.remember(
data=["page.html"],
preferred_loaders=[
{
"beautiful_soup_loader": {
"extraction_rules": {
"title": {"selector": "h1", "all": False},
"body": {"selector": "article.main-content", "all": True, "join_with": "\n\n"},
"og_image": {"selector": "meta[property='og:image']", "attr": "content"}
}
}
}
]
)
```
**Overlapping CSS rules are deduplicated.** Rules are applied in the order they appear in `extraction_rules`, against a single parsed document. Once a rule extracts an element, elements nested inside it are skipped by later rules, so content covered by a broad selector is not emitted a second time by a narrower one. With the default rules, for example, `article` runs before `paragraphs`, so a `` inside an `` is extracted once as part of the article rather than repeated. Deduplication works in one direction only: a broad rule listed *after* a narrower one is not suppressed by it and re-emits the shared content anyway, so list broader selectors before narrower ones — as the default rules do — to avoid duplicates.
Deduplication only skips *descendants* of already-extracted elements. Two rules whose selectors match the exact same element still extract that element twice. Matches within a single rule are likewise not deduplicated against each other, so one selector matching both an element and its descendant emits the shared text twice.
For XPath-based extraction (requires `pip install lxml`):
```python theme={null}
await cognee.remember(
data=["page.html"],
preferred_loaders=[
{
"beautiful_soup_loader": {
"extraction_rules": {
"content": {"xpath": "//div[@class='content']//p"}
}
}
}
]
)
```
XPath rules are evaluated against a separate `lxml` tree and do not take part in the deduplication described above. An XPath rule never suppresses a later CSS rule, and it is never suppressed by an earlier one — so mixing XPath and CSS rules that cover the same elements can still repeat content.