Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 47 additions & 1 deletion src/removals/selectors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,13 +9,49 @@ import {
FOOTNOTE_LIST_SELECTORS
} from '../constants';
import { DebugRemoval } from '../types';
import { textPreview, logDebug } from '../utils';
import { textPreview, logDebug, normalizeText } from '../utils';
import { getClassName, hasResponsiveShowClass } from '../utils/dom';

const MAJORITY_CONTENT_MIN_TEXT_LENGTH = 200;
const MAJORITY_CONTENT_RATIO = 0.8;
const AMBIGUOUS_MAJORITY_EXACT_SELECTOR = [
'[class^="ad-" i]',
'[class$="-ad" i]',
'[id^="ad-" i]',
'[id$="-ad" i]'
].join(',');
const ALWAYS_REMOVABLE_EXACT_TAGS = new Set([
'SCRIPT',
'STYLE',
'NOSCRIPT',
'IFRAME',
'FRAME',
'FRAMESET',
'OBJECT',
'EMBED',
'APPLET',
'BASE',
'META',
'LINK'
]);

function isAlwaysRemovableExactElement(el: Element): boolean {
return ALWAYS_REMOVABLE_EXACT_TAGS.has(el.tagName.toUpperCase());
}

function isAmbiguousMajorityExactElement(el: Element): boolean {
return (
(el.tagName.includes('-') && el.matches('.ad:not([class*="gradient"])')) ||
el.matches(AMBIGUOUS_MAJORITY_EXACT_SELECTOR)
);
}

export function removeBySelector(doc: Document, debug: boolean, removeExact: boolean = true, removePartial: boolean = true, mainContent?: Element | null, debugRemovals?: DebugRemoval[], skipHiddenExactSelectors: boolean = false) {
const startTime = Date.now();
let exactSelectorCount = 0;
let partialSelectorCount = 0;
const mainContentTextLength = mainContent ? normalizeText(mainContent.textContent || '').length : 0;
const shouldGuardMajorityContent = mainContentTextLength >= MAJORITY_CONTENT_MIN_TEXT_LENGTH;

// Track all elements to be removed, with their match type
const elementsToRemove = new Map<Element, { type: 'exact' | 'partial'; selector?: string }>();
Expand Down Expand Up @@ -138,6 +174,16 @@ export function removeBySelector(doc: Document, debug: boolean, removeExact: boo
if (mainContent && el.contains(mainContent)) {
return;
}
if (
shouldGuardMajorityContent &&
type === 'exact' &&
!el.matches(HIDDEN_EXACT_SELECTOR) &&
!isAlwaysRemovableExactElement(el) &&
isAmbiguousMajorityExactElement(el) &&
normalizeText(el.textContent || '').length >= mainContentTextLength * MAJORITY_CONTENT_RATIO
) {
return;
}
if (el.tagName === 'A' && el.closest('h1, h2, h3, h4, h5, h6')) {
return;
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
<body> <div> <p> The Modular Supply Cart keeps shared equipment organized for small teams that need a predictable workflow without a dedicated storage room. </p> <H2>Description</H2> <p> Each cart ships with removable bins, a labeled front panel, and locking casters for stable positioning during inventory checks. The frame is designed for common shop, studio, and classroom supplies. </p> <H2>Features</H2> <ul> <li>Adjustable shelves with three height positions</li> <li>Four removable bins for cables, tools, and consumables</li> <li>Integrated label rail for team-specific organization</li> </ul> <H2>Specifications</H2> <ul> <li>Powder-coated steel frame with recycled plastic bins</li> <li>Supports up to 120 pounds of evenly distributed equipment</li> <li>Ships flat with hardware and assembly instructions</li> </ul> </div> </body>
26 changes: 26 additions & 0 deletions tests/expected/issues--283-clutter-selector-removes-content.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
```json
{
"title": "Modular Supply Cart",
"author": "",
"site": "Example Catalog",
"published": ""
}
```

The Modular Supply Cart keeps shared equipment organized for small teams that need a predictable workflow without a dedicated storage room.

## Description

Each cart ships with removable bins, a labeled front panel, and locking casters for stable positioning during inventory checks. The frame is designed for common shop, studio, and classroom supplies.

## Features

- Adjustable shelves with three height positions
- Four removable bins for cables, tools, and consumables
- Integrated label rail for team-specific organization

## Specifications

- Powder-coated steel frame with recycled plastic bins
- Supports up to 120 pounds of evenly distributed equipment
- Ships flat with hardware and assembly instructions
60 changes: 60 additions & 0 deletions tests/fixtures/issues--283-clutter-selector-removes-content.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<!-- {"url":"https://example.com/products/modular-supply-cart/"} -->
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta property="og:site_name" content="Example Catalog">
<meta property="og:title" content="Modular Supply Cart">
<title>Modular Supply Cart | Example Catalog</title>
</head>
<body>
<header class="site-header">
<a href="/">Example Catalog</a>
<nav>
<a href="/products/">Products</a>
<a href="/support/">Support</a>
</nav>
</header>

<app-root class="ad" ng-version="17.0.0">
<script type="application/json">
{"payload":"Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML. Runtime state should not appear in extracted HTML."}
</script>
<h1>Modular Supply Cart</h1>
<p>
The Modular Supply Cart keeps shared equipment organized for small teams that need
a predictable workflow without a dedicated storage room.
</p>

<h2>Description</h2>
<p>
Each cart ships with removable bins, a labeled front panel, and locking casters for
stable positioning during inventory checks. The frame is designed for common shop,
studio, and classroom supplies.
</p>

<h2>Features</h2>
<ul>
<li>Adjustable shelves with three height positions</li>
<li>Four removable bins for cables, tools, and consumables</li>
<li>Integrated label rail for team-specific organization</li>
</ul>

<h2>Specifications</h2>
<ul>
<li>Powder-coated steel frame with recycled plastic bins</li>
<li>Supports up to 120 pounds of evenly distributed equipment</li>
<li>Ships flat with hardware and assembly instructions</li>
</ul>
</app-root>

<div class="print-copy" style="display: none">
<h1>Modular Supply Cart</h1>
<p>This duplicate print copy should be removed as hidden content.</p>
</div>

<footer>
<p>Example Catalog footer links</p>
</footer>
</body>
</html>
Loading