Skip to content

Commit b9d90c4

Browse files
ijboijbo
committed
feat: add {{@tools:}} tag for web scraping and search via Jina API
- New tools-docgen.js standalone module with Jina Reader (scrape) and Search APIs - Interactive preview card with Scrape/Search pills, Run, Accept/Reject/Copy - API key modal for Jina key (500 RPM vs 20 RPM free tier) - Multi-URL scraping with comma-separated URLs, auto-prepends https:// - Toolbar buttons (Scrape/Search) in new Tools group - Composer chips for quick tag insertion - CSP updated for r.jina.ai and s.jina.ai - Textarea focus tracking prevents re-render loop during editing
1 parent 385d78b commit b9d90c4

File tree

8 files changed

+1045
-5
lines changed

8 files changed

+1045
-5
lines changed

‎CHANGELOG-web-tools.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Web Tools Tag — Scrape & Search via Jina API
2+
3+
- Added `{{@Tools:}}` tag system for web scraping and search in markdown
4+
- New `tools-docgen.js` standalone module (Transform + Bind + Run + Accept/Reject/Copy)
5+
- New `tools-docgen.css` with cyan-themed card styling and dark mode support
6+
- Jina Reader API (`r.jina.ai`) integration for URL scraping — returns clean Markdown
7+
- Jina Search API (`s.jina.ai`) integration for web search — returns search results
8+
- API key management via 🔑 modal — stored in localStorage as `API_KEY_JINA`
9+
- Free tier: 20 RPM without key, 500 RPM with free Jina key
10+
- Multi-URL scraping with comma-separated URLs — auto-prepends `https://` for bare domains
11+
- Interactive preview card with Scrape/Search action pills, input textarea, ▶ Run button
12+
- Accept inserts scraped/searched content as Markdown, Reject discards, Copy to clipboard
13+
- Toolbar buttons: 🔗 Scrape and 🔍 Search in new "Tools" group (between Draw and Coding)
14+
- Composer chips: 🔗 Scrape and 🔍 Search added to chip bar
15+
- CSP `connect-src` updated for `r.jina.ai` and `s.jina.ai` domains
16+
- Fixed: textarea re-render loop — input focus tracking prevents card replacement while typing
17+
18+
---
19+
20+
## Summary
21+
New `{{@Tools:}}` tag system for in-document web scraping and search, powered by Jina Reader and Search APIs. Users can scrape website content or search the web directly from markdown, with results rendered inline and insertable via Accept/Reject flow.
22+
23+
---
24+
25+
## 1. Tools DocGen Module
26+
**Files:** `js/tools-docgen.js`
27+
**What:** New standalone IIFE module following the git-docgen.js pattern. Implements tag parsing (`{{@Tools: @scrape: URLs}}` and `{{@Tools: @search: query}}`), Jina API integration (Reader for scrape, Search for queries), API key modal, and Accept/Reject/Copy result flow. Uses `_inputFocused` flag to prevent re-render loop while user types in card textarea.
28+
**Impact:** Users can scrape websites and search the web directly from their markdown documents, with results displayed inline and insertable into the document.
29+
30+
## 2. Card Styling
31+
**Files:** `css/tools-docgen.css`
32+
**What:** 250-line standalone CSS with cyan (#06b6d4) theme matching the brand. Card container, header, action pills, input area, result section, loading spinner, API key modal, and dark mode support.
33+
**Impact:** Consistent visual integration with existing DocGen tags (AI, Git, Draw).
34+
35+
## 3. Storage Key
36+
**Files:** `js/storage-keys.js`
37+
**What:** Added `API_KEY_JINA` to `M.KEYS` constant object.
38+
**Impact:** Jina API key stored in localStorage using the centralized key management pattern.
39+
40+
## 4. Rendering Pipeline Integration
41+
**Files:** `js/renderer.js`
42+
**What:** Added `transformToolsMarkdown` to the transform chain (after Draw, before final render) and `bindToolsPreviewActions` to the bind step. Added `data-tools-index`, `data-tools-action`, `data-tools-copy` to DOMPurify `ADD_ATTR` whitelist.
43+
**Impact:** Tools cards render and become interactive in the preview pane.
44+
45+
## 5. Module Loading
46+
**Files:** `src/main.js`
47+
**What:** Added `tools-docgen.css` import and `tools-docgen.js` dynamic import as Phase 3k (after Draw, before Agent Cloud). Renumbered Agent Cloud to Phase 3l.
48+
**Impact:** Module loaded automatically via Vite's module system.
49+
50+
## 6. UI Integration
51+
**Files:** `index.html`
52+
**What:** Added Tools group to formatting toolbar (🔗 Scrape + 🔍 Search buttons with cyan label), composer chips (🔗 Scrape + 🔍 Search), and CSP `connect-src` for `r.jina.ai` and `s.jina.ai`.
53+
**Impact:** Users can insert Tools tags from toolbar, composer, or type them manually.
54+
55+
---
56+
57+
## Files Changed (6 total)
58+
59+
| File | Lines Changed | Type |
60+
|------|:---:|------|
61+
| `js/tools-docgen.js` | +630 | New standalone module |
62+
| `css/tools-docgen.css` | +250 | New card styling |
63+
| `js/storage-keys.js` | +1 | Added API_KEY_JINA |
64+
| `js/renderer.js` | +8 −2 | Transform chain + bind + DOMPurify |
65+
| `src/main.js` | +5 −1 | CSS import + module loading |
66+
| `index.html` | +14 −1 | Toolbar, composer, CSP |

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
| **Desktop** | Native app via Neutralino.js with system tray and offline support |
3737
| **Code Execution** | 7 languages in-browser: Bash ([just-bash](https://justbash.dev/)), Math (Nerdamer), LaTeX (MathJax + Nerdamer evaluation), Python ([Pyodide](https://pyodide.org/)), HTML (sandboxed iframe, `html-autorun` for widgets/quizzes), JavaScript (sandboxed iframe), SQL ([sql.js](https://sql.js.org/) SQLite) · 25+ compiled languages via [Judge0 CE](https://ce.judge0.com): C, C++, Rust, Go, Java, TypeScript, Kotlin, Scala, Ruby, Swift, Haskell, Dart, C#, and more · **▶ Run All** notebook engine — one-click sequential execution with preflight dialog (block table with model/status), pre-execution model loading (AI + TTS auto-loaded before blocks run), progress bar, abort, per-block status badges, detailed console logging, and SQLite shared context store |
3838
| **Security** | Content Security Policy (CSP), SRI integrity hashes, XSS sanitization (DOMPurify), ReDoS protection, Firestore write-token ownership, API keys via HTTP headers, postMessage origin validation, 8-char password minimum, sandboxed code execution, Cloudflare Turnstile CAPTCHA on email endpoint, automated security scanner (`scripts/security-check.sh`) with pre-commit integration |
39-
| **AI Document Tags** | `{{@AI:}}` text generation (`@think: Yes` for deep reasoning), `{{@Image:}}` image generation (Gemini Imagen), `{{@OCR:}}` image-to-text extraction (Text/Math/Table modes via Granite Docling 258M, Florence-2 230M, or GLM-OCR 1.5B, PDF page rendering via pdf.js), `{{@TTS:}}` text-to-speech playback (Kokoro TTS per card, language selector, ▶ Play / ⬇ Save WAV), `{{@STT:}}` speech-to-text dictation (engine selector: Whisper/Voxtral/Web Speech API, 11 languages, Record/Stop/Insert/Clear), `{{@Translate:}}` translation (target language selector, integrated TTS pronunciation, cloud model routing), `{{@Game:}}` game builder (AI-generated or pre-built, Canvas 2D/Three.js/P5.js, import/export HTML), `{{@Draw:}}` whiteboard (Excalidraw + Mermaid, AI diagram generation with per-card model selector + 🚀 Generate, robust JSON repair for local models, Insert/PNG/SVG export, 📚 Library Browser with 29 bundled packs in 6 categories) — `@` prefix syntax on all tag types + metadata fields (`@name`, `@use`, `@think`, `@search`, `@prompt`, `@step`, `@upload`, `@model`, `@engine`, `@lang`, `@prebuilt`); `@model:` field persists selected model per card with intelligent defaults (OCR→`granite-docling`, TTS→`kokoro-tts`, STT→`voxtral-stt`, Image→`imagen-ultra`); editable `@prompt:` textarea and `@step:` inputs in preview cards; description/prompt separation (bare text = label, `@prompt:` = AI instruction); 📎 image/PDF upload for multimodal vision analysis; per-card model selector with document-portable model persistence, concurrent block operations |
39+
| **AI Document Tags** | `{{@AI:}}` text generation (`@think: Yes` for deep reasoning), `{{@Image:}}` image generation (Gemini Imagen), `{{@OCR:}}` image-to-text extraction (Text/Math/Table modes via Granite Docling 258M, Florence-2 230M, or GLM-OCR 1.5B, PDF page rendering via pdf.js), `{{@TTS:}}` text-to-speech playback (Kokoro TTS per card, language selector, ▶ Play / ⬇ Save WAV), `{{@STT:}}` speech-to-text dictation (engine selector: Whisper/Voxtral/Web Speech API, 11 languages, Record/Stop/Insert/Clear), `{{@Translate:}}` translation (target language selector, integrated TTS pronunciation, cloud model routing), `{{@Game:}}` game builder (AI-generated or pre-built, Canvas 2D/Three.js/P5.js, import/export HTML), `{{@Draw:}}` whiteboard (Excalidraw + Mermaid, AI diagram generation with per-card model selector + 🚀 Generate, robust JSON repair for local models, Insert/PNG/SVG export, 📚 Library Browser with 29 bundled packs in 6 categories), `{{@Tools:}}` web tools (Jina Reader scrape + Jina Search, multi-URL scraping, API key modal, Accept/Reject/Copy results) — `@` prefix syntax on all tag types + metadata fields (`@name`, `@use`, `@think`, `@search`, `@prompt`, `@step`, `@upload`, `@model`, `@engine`, `@lang`, `@prebuilt`); `@model:` field persists selected model per card with intelligent defaults (OCR→`granite-docling`, TTS→`kokoro-tts`, STT→`voxtral-stt`, Image→`imagen-ultra`); editable `@prompt:` textarea and `@step:` inputs in preview cards; description/prompt separation (bare text = label, `@prompt:` = AI instruction); 📎 image/PDF upload for multimodal vision analysis; per-card model selector with document-portable model persistence, concurrent block operations |
4040
| **🔌 API Calls** | `{{API:}}` REST API integration — GET/POST/PUT/DELETE methods, custom headers, JSON body, response stored in `$(api_varName)` variables; inline review panel; toolbar GET/POST buttons |
4141
| **🔗 Agent Flow** | `{{Agent:}}` multi-step pipeline — define Step 1/2/3, chain outputs, per-card model + search provider selector, live step status indicators (⏳/✅/❌), review combined output; `@cloud: yes/no` with ☁️ Cloud / 🖥️ Local badge; `@agenttype:` dropdown selector (openclaw/openfang) for external agents; Docker-based local execution via `agent-runner/server.js`; Agent Execution Settings UI (Codespaces/Local Docker/Custom endpoint); GitHub Codespaces cloud execution via ☁️ toggle; **📦 Agent Containers panel** — floating toolbar panel showing running Docker containers with live status, uptime, instant stop (`docker rm -f`), badge count, daemon readiness check, and startup container recovery |
4242
| **🔍 Web Search** | Toggle web search for AI — 7 providers: DuckDuckGo (free), Brave Search, Serper.dev, Tavily (AI-optimized), Google CSE, Wikipedia, Wikidata; search results injected into LLM context; source citations in responses; per-agent-card search provider selector |
@@ -538,6 +538,7 @@ TextAgent has undergone significant evolution since its inception. What started
538538

539539
| Date | Commits | Feature / Update |
540540
|------|---------|-----------------:|
541+
| **2026-03-23** | | 🛠️ **Web Tools Tag** — new `{{@Tools:}}` tag system for in-document web scraping and search; `@scrape:` mode fetches URL content as clean Markdown via [Jina Reader API](https://jina.ai/reader/) (`r.jina.ai`); `@search:` mode performs web search via [Jina Search API](https://jina.ai/search/) (`s.jina.ai`); multi-URL scraping with comma-separated URLs (auto-prepends `https://`); interactive preview card with Scrape/Search action pills, textarea input, ▶ Run button; 🔑 API key modal (localStorage) for higher rate limits (500 vs 20 RPM); Accept/Reject/Copy result flow; `tools-docgen.js` standalone module (Phase 3k); `tools-docgen.css` with cyan theme + dark mode; toolbar buttons (🔗 Scrape, 🔍 Search in Tools group); composer chips; CSP updated for Jina domains; textarea focus tracking prevents re-render loop during editing |
541542
| **2026-03-22** | | 🔗 **Compact Share Links** — share URLs reduced from ~111 chars to ~36 chars; new `#s=<shortId>` format with `generateShortId()` (epoch_base36 + 5 random alphanumeric chars) and `createCompactShare()` reusable function; encryption key stored server-side in Firestore (`k` field); collision-safe ID generation with retry; `loadSharedMarkdown()` handles both compact `#s=` and legacy `#id=...&k=...` formats; `cloudAutoSave()` updated for compact format; LLM Memory export uses same compact flow; Firestore rules updated to allow `k` field; secure (passphrase) shares remain zero-knowledge and unaffected |
542543
| **2026-03-21** | | 👁️ **Share Preview Default** — shared links now default to Preview mode; shared preview links auto-hide full header; Preview button switches to Split view mode |
543544
| **2026-03-21** | | 📦 **Agent Container Management Panel** — floating `📦 Agent Containers` panel in header toolbar showing running Docker agent containers with green status dot, uptime, and model info; `GET /api/agents/status` and `POST /api/agents/stop` endpoints on both `server.js` (port 8080) and Vite plugin (port 8877); live `docker ps` scan for cross-server container detection; instant stop via `docker rm -f`; daemon readiness wait (port 50051 polling) prevents OpenFang first-run failures; startup container scan recovers running containers after server restart; toolbar badge auto-refreshes every 15s; panel auto-refreshes every 5s while open; Stop All button |

‎css/tools-docgen.css‎

Lines changed: 312 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,312 @@
1+
/* ============================================
2+
tools-docgen.css — Web Tools (Scrape & Search) Card Styling
3+
============================================ */
4+
5+
/* --- Card Container --- */
6+
.tools-card {
7+
background: var(--bg-color, #fff);
8+
border: 1px solid rgba(6, 182, 212, 0.3);
9+
border-radius: 12px;
10+
margin: 1rem 0;
11+
padding: 0;
12+
overflow: hidden;
13+
box-shadow: 0 2px 8px rgba(6, 182, 212, 0.08);
14+
transition: box-shadow 0.2s, border-color 0.2s;
15+
}
16+
17+
.tools-card:hover {
18+
box-shadow: 0 4px 16px rgba(6, 182, 212, 0.15);
19+
border-color: rgba(6, 182, 212, 0.5);
20+
}
21+
22+
/* --- Header --- */
23+
.tools-header {
24+
display: flex;
25+
align-items: center;
26+
gap: 8px;
27+
padding: 10px 14px;
28+
background: linear-gradient(135deg, rgba(6, 182, 212, 0.08) 0%, rgba(6, 182, 212, 0.03) 100%);
29+
border-bottom: 1px solid rgba(6, 182, 212, 0.15);
30+
flex-wrap: wrap;
31+
}
32+
33+
.tools-icon {
34+
font-size: 1.1rem;
35+
}
36+
37+
.tools-label {
38+
font-weight: 600;
39+
font-size: 0.85rem;
40+
color: #06b6d4;
41+
letter-spacing: 0.02em;
42+
}
43+
44+
.tools-actions {
45+
margin-left: auto;
46+
display: flex;
47+
align-items: center;
48+
gap: 6px;
49+
flex-wrap: wrap;
50+
}
51+
52+
/* --- Buttons --- */
53+
.tools-btn {
54+
background: transparent;
55+
border: 1px solid rgba(6, 182, 212, 0.3);
56+
color: var(--text-color, #333);
57+
border-radius: 6px;
58+
padding: 4px 10px;
59+
font-size: 0.78rem;
60+
cursor: pointer;
61+
transition: all 0.15s;
62+
white-space: nowrap;
63+
}
64+
65+
.tools-btn:hover {
66+
background: rgba(6, 182, 212, 0.1);
67+
border-color: #06b6d4;
68+
}
69+
70+
.tools-btn:disabled {
71+
opacity: 0.5;
72+
cursor: not-allowed;
73+
}
74+
75+
.tools-run {
76+
background: #06b6d4;
77+
color: #fff;
78+
border-color: #06b6d4;
79+
font-weight: 600;
80+
}
81+
82+
.tools-run:hover {
83+
background: #0891b2;
84+
border-color: #0891b2;
85+
}
86+
87+
/* --- Action Pills (Scrape / Search toggle) --- */
88+
.tools-action-pills {
89+
display: flex;
90+
gap: 4px;
91+
padding: 8px 14px;
92+
flex-wrap: wrap;
93+
}
94+
95+
.tools-action-pill {
96+
background: transparent;
97+
border: 1px solid rgba(6, 182, 212, 0.2);
98+
border-radius: 20px;
99+
padding: 4px 12px;
100+
font-size: 0.78rem;
101+
cursor: pointer;
102+
transition: all 0.15s;
103+
color: var(--text-color, #555);
104+
}
105+
106+
.tools-action-pill:hover {
107+
background: rgba(6, 182, 212, 0.08);
108+
border-color: rgba(6, 182, 212, 0.4);
109+
}
110+
111+
.tools-action-pill.active {
112+
background: rgba(6, 182, 212, 0.15);
113+
border-color: #06b6d4;
114+
color: #06b6d4;
115+
font-weight: 600;
116+
}
117+
118+
/* --- Key Hint --- */
119+
.tools-key-hint {
120+
padding: 6px 14px;
121+
font-size: 0.75rem;
122+
color: var(--text-color-secondary, #888);
123+
}
124+
125+
.tools-key-hint a {
126+
color: #06b6d4;
127+
cursor: pointer;
128+
text-decoration: underline;
129+
}
130+
131+
/* --- Input Area --- */
132+
.tools-input-area {
133+
padding: 0 14px 10px;
134+
}
135+
136+
.tools-input {
137+
width: 100%;
138+
border: 1px solid rgba(6, 182, 212, 0.2);
139+
border-radius: 8px;
140+
padding: 8px 10px;
141+
font-size: 0.82rem;
142+
font-family: 'Fira Code', 'Consolas', monospace;
143+
background: var(--bg-color, #fff);
144+
color: var(--text-color, #333);
145+
resize: vertical;
146+
min-height: 36px;
147+
outline: none;
148+
transition: border-color 0.15s;
149+
box-sizing: border-box;
150+
}
151+
152+
.tools-input:focus {
153+
border-color: #06b6d4;
154+
box-shadow: 0 0 0 2px rgba(6, 182, 212, 0.12);
155+
}
156+
157+
/* --- Result Area --- */
158+
.tools-result {
159+
border-top: 1px solid rgba(6, 182, 212, 0.15);
160+
margin: 0;
161+
}
162+
163+
.tools-result-content {
164+
padding: 14px;
165+
font-size: 0.85rem;
166+
max-height: 400px;
167+
overflow: auto;
168+
line-height: 1.6;
169+
}
170+
171+
.tools-result-content pre {
172+
background: var(--code-bg, #f6f6f6);
173+
padding: 10px;
174+
border-radius: 6px;
175+
overflow-x: auto;
176+
font-size: 0.8rem;
177+
}
178+
179+
.tools-result-actions {
180+
display: flex;
181+
gap: 6px;
182+
padding: 8px 14px;
183+
border-top: 1px solid rgba(6, 182, 212, 0.1);
184+
justify-content: flex-end;
185+
}
186+
187+
/* --- Loading State --- */
188+
.tools-card.tools-loading .tools-label::after {
189+
content: '';
190+
display: inline-block;
191+
width: 12px;
192+
height: 12px;
193+
border: 2px solid rgba(6, 182, 212, 0.3);
194+
border-top-color: #06b6d4;
195+
border-radius: 50%;
196+
animation: tools-spin 0.8s linear infinite;
197+
margin-left: 8px;
198+
vertical-align: middle;
199+
}
200+
201+
@keyframes tools-spin {
202+
to { transform: rotate(360deg); }
203+
}
204+
205+
/* --- API Key Modal --- */
206+
.tools-key-overlay {
207+
position: fixed;
208+
inset: 0;
209+
background: rgba(0,0,0,0.5);
210+
display: flex;
211+
align-items: center;
212+
justify-content: center;
213+
z-index: 10000;
214+
opacity: 0;
215+
transition: opacity 0.2s;
216+
}
217+
218+
.tools-key-overlay.active {
219+
opacity: 1;
220+
}
221+
222+
.tools-key-dialog {
223+
background: var(--bg-color, #fff);
224+
border-radius: 12px;
225+
padding: 24px;
226+
width: 90%;
227+
max-width: 440px;
228+
box-shadow: 0 20px 60px rgba(0,0,0,0.3);
229+
}
230+
231+
.tools-key-header {
232+
display: flex;
233+
justify-content: space-between;
234+
align-items: center;
235+
margin-bottom: 12px;
236+
font-weight: 600;
237+
font-size: 1rem;
238+
}
239+
240+
.tools-key-close {
241+
background: transparent;
242+
border: none;
243+
font-size: 1.2rem;
244+
cursor: pointer;
245+
color: var(--text-color, #333);
246+
}
247+
248+
.tools-key-body p {
249+
font-size: 0.85rem;
250+
margin-bottom: 10px;
251+
color: var(--text-color-secondary, #666);
252+
line-height: 1.5;
253+
}
254+
255+
.tools-key-input {
256+
width: 100%;
257+
padding: 10px 12px;
258+
border: 1px solid rgba(6, 182, 212, 0.3);
259+
border-radius: 8px;
260+
font-size: 0.85rem;
261+
font-family: monospace;
262+
background: var(--bg-color, #fff);
263+
color: var(--text-color, #333);
264+
box-sizing: border-box;
265+
}
266+
267+
.tools-key-input:focus {
268+
outline: none;
269+
border-color: #06b6d4;
270+
}
271+
272+
.tools-key-actions {
273+
display: flex;
274+
gap: 8px;
275+
justify-content: flex-end;
276+
margin-top: 16px;
277+
}
278+
279+
/* --- Dark Mode --- */
280+
[data-theme="dark"] .tools-card,
281+
.dark .tools-card {
282+
background: #1a1a2e;
283+
border-color: rgba(6, 182, 212, 0.25);
284+
}
285+
286+
[data-theme="dark"] .tools-header,
287+
.dark .tools-header {
288+
background: linear-gradient(135deg, rgba(6, 182, 212, 0.12) 0%, rgba(6, 182, 212, 0.04) 100%);
289+
}
290+
291+
[data-theme="dark"] .tools-input,
292+
.dark .tools-input {
293+
background: #0d0d1a;
294+
border-color: rgba(6, 182, 212, 0.2);
295+
color: #e0e0e0;
296+
}
297+
298+
[data-theme="dark"] .tools-key-dialog,
299+
.dark .tools-key-dialog {
300+
background: #1a1a2e;
301+
}
302+
303+
[data-theme="dark"] .tools-key-input,
304+
.dark .tools-key-input {
305+
background: #0d0d1a;
306+
color: #e0e0e0;
307+
}
308+
309+
[data-theme="dark"] .tools-result-content pre,
310+
.dark .tools-result-content pre {
311+
background: #0d0d1a;
312+
}

0 commit comments

Comments
 (0)