KnowledgeHarvester queue needs explicit review/promote lifecycle before scheduled use

[ryan-baum]

Summary

KnowledgeHarvester.ts has the pieces of a queue-based curation workflow, but not the lifecycle semantics needed to make it safe for scheduled or repeated use.

Related issue #1171 correctly covers the primary bug: harvested candidates can land directly in canonical KNOWLEDGE/<domain>/. This issue is narrower: even when _harvest-queue is used, the queue consumer currently mutates on read and immediately promotes candidates through the normal harvest path.

Why this matters

KNOWLEDGE/ is the curated long-term memory surface. A periodic learning rhythm needs a safe proposed-vs-canonical boundary:

1. collect candidates
2. review candidates
3. explicitly promote, reject, or edit them
4. regenerate indexes

The current queue behavior collapses steps 2 and 3. If someone wires KnowledgeHarvester into a weekly cron, launchd job, or session-end hook, queue candidates can be silently promoted and deleted from the queue without review.

Evidence

In PAI/TOOLS/KnowledgeHarvester.ts:

• HARVEST_QUEUE_DIR points to KNOWLEDGE/_harvest-queue.
• scanHarvestQueue reads queue JSON files, converts them into candidates, then deletes the queue file with fs.unlinkSync.
• the normal harvest command appends those queue candidates into allCandidates.
• writeNote writes candidates directly to KNOWLEDGE/<domain>/<slug>.md.

So _harvest-queue currently behaves as a destructive ingest source, not a durable review queue.

There is an adjacent input-quality issue: scanResearch walks Markdown files primarily by length, so scaffolding files such as README.md can become canonical notes like Research/readme.md unless filename/frontmatter filters are added before queueing or promoting.

Suggested lifecycle

Add explicit verbs and make mutation opt-in:

• harvest --enqueue or default harvest writes candidates to _harvest-queue.
• review lists queued candidates without mutation.
• promote <slug> writes one reviewed item into KNOWLEDGE/<domain>/.
• reject <slug> archives or deletes one reviewed item.
• promote-all --domain Ideas exists only as an explicit batch operation.
• default harvest does not consume _harvest-queue unless passed an explicit flag such as --from-queue or --promote.

Acceptance criteria

• Running harvest cannot silently promote or delete queued candidates.
• Running review is read-only.
• Promotion preserves harvested_from provenance.
• Rejection leaves either an audit trail or an explicit deletion record.
• Research scanning excludes obvious scaffolding files such as README.md, _index.md, and generated dashboard files.
• Documentation describes the intended learning cadence: candidate capture vs curated knowledge promotion.

Related

• KnowledgeHarvester writes directly to KNOWLEDGE/<domain>/ — bypasses _harvest-queue/ curation step described in README #1171: direct writes bypass _harvest-queue curation boundary.
• /knowledge dashboard UX: 6 compounding gaps make populated knowledge graph feel empty (Research hidden, task-titles, no summaries, wrong dates, no grouping, 0 edges) #1172: downstream dashboard quality gaps after harvest.
• Discussion PAI's Single Biggest Upgrade: Close the Broken Learning Loop #1022: broader broken learning loop / capture-readback gap.

This is intended as the operational half of #1171: once the queue exists, the review/promote lifecycle needs safe defaults before KnowledgeHarvester is suitable for automation.

[ryan-baum]

Evidence appendix

Adding exact anchors/provenance so the issue is independently checkable.

Source provenance

Upstream v5.0.0 release source file:

• Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts
• GitHub blob SHA: bc5e90551a0c3afaaca319f7183897dd4a268184
• Source: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts

Local verification from my stock v5.0.0-derived install:

• git hash-object PAI/TOOLS/KnowledgeHarvester.ts returns the same blob SHA: bc5e90551a0c3afaaca319f7183897dd4a268184
• Local stock tag is stock-import-v5.0.0 and git diff --stat stock-import-v5.0.0^{} -- PAI/TOOLS/KnowledgeHarvester.ts returns no diff.

So the upstream line anchors below correspond to the same file content I reproduced locally.

Queue behavior anchors

• _harvest-queue path is defined at line 36: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L36
• scanHarvestQueue starts at line 257 and reads queue JSON files: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L257-L264
• scanHarvestQueue deletes each queue file after reading it at line 274: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L274
• queued candidates are appended into the normal harvest candidate list at line 775: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L775
• writeNote writes to KNOWLEDGE/<domain>/ via targetDir = path.join(KNOWLEDGE_DIR, candidate.domain) at lines 415-417: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L415-L417
• normal harvest calls writeNote(candidate) and records the source path at lines 800-801: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L800-L801

Conclusion from those anchors: queue candidates are read, deleted from _harvest-queue, appended to normal candidates, and written directly into canonical KNOWLEDGE/<domain>/. That is destructive ingest, not a read-only review queue.

Research scaffold-filter anchor

• scanResearch starts at line 225: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L225
• its visible filters are Markdown extension and minimum content length at lines 234 and 238: https://github.com/danielmiessler/Personal_AI_Infrastructure/blob/main/Releases/v5.0.0/.claude/PAI/TOOLS/KnowledgeHarvester.ts#L234-L238

That supports the narrower input-quality claim: scaffold Markdown files such as README.md are not excluded by filename or frontmatter before becoming harvest candidates.

No known PR fix from exact search

On 2026-06-15, GitHub public issue/PR search for:

repo:danielmiessler/Personal_AI_Infrastructure KnowledgeHarvester is:pr

returned total_count: 0 and incomplete_results: false.

API URL: https://api.github.com/search/issues?q=repo:danielmiessler/Personal_AI_Infrastructure%20KnowledgeHarvester%20is:pr

This does not prove no adjacent PR exists under a different term; it only supports the narrower statement that I found no exact KnowledgeHarvester pull request.