
Ebook Analysis
- 372 installs
- 135 repo stars
- Updated February 24, 2026
- jwynia/agent-skills
ebook-analysis is an agent skill that extracts themes, arguments, chapter summaries, and quotable insights with full citation traceability for developers who need structured knowledge from ebooks or long PDFs.
About
ebook-analysis in jwynia/agent-skills performs non-fiction knowledge extraction from ebooks and long PDFs with mandatory citation traceability to exact sources. The skill supports two complementary extraction modes: Concept Extraction classifies ideas from principle down to tactic, and Entity Extraction captures named studies, researchers, frameworks, and anecdotes that persist across books. The core rule is to extract less with full provenance rather than more without it, making outputs suitable for reviews, study guides, and derivative content planning. Developers reach for ebook-analysis when summarizing technical books, building reading notes with quotable citations, or comparing frameworks across multiple titles. Agents produce chapter-level summaries, argument maps, and insight lists tied to page or section references. The workflow favors traceable extractions over bulk paraphrase, which keeps downstream publishing and documentation auditable.
- Chapter-level summarization
- Theme and thesis extraction
- Argument mapping
- Citation-friendly quotes
- Comparative motif tracking
Ebook Analysis by the numbers
- 372 all-time installs (skills.sh)
- +3 installs in the week ending Aug 2, 2026 (Skillselion tracking)
- Ranked #175 of 688 Office & Documents skills by installs in the Skillselion catalog
- Data as of Aug 6, 2026 (Skillselion catalog sync)
npx skills add https://github.com/jwynia/agent-skills --skill ebook-analysisAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 372 |
|---|---|
| repo stars | ★ 135 |
| Last updated | February 24, 2026 |
| Repository | jwynia/agent-skills ↗ |
How do you extract themes from ebook PDFs?
Extract themes, arguments, chapter summaries, and quotable insights from ebooks or long PDFs to inform reviews, study guides, or derivative content planning.
Who is it for?
Developers and technical writers who need citation-traceable summaries from non-fiction ebooks or long PDFs before reviews or study guides.
Skip if: Users who only need OCR text dumps without thematic analysis or projects requiring extraction without source citations.
When should I use this skill?
User uploads or references an ebook or long PDF and asks for themes, chapter summaries, arguments, quotable insights, or structured book analysis.
What you get
Citation-linked concept extractions, entity lists, chapter summaries, and quotable insight notes with source provenance.
- Chapter summaries with citations
- Concept and entity extraction notes
By the numbers
- Supports 2 extraction modes: Concept Extraction and Entity Extraction
- Listed at 357 installs on skills.sh
Files
Ebook Analysis: Non-Fiction Knowledge Extraction
You analyze ebooks to extract knowledge with full citation traceability. This skill supports two complementary extraction modes:
1. Concept Extraction - Extract ideas classified by abstraction (principle → tactic) 2. Entity Extraction - Extract named things (studies, researchers, frameworks, anecdotes) that persist across books
Core Principle
Every extraction must be traceable to its exact source. Citation traceability is non-negotiable. Extract less with full provenance rather than more without it.
---
Two Extraction Modes
Mode 1: Concept Extraction
For extracting IDEAS organized by abstraction level.
Use when: Analyzing a book for transferable ideas, building a concept taxonomy, understanding how abstract principles relate to concrete tactics.
Output: JSON files (analysis.json, concepts.json)
Example: "Spaced repetition improves retention" is a MECHANISM at Layer 2.
Mode 2: Entity Extraction
For extracting NAMED THINGS that can be cross-referenced across books.
Use when: Building a knowledge base where the same study, researcher, or framework appears in multiple books. The goal is entity resolution—recognizing that "Hogarth's framework" in Range is the same as "kind/wicked environments" mentioned elsewhere.
Output: Markdown files in knowledge base structure
Example: "Kind vs Wicked Environments" is a FRAMEWORK by Robin Hogarth.
Choosing a Mode
| If you want to... | Use Mode |
|---|---|
| Understand a book's argument structure | Concept Extraction |
| Build a reference library across books | Entity Extraction |
| Create actionable takeaways | Concept Extraction |
| Track what researchers say across sources | Entity Extraction |
| Both | Run both modes sequentially |
---
Entity Extraction Mode (Detailed)
Entity Types
| Type | What It Captures | Example |
|---|---|---|
| study | Research findings, experiments, data | Flynn Effect, Marshmallow Test |
| researcher | People and their contributions | Anders Ericsson, Robin Hogarth |
| framework | Mental models, taxonomies, systems | Kind vs Wicked, Desirable Difficulties |
| anecdote | Stories used to illustrate points | Tiger vs Roger, Challenger Disaster |
| concept | Ideas that aren't frameworks | Cognitive entrenchment, Match quality |
Extended Entity Type Guidance
Some entities don't fit cleanly into the five types. Guidelines:
| Entity Kind | Use Type | Rationale |
|---|---|---|
| Simulations/Games (Superstruct, EVOKE) | anecdote | Illustrative events, even if hypothetical |
| Institutions (IFTF, WEF) | researcher | Organizations contribute ideas like individuals |
| Historical events (Challenger disaster) | anecdote | Stories that illustrate principles |
| Hypothetical scenarios | anecdote | Future scenarios from books like Imaginable |
| Thought experiments | framework | If systematic; otherwise concept |
When uncertain: Default to anecdote for narratives/events, concept for ideas, framework for systematic methods.
Author-as-Subject Pattern
When the book's author is also a significant entity (e.g., Jane McGonigal in Imaginable):
Create a researcher entity if:
- Author has notable prior work or institutional affiliation
- Author appears in Wikipedia or other reference sources
- Author's background/credentials are relevant to understanding the book
- Other books in your collection might reference them
Skip if:
- Author is primarily known only for this book
- No external sources to verify/enrich the entity
Template addition for author-subjects:
## Note
This researcher is the author of [Book] in our collection. Their frameworks and concepts are documented separately.Entity File Template
# [Entity Name]
**Type:** study | researcher | framework | anecdote | concept
**Status:** stub | partial | solid | authoritative
**Last Updated:** YYYY-MM-DD
**Aliases:** alias1, alias2, alias3
## Summary
[2-3 sentence synthesized understanding]
## Key Findings / What It Illustrates
1. [Claim or finding with source]
— Source: [Book], Ch.[X]
2. [Another claim]
— Source: [Book], Ch.[X]
## Key Quotes
> "Quotable text here."
> "Another memorable quote."
## Sources in Collection
| Book | Author | How It's Used | Citation |
|------|--------|---------------|----------|
| Range | Epstein | [Role in book] | Ch.X |
## Sources NOT in Collection
- [Book that would enrich this entity]
## Related Entities
- [Other Entity](../type/other-entity.md) - Relationship description
## Open Questions
- [What we don't yet know]Knowledge Base Structure
/knowledge/
├── _index.md # Master registry
├── _entities.json # Searchable index (generated)
│
├── nonfiction/
│ ├── _index.md # Domain index
│ ├── _[book]-quotes.md # Book-specific quotes file
│ ├── studies/
│ │ ├── flynn-effect.md
│ │ └── chase-simon-chunking.md
│ ├── researchers/
│ │ ├── hogarth-robin.md
│ │ └── tetlock-philip.md
│ ├── frameworks/
│ │ ├── kind-vs-wicked-environments.md
│ │ └── desirable-difficulties.md
│ ├── anecdotes/
│ │ ├── tiger-vs-roger.md
│ │ └── challenger-disaster.md
│ └── concepts/
│ ├── cognitive-entrenchment.md
│ └── match-quality.md
│
├── cooking/ # Domain-specific structure
│ ├── techniques/
│ ├── ingredients/
│ └── equipment/
│
└── technical/
├── patterns/
└── technologies/Quotes Extraction
Quotable quotes are a distinct extraction type. For each book, create a quotes file:
File: _[book-slug]-quotes.md
Structure:
# Quotable Quotes from [Book Title]
**Author:** [Author]
**Last Updated:** YYYY-MM-DD
## On [Theme 1]
> "Quote text here."
> "Another quote on same theme."
## On [Theme 2]
> "Quote on different theme."What makes a good quote:
- Memorable phrasing that captures a key insight
- Self-contained (understandable without context)
- Surprising or counterintuitive formulation
- Useful for presentations, writing, or reference
Entity Extraction Workflow
1. Scan book - Read through identifying named studies, researchers, frameworks, illustrative stories 2. Check existing entities - Use kb-resolve-entity.ts to see if entity already exists 3. Create or update - New entity → create file; existing → add as source 4. Add quotes - Extract memorable quotes to quotes file 5. Cross-link - Add Related Entities sections 6. Regenerate index - Run kb-generate-index.ts
Entity Extraction States (KB0-KB5)
| State | Symptoms | Intervention |
|---|---|---|
| KB0 | No knowledge base | Create directory structure |
| KB1 | Structure exists, no entities | Begin extraction |
| KB2 | Extracting from book | Create entity files |
| KB3 | Entities created, not linked | Add Related Entities |
| KB4 | Linked, no index | Run kb-generate-index.ts |
| KB5 | Complete for this book | Proceed to next book |
Cross-Book Synthesis Workflow
Triggered when: 2+ books have been extracted to the knowledge base.
Goals: 1. Find entities that appear in multiple books 2. Identify conceptual connections between books 3. Surface contradictions or complementary perspectives 4. Update entity files with multi-source synthesis
Process:
1. Entity overlap detection
# Find entities with 2+ sources
grep -l "Sources in Collection" knowledge/nonfiction/**/*.md | \
xargs grep -l "| .* | .* |" | head -20Or manually review entities updated with new source.
2. Conceptual connection mapping
- Compare frameworks across books (e.g., Range's "wicked environments" ↔ Imaginable's "futures thinking")
- Identify shared researchers (e.g., Tetlock appears in both Range and Imaginable)
- Look for complementary themes (prediction failure → preparation despite uncertainty)
3. Synthesis documentation For entities appearing in 2+ books, update the Summary section:
## Summary
[Synthesized understanding from BOTH sources, noting agreements and differences]4. Cross-book insights Document thematic connections in context/insights/cross-book-{theme}.md:
# Cross-Book Insight: [Theme]
## Books Contributing
- Range (Epstein) - [perspective]
- Imaginable (McGonigal) - [perspective]
## Synthesis
[How the books complement or contradict each other]---
Concept Extraction Mode (Detailed)
Concept Types (Abstract → Concrete)
| Type | Definition | Example |
|---|---|---|
| Principle | Foundational truth or axiom | "Communities form around shared identity" |
| Mechanism | How something works | "Reciprocity creates social bonds" |
| Pattern | Recurring structure or approach | "The community lifecycle pattern" |
| Strategy | High-level approach to achieve goals | "Build trust before asking for contribution" |
| Tactic | Specific actionable technique | "Send welcome emails within 24 hours" |
Abstraction Layers
| Layer | Name | Abstraction | Example |
|---|---|---|---|
| 0 | Foundational | Universal principles | "Humans seek belonging" |
| 1 | Theoretical | Domain-specific theory | "Community requires shared purpose" |
| 2 | Strategic | Approaches and frameworks | "The funnel model of engagement" |
| 3 | Tactical | Specific methods | "Onboarding sequences" |
| 4 | Specific | Concrete implementations | "Use Discourse for forums" |
Relationship Types
| Relationship | Meaning | When to Use |
|---|---|---|
| INFLUENCES | A affects B | Causal or correlational connection |
| SUPPORTS | A provides evidence for B | Citation, example, validation |
| CONTRADICTS | A conflicts with B | Opposing claims |
| COMPOSED_OF | A contains B | Part-whole relationships |
| DERIVES_FROM | A is derived from B | Logical conclusions |
Concept Extraction States (EA0-EA7)
| State | Symptoms | Intervention |
|---|---|---|
| EA0 | No input file | Guide file preparation |
| EA1 | Raw file, not parsed | Run ea-parse.ts |
| EA2 | Parsed, not extracted | LLM extracts concepts |
| EA3 | Extracted, not classified | Assign types and layers |
| EA4 | Classified, not annotated | Add themes, relationships |
| EA5 | Single book complete | Export or proceed to synthesis |
| EA6 | Multi-book ready | Cross-book synthesis |
| EA7 | Analysis complete | Generate reports |
Concept Extraction Workflow
1. Parse - Run ea-parse.ts to chunk book with position tracking 2. Extract - Present chunks to LLM for concept identification with exact quotes 3. Classify - Assign type (principle→tactic) and layer (0-4) 4. Annotate - Add themes and functional analysis 5. Link - Connect related concepts 6. Export - Generate analysis.json, concepts.json, report.md
---
Available Tools
Parsing Tools
ea-parse.ts
Parse ebook files into chunks with metadata and position tracking.
deno run --allow-read scripts/ea-parse.ts path/to/book.txt
deno run --allow-read scripts/ea-parse.ts path/to/book.epub --format epub
deno run --allow-read scripts/ea-parse.ts book.txt --chunk-size 1500 --overlap 150Output: JSON with metadata, chapters (if detected), and chunks with positions.
Knowledge Base Tools
kb-generate-index.ts
Scan knowledge base and generate searchable entity index.
deno run --allow-read --allow-write scripts/kb-generate-index.ts /path/to/knowledgeOutput: Creates _entities.json with all entities, aliases, and metadata.
kb-resolve-entity.ts
Search for existing entities before creating duplicates.
deno run --allow-read scripts/kb-resolve-entity.ts "Flynn Effect"
deno run --allow-read scripts/kb-resolve-entity.ts "Hogarth" --threshold 0.5
deno run --allow-read scripts/kb-resolve-entity.ts "kind learning" --jsonOptions:
--threshold <0-1>- Minimum match score (default: 0.3)--limit <n>- Maximum results (default: 5)--json- Output as JSON
Validation Tools
ea-validate.ts
Validate analysis output for citation accuracy and schema completeness.
deno run --allow-read scripts/ea-validate.ts analysis.json --report---
Anti-Patterns
The Extraction Flood
Pattern: Extracting every potentially interesting phrase. Fix: Ask "Would I cite this?" before extracting. Quality over quantity.
The Citation Black Hole
Pattern: Extracting without preserving exact quotes or positions. Fix: Always capture: exact quote, chapter reference, context.
The Duplicate Entity
Pattern: Creating new entity without checking if it exists. Fix: Always run kb-resolve-entity.ts first.
The Orphan Entity
Pattern: Entities without Related Entities links. Fix: Every entity should connect to at least 2 others.
The Quote-Free Entity
Pattern: Entity captures ideas but no memorable phrasing. Fix: Include Key Quotes section with author's exact words.
The Single-Book Silo
Pattern: Analyzing books without cross-referencing. Fix: After 2+ books, run synthesis to find connections.
---
Example Workflows
Full Entity Extraction (Range Example)
1. Scan book chapter by chapter
2. Identify all named studies, researchers, frameworks, anecdotes
3. Create inventory document listing all potential entities
4. For each entity:
a. kb-resolve-entity.ts "[entity name]" to check existence
b. Create markdown file in appropriate type directory
c. Fill in template with findings and citations
d. Add Key Quotes section
5. Create _range-quotes.md with all memorable quotes
6. Update _index.md with new entities
7. kb-generate-index.ts to rebuild _entities.jsonQuick Concept Scan
1. ea-parse.ts book.txt --chunk-size 2000
2. For each chunk, extract top 3-5 concepts
3. Classify by type and layer
4. Generate concepts.json and report.md---
Output Persistence
Entity Extraction Output
| File | Location |
|---|---|
| Entity files | knowledge/{domain}/{type}/{entity-slug}.md |
| Quotes file | knowledge/{domain}/_[book]-quotes.md |
| Entity index | knowledge/_entities.json |
| Domain index | knowledge/{domain}/_index.md |
Concept Extraction Output
| File | Location |
|---|---|
| Full analysis | ebook-analysis/{author}-{title}/analysis.json |
| Concepts only | ebook-analysis/{author}-{title}/concepts.json |
| Citations | ebook-analysis/{author}-{title}/citations.json |
| Report | ebook-analysis/{author}-{title}/report.md |
---
Verification (Oracle)
What This Skill Can Verify
- Citation positions exist - Validate quoted text appears at claimed position
- Schema completeness - Required fields present
- Cross-reference integrity - Referenced entities exist
- Duplicate detection - Entity doesn't already exist (via kb-resolve-entity.ts)
What Requires Human Judgment
- Significance - Is this worth extracting?
- Classification - Is this really a "framework" vs "concept"?
- Relationship validity - Does A really influence B?
- Quote quality - Is this actually memorable?
---
Integration Graph
Inbound (From Other Skills)
| Source | Leads to |
|---|---|
| research | Multi-book synthesis ready |
| reverse-outliner | Structural data for concept extraction |
Outbound (To Other Skills)
| From State | Leads to |
|---|---|
| Entity extraction complete | dna-extraction (deep functional analysis) |
| Concept extraction complete | media-meta-analysis (cross-source synthesis) |
Complementary Skills
| Skill | Relationship |
|---|---|
| dna-extraction | 6-axis functional analysis for annotation |
| reverse-outliner | Structural approach for fiction |
| voice-analysis | Author style fingerprinting |
| context-network | Knowledge base maintenance |
---
Calibration Data (from Range + Imaginable extractions)
By Book Density
| Book Type | Expected Entities | Estimated Effort |
|---|---|---|
| Dense non-fiction (Range, Thinking Fast & Slow) | 60-100 | 4-6 hours |
| Moderate non-fiction (most business books) | 30-50 | 2-3 hours |
| Light non-fiction (popular science) | 15-30 | 1-2 hours |
| Technical books | 20-40 | 2-3 hours |
By Book Subtype
Different non-fiction subtypes yield different entity profiles:
| Subtype | Example | Entity Profile | Expected Count |
|---|---|---|---|
| Research synthesis | Range | Many studies, researchers, frameworks | 60-100 |
| Methodological/How-to | Imaginable | Many frameworks, few studies | 30-50 |
| Memoir/Narrative | Educated | Few frameworks, many anecdotes | 20-40 |
| Reference | Technical manuals | Many concepts, few anecdotes | Variable |
Research synthesis books cite many studies and researchers, connecting ideas across domains. Methodological books teach techniques and frameworks but cite fewer external sources. Memoir/narrative books use personal stories to illustrate points rather than research.
Metadata Reliability Warning
Book classification metadata (Calibre tags, library categories) is often:
- Wrong - Fiction/non-fiction misclassified
- Generic - "General Fiction" or "Self-Help" applied broadly
- Inconsistent - Same book categorized differently across sources
Always verify classification makes sense before extraction. A "fiction" tag on a methodology book like Imaginable is a metadata error.
---
Reasoning Requirements
Standard Reasoning
- Single chunk concept extraction
- Type/layer classification
- Simple relationship identification
- Individual entity creation
Extended Reasoning (ultrathink)
Use extended thinking for:
- Multi-book synthesis - requires holding multiple networks simultaneously
- Contradiction detection - semantic comparison across sources
- Theme emergence - identifying patterns across large sets
- Knowledge gap identification - reasoning about what's missing
Trigger phrases: "synthesize across books", "find contradictions", "identify gaps", "comprehensive analysis"
---
What You Do NOT Do
- Extract without citation traceability
- Create entities without checking for duplicates
- Skip the linking phase (orphan entities are not useful)
- Leave entities without quotes
- Treat fiction as non-fiction
- Use regex for semantic analysis (LLM judgment only)
{
"_meta": {
"description": "Rules for mapping Calibre tags to classification categories",
"version": "1.0",
"last_updated": "2026-01-16",
"categories": ["fiction", "cookbooks", "technical", "business", "self_help", "other_nonfiction"]
},
"priority_order": ["fiction", "cookbooks", "technical", "business", "self_help", "other_nonfiction"],
"excluded_tags": ["General", "-", "_NB_fixed", "_rt_yes"],
"category_rules": {
"fiction": {
"exact_matches": [
"Fiction",
"Science Fiction",
"Fantasy",
"Thrillers",
"Mystery & Detective",
"Suspense",
"Crime",
"Horror",
"Romance",
"Action & Adventure",
"Cozy",
"Women Sleuths",
"Police Procedural",
"Amateur Sleuth",
"Space Opera",
"Hard Science Fiction",
"Dark Fantasy",
"Historical",
"Contemporary",
"Psychological",
"Juvenile Fiction",
"Collections & Anthologies",
"Short Stories (Single Author)",
"Anthologies (Multiple Authors)",
"Classics",
"Literary Collections",
"Essays",
"literary",
"short stories",
"Epic",
"Urban",
"Paranormal",
"Dystopian",
"Cyberpunk",
"Steampunk",
"Hard-Boiled",
"Noir",
"Westerns",
"Young Adult Fiction",
"Comics & Graphic Novels",
"Magical Realism",
"Alternate History",
"Military",
"Espionage",
"Legal",
"Humorous",
"Satire",
"Gothic",
"Apocalyptic & Post-Apocalyptic"
],
"pattern_matches": [
".*Fiction$",
".*Stories$",
".*Sleuth.*",
".*Detective.*",
".*Mystery.*",
".*Thriller.*"
]
},
"cookbooks": {
"exact_matches": [
"Cooking",
"cookbook",
"Culinary",
"Beverages",
"Alcoholic",
"Beer",
"Wine",
"Specific Ingredients",
"Courses & Dishes",
"Regional & Ethnic",
"Diet & Nutrition",
"Diets",
"Quick & Easy",
"Special Appliances",
"Baking",
"Recipes",
"Bread",
"Breakfast",
"Cakes",
"Canning & Preserving",
"Comfort Food",
"Desserts",
"Food Science",
"Gluten-Free",
"Gourmet",
"Herbs",
"Meat",
"Pasta",
"Pizza",
"Sandwiches",
"Sauces",
"Slow Cooking",
"Soups & Stews",
"Vegan",
"Vegetables",
"Vegetarian",
"Bartending & Cocktails",
"Canning",
"Grilling"
],
"pattern_matches": [
".*Cook.*",
".*Recipe.*",
".*Baking.*",
".*Food.*"
]
},
"technical": {
"exact_matches": [
"Computers",
"Software Development & Engineering",
"Programming",
"Internet",
"Technology & Engineering",
"Languages",
"Web Programming",
"Web Design",
"Computer Graphics",
"Database",
"Data Science",
"Machine Learning",
"Artificial Intelligence",
"Algorithms",
"Computer Architecture",
"Networking",
"Linux",
"Python",
"Java",
"JavaScript",
"Information Technology",
"Robotics",
"Automation",
"Electronics",
"Engineering",
"Operating Systems",
"Security",
"Software",
"Hardware",
"Programming Languages",
"Object-Oriented Programming",
"Functional Programming",
"DevOps",
"Cloud Computing",
"APIs"
],
"pattern_matches": [
".*Programming.*",
".*Software.*",
".*Computer.*",
".*Developer.*",
".*Code.*"
]
},
"business": {
"exact_matches": [
"Business & Economics",
"Management",
"Leadership",
"Entrepreneurship",
"Marketing",
"Sales",
"Finance",
"Investing",
"Strategic Planning",
"Business Communication",
"Human Resources",
"E-Commerce",
"Organizational Behavior",
"Economics",
"Personal Finance",
"Small Business",
"Project Management",
"Business Development",
"Advertising & Promotion",
"Accounting",
"Banking",
"Careers",
"Commerce",
"Corporate Finance",
"International",
"Money & Monetary Policy",
"Real Estate",
"Workplace Culture"
],
"pattern_matches": [
".*Business.*",
".*Economics.*",
".*Finance.*",
".*Management.*"
]
},
"self_help": {
"exact_matches": [
"Self-Help",
"Personal Growth",
"Happiness",
"Motivational & Inspirational",
"Motivational",
"Personal Success",
"Success",
"Psychology",
"Social Psychology",
"Health & Healing",
"Healthy Living & Personal Hygiene",
"Body; Mind & Spirit",
"Creativity",
"Family & Relationships",
"Self-Esteem",
"Stress Management",
"Time Management",
"Mindfulness",
"Meditation",
"Spiritual",
"Inspiration",
"Communication",
"Relationships",
"Mental Health",
"Emotional Intelligence",
"Habits",
"Productivity",
"Positive Psychology",
"Life Coaching",
"Personal Transformation"
],
"pattern_matches": [
".*Self-Help.*",
".*Personal.*Growth.*",
".*Motivational.*",
".*Mindfulness.*"
]
},
"other_nonfiction": {
"exact_matches": [
"History",
"Science",
"Reference",
"Biography & Autobiography",
"Personal Memoirs",
"Philosophy",
"Religion",
"Social Science",
"Political Science",
"Crafts & Hobbies",
"Art",
"House & Home",
"Techniques",
"Woodwork",
"Do-It-Yourself",
"Education",
"Medical",
"Health & Fitness",
"Nutrition",
"Life Sciences",
"Performing Arts",
"music",
"Language Arts & Disciplines",
"Authorship",
"Writing Skills",
"Fiction Writing",
"writing",
"Literary Criticism",
"American",
"Women",
"United States",
"Nature",
"Travel",
"Sports & Recreation",
"Games",
"Gardening",
"Pets",
"Architecture",
"Photography",
"Design",
"Mathematics",
"Physics",
"Chemistry",
"Biology",
"Astronomy",
"Anthropology",
"Sociology",
"Law",
"True Crime",
"Journalism",
"Media Studies"
],
"pattern_matches": []
}
},
"confidence_rules": {
"single_definitive_tag": 1.0,
"multiple_same_category": 0.95,
"primary_clear_secondary_different": 0.85,
"ambiguous_resolved_by_priority": 0.75,
"very_ambiguous": 0.60
}
}
{
"_meta": {
"description": "Seed vocabulary for thematic annotation. LLM should extend beyond this list when appropriate.",
"usage": "Use as starting points for theme identification, not as constraints",
"note": "These are common non-fiction themes. Domain-specific themes should be added during analysis."
},
"human_nature": {
"description": "Themes about fundamental human characteristics",
"terms": [
"belonging",
"identity",
"motivation",
"cooperation",
"competition",
"trust",
"fear",
"hope",
"meaning",
"purpose",
"autonomy",
"connection",
"growth",
"security",
"status"
]
},
"social_dynamics": {
"description": "Themes about how groups and societies function",
"terms": [
"community",
"leadership",
"hierarchy",
"power",
"influence",
"norms",
"culture",
"ritual",
"inclusion",
"exclusion",
"conflict",
"consensus",
"networks",
"reciprocity",
"reputation"
]
},
"knowledge_learning": {
"description": "Themes about acquiring and sharing knowledge",
"terms": [
"learning",
"expertise",
"skill",
"mastery",
"practice",
"feedback",
"transfer",
"memory",
"understanding",
"wisdom",
"education",
"mentorship",
"discovery",
"curiosity",
"innovation"
]
},
"systems_change": {
"description": "Themes about systems and how they evolve",
"terms": [
"systems",
"complexity",
"emergence",
"adaptation",
"evolution",
"transformation",
"disruption",
"stability",
"resilience",
"feedback_loops",
"tipping_points",
"equilibrium",
"growth",
"decay",
"renewal"
]
},
"work_productivity": {
"description": "Themes about work and getting things done",
"terms": [
"productivity",
"efficiency",
"effectiveness",
"focus",
"attention",
"prioritization",
"delegation",
"collaboration",
"execution",
"planning",
"measurement",
"improvement",
"automation",
"workflow",
"burnout"
]
},
"decision_making": {
"description": "Themes about choices and judgment",
"terms": [
"decision",
"judgment",
"bias",
"heuristics",
"risk",
"uncertainty",
"tradeoffs",
"options",
"constraints",
"criteria",
"intuition",
"analysis",
"framing",
"commitment",
"reversibility"
]
},
"communication": {
"description": "Themes about conveying and receiving information",
"terms": [
"communication",
"persuasion",
"storytelling",
"clarity",
"listening",
"feedback",
"dialogue",
"rhetoric",
"framing",
"narrative",
"transparency",
"authenticity",
"empathy",
"misunderstanding",
"context"
]
},
"value_ethics": {
"description": "Themes about what matters and right action",
"terms": [
"values",
"ethics",
"integrity",
"fairness",
"justice",
"responsibility",
"accountability",
"virtue",
"principle",
"character",
"morality",
"duty",
"rights",
"harm",
"benefit"
]
},
"time_change": {
"description": "Themes about temporality and transformation",
"terms": [
"time",
"change",
"progress",
"regression",
"cycles",
"phases",
"stages",
"transitions",
"persistence",
"urgency",
"patience",
"timing",
"legacy",
"future",
"history"
]
},
"resources_constraints": {
"description": "Themes about limitations and allocation",
"terms": [
"scarcity",
"abundance",
"resources",
"constraints",
"allocation",
"investment",
"returns",
"sustainability",
"leverage",
"capacity",
"limits",
"opportunity_cost",
"tradeoffs",
"efficiency",
"waste"
]
}
}
Abstraction Layers (0-4)
This reference defines the five abstraction layers used to classify concepts from most abstract (Layer 0) to most concrete (Layer 4). Use these definitions to guide classification.
Overview
| Layer | Name | Scope | Example Domain: Community Building |
|---|---|---|---|
| 0 | Foundational | Universal human truths | "Humans seek belonging" |
| 1 | Theoretical | Domain-specific theory | "Communities require shared purpose" |
| 2 | Strategic | Approaches and frameworks | "The funnel model of engagement" |
| 3 | Tactical | Specific methods | "Onboarding sequences" |
| 4 | Specific | Concrete implementations | "Use Discourse for forums" |
---
Layer 0: Foundational
Definition: Universal principles that apply across domains and cultures. These are the bedrock truths about human nature, systems, or reality.
Characteristics:
- Domain-agnostic (applies everywhere)
- Timeless (true across eras)
- About fundamental nature
- Few in number per domain
Questions to Identify Layer 0:
- Would this be true in any culture or time period?
- Does this apply beyond the specific domain discussed?
- Is this about human nature or universal systems?
Examples:
- "Humans form groups for survival advantage"
- "Complex systems resist change"
- "Scarcity increases perceived value"
- "Trust enables cooperation"
Not Layer 0:
- Anything that requires specific context
- Concepts tied to modern technology or practices
- Domain-specific frameworks
---
Layer 1: Theoretical
Definition: Domain-specific theory that explains how the foundational principles manifest in a particular field. These are the "laws" of the domain.
Characteristics:
- Specific to a domain (communities, learning, leadership, etc.)
- Explanatory (tells why things work in this domain)
- Forms basis for strategies
- Multiple theories may exist
Questions to Identify Layer 1:
- Is this specific to one domain but general within it?
- Does this explain how foundational principles apply here?
- Would experts in this field recognize this as theoretical foundation?
Examples (Community Building):
- "Communities form around shared identity"
- "Engagement follows the commitment-consistency principle"
- "Network effects drive community value"
Examples (Learning):
- "Memory consolidates during sleep"
- "Active recall strengthens retention"
- "Spacing improves long-term learning"
Not Layer 1:
- Universal truths (those are Layer 0)
- Specific practices or methods (those are Layer 3-4)
- Frameworks with multiple components (those are often Layer 2)
---
Layer 2: Strategic
Definition: Approaches, frameworks, and models that translate theory into actionable direction. These provide structure for how to think about problems in the domain.
Characteristics:
- Has named components or phases
- Provides decision-making guidance
- Can be implemented multiple ways
- Often visualizable (diagrams, matrices)
Questions to Identify Layer 2:
- Is this a framework or model?
- Does it have multiple components or stages?
- Does it guide how to approach problems?
Examples (Community Building):
- "The community lifecycle: formation, growth, maturation, renewal"
- "The engagement ladder: lurker → participant → contributor → leader"
- "The three pillars of community: connection, content, collaboration"
Examples (Product Development):
- "Build-Measure-Learn cycle"
- "The MVP approach"
- "Jobs to be done framework"
Not Layer 2:
- Single techniques (those are Layer 3)
- Theoretical explanations (those are Layer 1)
- Tool recommendations (those are Layer 4)
---
Layer 3: Tactical
Definition: Specific methods, techniques, and practices that implement strategies. These are concrete enough to be directly actionable but not tied to specific tools.
Characteristics:
- Directly actionable
- Can be done without specific tools
- Has clear success criteria
- Reusable across contexts
Questions to Identify Layer 3:
- Can someone implement this directly?
- Is it independent of specific tools?
- Would this work with various implementations?
Examples (Community Building):
- "Create onboarding sequences for new members"
- "Host regular events at consistent times"
- "Recognize contributions publicly"
- "Pair new members with experienced mentors"
Examples (Writing):
- "Outline before drafting"
- "Read your work aloud for rhythm"
- "Start with the end in mind"
Not Layer 3:
- Tool-specific instructions (those are Layer 4)
- High-level approaches (those are Layer 2)
- Explanations without actions (those are Layer 1)
---
Layer 4: Specific
Definition: Concrete implementations tied to specific tools, platforms, time periods, or contexts. These are the most actionable but also the most likely to become outdated.
Characteristics:
- Names specific tools, platforms, or products
- May include specific numbers or thresholds
- Time-bound or context-bound
- Most perishable layer
Questions to Identify Layer 4:
- Does this name a specific tool or platform?
- Would this become outdated if technology changes?
- Is this tied to a particular time, place, or organization?
Examples (Community Building):
- "Use Discourse for forums"
- "Set up a #introductions channel in Slack"
- "Send the welcome email at 9am local time"
- "Aim for 30% response rate in the first week"
Examples (Software Development):
- "Use React for the frontend"
- "Deploy to AWS Lambda"
- "Set MAX_CONNECTIONS to 100"
Not Layer 4:
- Generic techniques without tool names (those are Layer 3)
- Frameworks or models (those are Layer 2)
---
Layer Assignment Guidelines
Default to Higher Layers
When uncertain, assign to the higher (more abstract) layer. Reasons:
- Abstract concepts are more reusable
- They're more likely to remain valid over time
- Specifics can always be derived from abstractions
Consider the Author's Intent
The same phrase can exist at different layers depending on context:
- "Use events to build community" (Layer 3 - tactic)
- "Events create shared experiences that reinforce identity" (Layer 1 - theory)
- "Host a monthly Zoom meetup on the first Thursday" (Layer 4 - specific)
Track Layer Distribution
A well-extracted book should have concepts across multiple layers:
- Few Layer 0 concepts (2-5 typically)
- More Layer 1-2 concepts (the book's theoretical contribution)
- Many Layer 3-4 concepts (practical advice)
If extraction is heavily weighted to one layer, review for missed concepts at other layers.
---
Cross-Reference with Concept Types
Certain concept types cluster at certain layers:
| Layer | Common Types |
|---|---|
| 0 | Principles |
| 1 | Principles, Mechanisms |
| 2 | Patterns, Strategies |
| 3 | Tactics, Strategies |
| 4 | Tactics |
This is guidance, not rule. A Layer 0 mechanism exists ("Reciprocity operates through obligation"), as does a Layer 4 pattern ("The onboarding flow: welcome email → profile setup → first action").
---
When Layer is Unclear
1. Look at surrounding context for abstraction level 2. Consider how reusable/portable the concept is 3. Note uncertainty in the concept record 4. Prefer higher layers when genuinely ambiguous 5. Flag for human review if the distinction matters for synthesis
Concept Types
This reference defines the five concept types used in ebook analysis. Use these definitions to guide classification - they are guidance for judgment, not rigid rules.
The Five Types
Principle
Definition: A foundational truth, axiom, or universal claim about how things work.
Distinguishing Features:
- Asserts something IS true (not how to do something)
- Often stated as timeless or universal
- Serves as foundation for other concepts
- Usually abstract and broadly applicable
Language Signals:
- "X is essential for Y"
- "The fundamental truth is..."
- "At its core, X requires Y"
- Declarative statements about nature of things
Examples:
- "Communities form around shared identity" (community building)
- "Trust is earned through consistent action" (leadership)
- "Complex systems exhibit emergent behavior" (systems thinking)
Not Principles:
- Step-by-step processes (those are tactics)
- Frameworks with multiple components (those are patterns)
- Recommendations for action (those are strategies)
---
Mechanism
Definition: An explanation of HOW something works - the causal chain or process by which effects occur.
Distinguishing Features:
- Explains causation (A leads to B because...)
- Describes internal workings
- Can be observed or tested
- Shows the "gears" behind outcomes
Language Signals:
- "X works by..."
- "The way this happens is..."
- "When A occurs, B follows because..."
- "The underlying process involves..."
Examples:
- "Reciprocity creates social bonds by triggering obligation responses" (social dynamics)
- "Spaced repetition enhances memory by strengthening neural pathways during recall" (learning)
- "Network effects increase value as more users join because each new connection adds multiple potential relationships" (technology)
Not Mechanisms:
- Simple observations without explanation (those may be principles)
- How-to instructions (those are tactics)
- Descriptions of what something is (those may be patterns)
---
Pattern
Definition: A recurring structure, framework, or recognizable arrangement that appears across multiple instances.
Distinguishing Features:
- Describes a shape or structure
- Can be observed in multiple contexts
- Has named components or phases
- Provides a mental model or framework
Language Signals:
- "The X pattern involves..."
- "This follows a common structure of..."
- "The framework consists of..."
- "There are N stages/phases/components..."
Examples:
- "The hero's journey pattern: departure, initiation, return" (storytelling)
- "The community lifecycle: founding, growth, maturation, renewal" (communities)
- "The five stages of grief: denial, anger, bargaining, depression, acceptance" (psychology)
Not Patterns:
- Single actions or techniques (those are tactics)
- Explanations of why things work (those are mechanisms)
- Recommendations without structure (those are strategies)
---
Strategy
Definition: A high-level approach or orientation for achieving goals - the "how to think about" rather than "how to do."
Distinguishing Features:
- Provides direction without specifying exact steps
- Can be implemented in multiple ways
- Involves trade-offs and priorities
- Guides decision-making
Language Signals:
- "The approach should be..."
- "Focus on X before Y"
- "Prioritize..."
- "The key is to..."
- "Rather than X, consider Y"
Examples:
- "Build trust before asking for contribution" (community building)
- "Fail fast and iterate" (product development)
- "Invest in relationships during good times" (leadership)
Not Strategies:
- Specific techniques (those are tactics)
- Frameworks with defined components (those are patterns)
- Explanations of why things work (those are mechanisms)
---
Tactic
Definition: A specific, actionable technique or practice that can be directly implemented.
Distinguishing Features:
- Concrete and specific
- Can be done immediately
- Has clear success criteria
- Often includes tools, templates, or scripts
Language Signals:
- "Do X"
- "Send/Create/Write/Build..."
- "Use the following template..."
- "Every day, do..."
- Imperative verbs
Examples:
- "Send welcome emails within 24 hours of signup" (community building)
- "Use the STAR format for behavioral interviews" (hiring)
- "Write three things you're grateful for each morning" (wellbeing)
Not Tactics:
- General approaches (those are strategies)
- Explanations of why (those are mechanisms)
- Broad truths (those are principles)
---
Classification Decision Tree
When classifying a concept, ask:
1. Is it an actionable instruction?
- Yes → Probably TACTIC
- No → Continue
2. Does it explain HOW something causes an effect?
- Yes → Probably MECHANISM
- No → Continue
3. Does it describe a recurring structure or framework?
- Yes → Probably PATTERN
- No → Continue
4. Does it provide high-level guidance for achieving goals?
- Yes → Probably STRATEGY
- No → Continue
5. Does it assert a foundational truth about how things work?
- Yes → Probably PRINCIPLE
- No → Review - may need more context or may be too vague to extract
---
Boundary Cases
Principle vs Mechanism
- Principle: "Trust is essential for collaboration"
- Mechanism: "Trust enables collaboration by reducing transaction costs"
- The mechanism explains WHY the principle is true
Mechanism vs Pattern
- Mechanism: "Feedback loops amplify small changes"
- Pattern: "The growth-plateau-renewal cycle"
- Patterns describe structure; mechanisms explain causation
Strategy vs Tactic
- Strategy: "Prioritize quality over quantity"
- Tactic: "Review each post before publishing using this checklist"
- If you can do it in one sitting, it's probably a tactic
Pattern vs Principle
- Pattern: "The five stages of team development"
- Principle: "Teams require psychological safety to perform"
- Patterns have structure; principles are singular claims
---
When Classification is Unclear
If a concept doesn't clearly fit one type:
1. Consider the author's apparent intent 2. Look at surrounding context for clues 3. Choose the type that best serves retrieval 4. Note uncertainty in the concept record 5. Flag for human review if needed
It's better to classify with noted uncertainty than to skip extraction of a valuable concept.
Functional Axes for Non-Fiction Analysis
This reference adapts the six-axis functional analysis framework (from dna-extraction) for non-fiction concept analysis. Use these axes to understand not just WHAT a concept says, but what it DOES.
Core Principle
The value of a concept is in its function, not just its content.
A concept like "trust enables collaboration" isn't just a statement to record - it serves multiple functions in the book's argument, the reader's understanding, and the broader knowledge domain.
The Six Axes (Adapted for Non-Fiction)
| Axis | Question | What It Reveals |
|---|---|---|
| Form | How is it presented? | The surface articulation (adaptable to other forms) |
| Structural Function | How does it support the argument? | Role in the book's logical structure |
| Application Function | How can it be used? | Practical utility for the reader |
| Emotional Function | What response does it create? | Motivation, conviction, resonance |
| Thematic Function | What ideas does it connect to? | Broader intellectual context |
| Relational Function | How does it connect to other concepts? | Web of relationships within the domain |
---
Axis 1: Form
Question: How is this concept presented?
The surface articulation - the specific words, examples, and framing the author uses. This is what we directly extract and cite.
What to Capture:
- Exact phrasing used
- Examples given
- Analogies or metaphors
- Visual representations (if any)
- The author's specific terminology
Why It Matters:
- Preserves the author's voice
- Enables accurate citation
- May reveal author's perspective or bias
- Different forms suit different audiences
Example:
- Form: "Trust is the glue that holds communities together"
- Note: Metaphor-based presentation; emphasizes cohesion function
Form vs. Function: The form is "glue metaphor." The function is "explains why trust matters for community cohesion."
---
Axis 2: Structural Function
Question: How does this concept support the book's argument?
The role this concept plays in the book's overall logical structure. Is it a premise? A conclusion? Evidence? A transition?
Structural Roles:
- Foundation: Premises the argument rests on
- Pillar: Major supporting points
- Evidence: Data or examples backing claims
- Synthesis: Conclusions drawn from earlier points
- Bridge: Transitions between major sections
- Counterpoint: Anticipated objections and responses
What to Capture:
- Where this concept appears in the book's structure
- What it supports or enables
- What depends on it
- How it connects to the thesis
Example:
- Concept: "Communities form around shared identity"
- Structural Function: Foundation - this is a premise that subsequent chapters build on
Why It Matters:
- Helps understand what's load-bearing vs. decorative
- Enables reconstruction of the book's argument
- Shows what would break if this concept were removed
---
Axis 3: Application Function
Question: How can this concept be used by the reader?
The practical utility - what actions, decisions, or understanding this enables.
Application Types:
- Diagnostic: Helps identify problems or states
- Prescriptive: Guides specific actions
- Evaluative: Provides criteria for judgment
- Predictive: Enables forecasting outcomes
- Explanatory: Helps understand phenomena
- Generative: Enables creation of new things
What to Capture:
- What someone could DO with this knowledge
- What decisions it informs
- What problems it helps solve
- What capabilities it enables
Example:
- Concept: "Communities need shared rituals"
- Application Function: Prescriptive (guides action) + Diagnostic (explains why some communities fail)
Why It Matters:
- Determines the concept's practical value
- Helps readers apply knowledge
- Identifies actionable vs. theoretical concepts
---
Axis 4: Emotional Function
Question: What response does this concept create in readers?
The motivational or affective dimension - how this concept makes readers feel and why that matters.
Emotional Functions:
- Validates: Confirms reader's existing beliefs/experiences
- Challenges: Disrupts assumptions, creates productive discomfort
- Inspires: Motivates action or aspiration
- Reassures: Reduces anxiety about complexity
- Clarifies: Relief of understanding something confusing
- Connects: Creates sense of shared understanding
What to Capture:
- What emotional state the author seems to want to create
- How examples and framing contribute to emotional response
- Whether the concept is meant to comfort or challenge
Example:
- Concept: "Most community-building advice is backwards"
- Emotional Function: Challenges (disrupts), then Clarifies (the correction provides relief)
Why It Matters:
- Emotional resonance affects retention and application
- Helps understand why certain concepts "stick"
- Reveals author's rhetorical strategy
---
Axis 5: Thematic Function
Question: What broader ideas does this concept connect to?
The intellectual context - how this concept fits into larger conversations, fields, and themes.
Thematic Connections:
- Field placement: Where does this fit in the discipline?
- Historical context: What intellectual tradition is this part of?
- Adjacent themes: What other big ideas does this relate to?
- Debates: What ongoing conversations does this engage?
- Paradigms: What worldview does this assume or challenge?
What to Capture:
- What larger themes this concept exemplifies
- What intellectual traditions it draws from
- How it relates to debates in the field
- What assumptions it makes about the world
Example:
- Concept: "Community membership is performative, not given"
- Thematic Function: Connects to social constructionism, performance theory, identity politics debates
Why It Matters:
- Enables cross-book synthesis
- Reveals intellectual lineage
- Helps position concept in broader knowledge
---
Axis 6: Relational Function
Question: How does this concept connect to other concepts?
The web of relationships - how concepts interact, support, contradict, or build on each other.
Relational Types:
- Prerequisite: What must be understood first?
- Enables: What other concepts does this make possible?
- Conflicts: What concepts does this tension with?
- Exemplifies: What higher-level concepts does this demonstrate?
- Decomposes: What sub-concepts does this break into?
- Parallels: What concepts in other domains are similar?
What to Capture:
- Direct relationships mentioned by author
- Implicit relationships from proximity or structure
- Cross-book relationships
- Relationships to concepts from other domains
Example:
- Concept: "Trust requires competence AND character"
- Relational Function: Decomposes into two sub-concepts; contradicts "trust is just about likability"; enables "specific trust-building tactics"
Why It Matters:
- Builds the knowledge network
- Enables navigation and discovery
- Supports synthesis across sources
---
Applying the Six Axes
When to Use Full Six-Axis Analysis
Use full analysis for:
- Key concepts central to the book's argument
- Novel or surprising concepts
- Concepts that will be used in synthesis
Use abbreviated analysis for:
- Supporting examples
- Well-established concepts
- Highly specific implementations
Six-Axis Template
For each significant concept:
**Concept:** [Name or phrase]
**Citation:** [Exact quote with position]
**Axes:**
1. Form: [How it's presented]
2. Structural: [Role in argument]
3. Application: [How it can be used]
4. Emotional: [What response it creates]
5. Thematic: [What ideas it connects to]
6. Relational: [How it connects to other concepts]Example: Complete Six-Axis Extraction
Concept: "The community lifecycle follows a pattern of formation, growth, maturation, and renewal"
Citation: "Every community... goes through predictable stages..." (p. 45, chars 12340-12520)
Axes: 1. Form: Four-stage cyclical model, explicitly named stages, biological metaphor 2. Structural: Framework that organizes Part II of the book; each chapter addresses one stage 3. Application: Diagnostic (identify which stage you're in); Prescriptive (different tactics per stage) 4. Emotional: Reassures (stages are normal); Validates (explains past community experiences) 5. Thematic: Connects to lifecycle thinking in business, product management, organizational development 6. Relational:
- Prerequisite: "Communities exist" (basic definition)
- Enables: Stage-specific tactics (Chapters 5-8)
- Parallels: Product lifecycle, organizational development stages
- Decomposes: Four sub-concepts (formation, growth, maturation, renewal)
---
Integration with Concept Classification
The six axes complement type and layer classification:
| Classification | Six-Axis Addition |
|---|---|
| Type (principle/mechanism/pattern/strategy/tactic) | What kind of concept |
| Layer (0-4) | How abstract/concrete |
| Six Axes | What the concept DOES |
A complete concept record includes:
- Type classification
- Layer assignment
- Full citation with position
- Six-axis functional analysis (for key concepts)
- Relationship links
---
When Six-Axis Analysis Adds Value
High Value:
- Cross-book synthesis (understanding what each concept does enables better comparison)
- Concept retrieval (functional annotations improve search)
- Application planning (knowing application function helps implementation)
Lower Value:
- Simple concept extraction (full analysis is overkill for minor points)
- High-volume extraction (time-consuming for every concept)
- Well-known concepts (don't need to analyze "trust is important")
Recommendation: Apply full six-axis analysis to ~20% of extracted concepts (the most important ones). Use abbreviated analysis for the rest.
Relationship Types
This reference defines the relationship types used to link concepts. Use these definitions to guide relationship creation.
Overview
| Relationship | Meaning | Directionality |
|---|---|---|
| INFLUENCES | A affects B | Directed (A→B) |
| SUPPORTS | A provides evidence for B | Directed (A→B) |
| CONTRADICTS | A conflicts with B | Bidirectional |
| COMPOSED_OF | A contains B as component | Directed (A→B) |
| DERIVES_FROM | A is derived from B | Directed (A→B) |
---
INFLUENCES
Definition: Concept A has a causal or correlational effect on Concept B. A change in A would affect B.
When to Use:
- A causes B to happen
- A enables or facilitates B
- A inhibits or prevents B
- A is a precondition for B
Strength Values:
- Positive (+0.1 to +1.0): A increases/enables B
- Negative (-0.1 to -1.0): A decreases/inhibits B
- Strong (±0.7 to ±1.0): Direct causal relationship
- Moderate (±0.4 to ±0.6): Significant but indirect
- Weak (±0.1 to ±0.3): Correlational or minor
Examples:
- "Trust" INFLUENCES→ "Collaboration" (strength: +0.8)
- "Fear" INFLUENCES→ "Risk-taking" (strength: -0.7)
- "Shared identity" INFLUENCES→ "Community cohesion" (strength: +0.9)
Not INFLUENCES:
- A simply mentions B (use context, not relationship)
- A is a component of B (use COMPOSED_OF)
- A is evidence for B (use SUPPORTS)
---
SUPPORTS
Definition: Concept A provides evidence, validation, or backing for Concept B. A makes B more credible or justified.
When to Use:
- A is empirical evidence for B
- A is an example that demonstrates B
- A is a theoretical justification for B
- A is a citation or reference for B
Strength Values:
- Strong: Direct empirical evidence or proof
- Moderate: Good example or theoretical support
- Weak: Tangential or partial support
Examples:
- "Research study X" SUPPORTS→ "Spaced repetition improves retention"
- "Case study of Company Y" SUPPORTS→ "Community-led growth works"
- "Historical precedent Z" SUPPORTS→ "Revolutions follow pattern P"
Not SUPPORTS:
- A causes B to happen (use INFLUENCES)
- A is part of B (use COMPOSED_OF)
- B is derived from A (use DERIVES_FROM)
---
CONTRADICTS
Definition: Concept A conflicts with, opposes, or is incompatible with Concept B. They cannot both be true in the same context.
When to Use:
- A and B make opposite claims
- A and B recommend incompatible approaches
- Evidence for A undermines B
- A and B are mutually exclusive
Strength Values:
- Strong: Direct logical contradiction
- Moderate: Significant tension or incompatibility
- Weak: Partial tension or contextual conflict
Types of Contradiction: 1. Theoretical: Different claims about how things work 2. Practical: Different recommendations for action 3. Contextual: True in different contexts but not compatible
Examples:
- "Move fast and break things" CONTRADICTS↔ "Measure twice, cut once"
- "Communities need strong leadership" CONTRADICTS↔ "Communities should be self-organizing"
- "Research shows X" CONTRADICTS↔ "Research shows not-X"
Bidirectionality: CONTRADICTS is bidirectional - if A contradicts B, then B contradicts A.
Not CONTRADICTS:
- A is simply different from B (difference ≠ contradiction)
- A and B apply in different contexts (may just be contextual)
- A and B are different levels of abstraction
---
COMPOSED_OF
Definition: Concept A contains Concept B as a component, element, or part. B is nested within A.
When to Use:
- B is a named component of framework A
- B is a step in process A
- B is a required element of A
- A explicitly includes B in its definition
Component Types:
- Prerequisite: B must exist before A
- Component: B is a part of A
- Variant: B is a subtype of A
Examples:
- "Hero's journey" COMPOSED_OF→ "Departure"
- "Hero's journey" COMPOSED_OF→ "Initiation"
- "Hero's journey" COMPOSED_OF→ "Return"
- "Trust" COMPOSED_OF→ "Competence trust" (variant)
- "Trust" COMPOSED_OF→ "Character trust" (variant)
Not COMPOSED_OF:
- B merely relates to A (use INFLUENCES or other)
- B is evidence for A (use SUPPORTS)
- B causes A (use INFLUENCES)
---
DERIVES_FROM
Definition: Concept A is logically derived from, emerges from, or is a consequence of Concept B. A exists because of B.
When to Use:
- A is a logical conclusion from B
- A emerges from B in a system
- A is an application of B to a context
- A is a specialization of B
Derivation Types:
- Logical conclusion: A follows from B by reasoning
- Emergent property: A arises from B in complex systems
- Synthesis: A combines multiple sources including B
Examples:
- "Community guidelines" DERIVES_FROM→ "Shared values"
- "Specific onboarding tactics" DERIVES_FROM→ "Engagement theory"
- "Leadership practices" DERIVES_FROM→ "Trust principles"
Not DERIVES_FROM:
- B influences A (that's INFLUENCES)
- A is part of B (that's the reverse - B COMPOSED_OF A)
- A is evidence for B (that's the reverse - A SUPPORTS B)
---
Creating Relationships
When to Create a Relationship
Create a relationship when: 1. The connection is explicitly stated in the source 2. The connection is strongly implied by context 3. The connection would be useful for synthesis and querying
Do NOT create relationships:
- Based on mere co-occurrence in text
- When the connection is trivial or obvious
- When evidence for the connection is weak
Relationship Density
Aim for:
- Every concept linked to at least 1-2 others
- Key concepts (hubs) linked to many others
- Both within-book and cross-book relationships
Avoid:
- Orphan concepts with no relationships
- Over-connected concepts (>10 relationships may indicate over-extraction)
- Circular relationships without clear semantics
Relationship Confidence
Track confidence in relationships:
- High (0.8-1.0): Explicitly stated in source
- Medium (0.5-0.7): Strongly implied or inferred
- Low (0.2-0.4): Reasonable inference, less certain
Flag low-confidence relationships for human review.
---
Cross-Book Relationships
When analyzing multiple books, additional patterns emerge:
Agreement
Both authors say A INFLUENCES B → Stronger confidence
Complementarity
Author 1 says A INFLUENCES B Author 2 says B INFLUENCES C Combined: A → B → C pathway
Contradiction
Author 1 says A SUPPORTS claim X Author 2 says B CONTRADICTS claim X Flag for synthesis discussion
Extension
Author 1 defines framework F Author 2 applies F to new context DERIVES_FROM relationship across books
---
Relationship Examples by Domain
Community Building
- "Trust" INFLUENCES→ "Participation" (+0.8)
- "Shared identity" INFLUENCES→ "Retention" (+0.7)
- "Explicit guidelines" SUPPORTS→ "Safe participation"
- "Top-down control" CONTRADICTS↔ "Self-organization"
- "Engagement ladder" COMPOSED_OF→ "Lurker stage"
Learning
- "Active recall" INFLUENCES→ "Retention" (+0.9)
- "Spacing effect" SUPPORTS→ "Spaced repetition works"
- "Massed practice" CONTRADICTS↔ "Distributed practice"
- "Learning pyramid" COMPOSED_OF→ "Reading (10%)"
- "Retrieval practice" DERIVES_FROM→ "Testing effect research"
Leadership
- "Psychological safety" INFLUENCES→ "Team performance" (+0.8)
- "Google Project Aristotle" SUPPORTS→ "Safety matters"
- "Servant leadership" CONTRADICTS↔ "Command-and-control"
- "Situational leadership" COMPOSED_OF→ "Directing style"
- "Specific feedback tactics" DERIVES_FROM→ "Growth mindset theory"
#!/usr/bin/env -S deno run -A
/**
* bc-assemble-index.ts - Assemble final book classification index
*
* Merges tag-classified and LLM-classified books into a single index.
*
* Usage:
* deno run -A bc-assemble-index.ts <classified-by-tags.json> <llm-classified.json> [--output <file>]
*/
type Category = "fiction" | "cookbooks" | "technical" | "business" | "self_help" | "other_nonfiction";
interface TagClassifiedBook {
book_id: number;
title: string;
author: string;
category: Category;
confidence: number;
classification_source: "tag_mapping";
source_tags: string[];
matched_rules: string[];
classified_at: string;
}
interface LlmClassifiedBook {
book_id: number;
title: string;
author: string;
category: Category;
confidence: number;
classification_source: "llm";
llm_reasoning: string;
classified_at: string;
}
type ClassifiedBook = TagClassifiedBook | LlmClassifiedBook;
interface BookIndex {
version: string;
generated: string;
collection_path: string;
categories: {
[key in Category]: {
name: string;
description: string;
};
};
statistics: {
total_books: number;
classified_from_tags: number;
classified_by_llm: number;
category_counts: { [key in Category]: number };
};
books: ClassifiedBook[];
}
const categoryDescriptions: { [key in Category]: { name: string; description: string } } = {
fiction: {
name: "Fiction",
description: "Novels, short stories, literary works",
},
cookbooks: {
name: "Cookbooks",
description: "Cooking, recipes, food & beverage",
},
technical: {
name: "Technical/Computing",
description: "Programming, software, computers, engineering",
},
business: {
name: "Business",
description: "Business, economics, management, finance",
},
self_help: {
name: "Self-Help",
description: "Personal development, psychology, wellness",
},
other_nonfiction: {
name: "Other Non-Fiction",
description: "History, science, reference, crafts, etc.",
},
};
if (import.meta.main) {
const tagClassifiedPath = Deno.args[0];
const llmClassifiedPath = Deno.args[1];
let outputPath = "book-classification-index.json";
for (let i = 2; i < Deno.args.length; i++) {
if (Deno.args[i] === "--output" && Deno.args[i + 1]) {
outputPath = Deno.args[++i];
}
}
if (!tagClassifiedPath || !llmClassifiedPath) {
console.error("Usage: deno run -A bc-assemble-index.ts <classified-by-tags.json> <llm-classified.json> [--output <file>]");
Deno.exit(1);
}
console.log(`Reading tag-classified: ${tagClassifiedPath}`);
const tagClassified: TagClassifiedBook[] = JSON.parse(Deno.readTextFileSync(tagClassifiedPath));
console.log(`Reading LLM-classified: ${llmClassifiedPath}`);
const llmClassified: LlmClassifiedBook[] = JSON.parse(Deno.readTextFileSync(llmClassifiedPath));
// Merge books
const allBooks: ClassifiedBook[] = [...tagClassified, ...llmClassified];
// Sort by book_id
allBooks.sort((a, b) => a.book_id - b.book_id);
// Calculate statistics
const categoryCounts: { [key in Category]: number } = {
fiction: 0,
cookbooks: 0,
technical: 0,
business: 0,
self_help: 0,
other_nonfiction: 0,
};
for (const book of allBooks) {
categoryCounts[book.category]++;
}
const index: BookIndex = {
version: "1.0.0",
generated: new Date().toISOString(),
collection_path: Deno.env.get("BOOKS_DIR") || "./books",
categories: categoryDescriptions,
statistics: {
total_books: allBooks.length,
classified_from_tags: tagClassified.length,
classified_by_llm: llmClassified.length,
category_counts: categoryCounts,
},
books: allBooks,
};
// Write output
Deno.writeTextFileSync(outputPath, JSON.stringify(index, null, 2));
console.log(`\nWrote index to ${outputPath}`);
// Summary
console.log("\n=== Final Index Summary ===");
console.log(`Total books: ${index.statistics.total_books}`);
console.log(`Classified by tags: ${index.statistics.classified_from_tags}`);
console.log(`Classified by LLM: ${index.statistics.classified_by_llm}`);
console.log("\nCategory distribution:");
for (const [category, count] of Object.entries(categoryCounts)) {
const pct = ((count / allBooks.length) * 100).toFixed(1);
console.log(` ${category}: ${count} (${pct}%)`);
}
}
#!/usr/bin/env -S deno run -A
/**
* bc-classify-untagged.ts - Classify books without tags using heuristics
*
* Reads needs-llm.json and classifies books based on title/description patterns.
* Books that can't be confidently classified are flagged for manual review.
*
* Usage:
* deno run -A bc-classify-untagged.ts <needs-llm.json> [--output <file>]
*/
type Category = "fiction" | "cookbooks" | "technical" | "business" | "self_help" | "other_nonfiction";
interface UnclassifiedBook {
book_id: number;
title: string;
author: string;
description?: string;
source_tags: string[];
reason: string;
}
interface ClassifiedBook {
book_id: number;
title: string;
author: string;
category: Category;
confidence: number;
classification_source: "llm";
llm_reasoning: string;
classified_at: string;
}
// Keyword patterns for each category
const categoryPatterns: Record<Category, { keywords: RegExp[]; weight: number }[]> = {
fiction: [
{ keywords: [/\bnovel\b/i, /\bstories\b/i, /\bfiction\b/i], weight: 1.0 },
{ keywords: [/\bmystery\b/i, /\bthriller\b/i, /\bsuspense\b/i], weight: 0.9 },
{ keywords: [/\bfantasy\b/i, /\bsci-?fi\b/i, /\bscience fiction\b/i], weight: 0.9 },
{ keywords: [/\bromance\b/i, /\bhorror\b/i, /\bdrama\b/i], weight: 0.8 },
],
cookbooks: [
{ keywords: [/\brecipes?\b/i, /\bcookbook\b/i, /\bcooking\b/i], weight: 1.0 },
{ keywords: [/\bbaking\b/i, /\bdesserts?\b/i, /\bbreads?\b/i], weight: 0.9 },
{ keywords: [/\bmeals?\b/i, /\bdinner\b/i, /\blunch\b/i, /\bbreakfast\b/i], weight: 0.8 },
{ keywords: [/\bwine\b/i, /\bbeer\b/i, /\bcocktails?\b/i, /\bbeverages?\b/i], weight: 0.85 },
{ keywords: [/\blow-carb\b/i, /\bketo\b/i, /\bpaleo\b/i, /\bdiet\b/i], weight: 0.7 },
{ keywords: [/\bslow cooker\b/i, /\binstant pot\b/i, /\bair fryer\b/i], weight: 0.9 },
{ keywords: [/\bpudding\b/i, /\bcakes?\b/i, /\bpies?\b/i, /\bpasta\b/i], weight: 0.8 },
],
technical: [
{ keywords: [/\bprogramming\b/i, /\bcode\b/i, /\bcoding\b/i], weight: 1.0 },
{ keywords: [/\bjavascript\b/i, /\bpython\b/i, /\bjava\b/i, /\bc\+\+\b/i], weight: 1.0 },
{ keywords: [/\bweb\s*development\b/i, /\bsoftware\b/i], weight: 0.95 },
{ keywords: [/\bdatabase\b/i, /\bsql\b/i, /\bapi\b/i], weight: 0.9 },
{ keywords: [/\blinux\b/i, /\bdevops\b/i, /\bcloud\b/i], weight: 0.9 },
{ keywords: [/\balgorithms?\b/i, /\bdata structures?\b/i], weight: 0.95 },
{ keywords: [/\bmachine learning\b/i, /\bai\b/i, /\bdeep learning\b/i], weight: 0.9 },
],
business: [
{ keywords: [/\bmanagement\b/i, /\bleadership\b/i, /\bstrategy\b/i], weight: 0.9 },
{ keywords: [/\bbusiness\b/i, /\bentrepreneur\b/i, /\bstartup\b/i], weight: 0.9 },
{ keywords: [/\bmarketing\b/i, /\bsales\b/i, /\badvertising\b/i], weight: 0.85 },
{ keywords: [/\bfinance\b/i, /\binvesting\b/i, /\bmoney\b/i], weight: 0.8 },
{ keywords: [/\bcareer\b/i, /\bjob\b/i, /\bworkplace\b/i], weight: 0.7 },
{ keywords: [/\bllc\b/i, /\bcorporation\b/i, /\bcompany\b/i], weight: 0.8 },
],
self_help: [
{ keywords: [/\bself-help\b/i, /\bself help\b/i, /\bpersonal growth\b/i], weight: 1.0 },
{ keywords: [/\bhappiness\b/i, /\bmindfulness\b/i, /\bmeditation\b/i], weight: 0.9 },
{ keywords: [/\bproductivity\b/i, /\bhabits?\b/i, /\bmotivation\b/i], weight: 0.85 },
{ keywords: [/\banxiety\b/i, /\bdepression\b/i, /\bmental health\b/i], weight: 0.85 },
{ keywords: [/\brelationships?\b/i, /\bdating\b/i, /\bmarriage\b/i], weight: 0.75 },
{ keywords: [/\bconfidence\b/i, /\bself-esteem\b/i], weight: 0.85 },
],
other_nonfiction: [
{ keywords: [/\bhistory\b/i, /\bbiography\b/i, /\bmemoir\b/i], weight: 0.9 },
{ keywords: [/\bwriting\b/i, /\bwriter\b/i, /\bauthor\b/i], weight: 0.8 },
{ keywords: [/\bprompts?\b/i, /\bworldbuilding\b/i], weight: 0.85 },
{ keywords: [/\breference\b/i, /\bguide\b/i, /\bhandbook\b/i], weight: 0.7 },
{ keywords: [/\bcraft\b/i, /\bdiy\b/i, /\bprojects?\b/i], weight: 0.8 },
{ keywords: [/\btravel\b/i, /\blocals\b/i, /\bplaces\b/i], weight: 0.85 },
{ keywords: [/\bwoodwork\b/i, /\bfurniture\b/i, /\bhome improvement\b/i], weight: 0.8 },
{ keywords: [/\bfacts?\b/i, /\btrivia\b/i, /\bquiz\b/i], weight: 0.75 },
{ keywords: [/\bscience\b/i, /\bphysics\b/i, /\bbiology\b/i], weight: 0.8 },
{ keywords: [/\bphilosophy\b/i, /\breligion\b/i, /\bspiritual\b/i], weight: 0.8 },
{ keywords: [/\bmusic\b/i, /\bart\b/i, /\bphotography\b/i], weight: 0.8 },
],
};
function classifyBook(book: UnclassifiedBook): ClassifiedBook {
const text = `${book.title} ${book.description || ""}`.toLowerCase();
const scores: Map<Category, { score: number; matches: string[] }> = new Map();
// Calculate scores for each category
for (const [category, patterns] of Object.entries(categoryPatterns)) {
let totalScore = 0;
const matches: string[] = [];
for (const { keywords, weight } of patterns) {
for (const keyword of keywords) {
if (keyword.test(text)) {
totalScore += weight;
matches.push(keyword.source);
}
}
}
scores.set(category as Category, { score: totalScore, matches });
}
// Find best category
let bestCategory: Category = "other_nonfiction";
let bestScore = 0;
let bestMatches: string[] = [];
for (const [category, { score, matches }] of scores) {
if (score > bestScore) {
bestScore = score;
bestCategory = category;
bestMatches = matches;
}
}
// Calculate confidence based on score
let confidence: number;
if (bestScore >= 1.5) {
confidence = 0.95;
} else if (bestScore >= 1.0) {
confidence = 0.85;
} else if (bestScore >= 0.7) {
confidence = 0.75;
} else if (bestScore > 0) {
confidence = 0.65;
} else {
confidence = 0.5;
}
// Generate reasoning
let reasoning: string;
if (bestMatches.length > 0) {
reasoning = `Matched patterns: ${bestMatches.slice(0, 3).join(", ")}`;
} else {
reasoning = "No strong pattern matches, defaulted to other_nonfiction";
}
return {
book_id: book.id,
title: book.title,
author: book.author,
category: bestCategory,
confidence,
classification_source: "llm",
llm_reasoning: reasoning,
classified_at: new Date().toISOString(),
};
}
// Fix: handle both book_id and id fields
function normalizeBook(book: UnclassifiedBook & { id?: number }): UnclassifiedBook & { id: number } {
return {
...book,
id: book.book_id || book.id || 0,
};
}
if (import.meta.main) {
const inputPath = Deno.args[0] || "needs-llm.json";
let outputPath = "llm-classified.json";
for (let i = 1; i < Deno.args.length; i++) {
if (Deno.args[i] === "--output" && Deno.args[i + 1]) {
outputPath = Deno.args[++i];
}
}
console.log(`Reading: ${inputPath}`);
const text = Deno.readTextFileSync(inputPath);
const books: UnclassifiedBook[] = JSON.parse(text);
console.log(`Classifying ${books.length} books...`);
const classified: ClassifiedBook[] = [];
const categoryStats: Map<Category, number> = new Map();
const lowConfidence: ClassifiedBook[] = [];
for (const book of books) {
const normalized = normalizeBook(book);
const result = classifyBook(normalized);
classified.push(result);
categoryStats.set(result.category, (categoryStats.get(result.category) || 0) + 1);
if (result.confidence < 0.7) {
lowConfidence.push(result);
}
}
// Write output
Deno.writeTextFileSync(outputPath, JSON.stringify(classified, null, 2));
console.log(`\nWrote ${classified.length} classifications to ${outputPath}`);
// Summary
console.log("\n=== Classification Summary ===");
console.log("Category distribution:");
const categories: Category[] = ["fiction", "cookbooks", "technical", "business", "self_help", "other_nonfiction"];
for (const category of categories) {
const count = categoryStats.get(category) || 0;
console.log(` ${category}: ${count}`);
}
console.log(`\nLow confidence (< 0.7): ${lowConfidence.length} books`);
if (lowConfidence.length > 0 && lowConfidence.length <= 20) {
console.log("Low confidence books:");
for (const book of lowConfidence) {
console.log(` [${book.book_id}] ${book.title.slice(0, 50)}... -> ${book.category} (${book.confidence})`);
}
}
}
#!/usr/bin/env -S deno run -A
/**
* bc-list-books.ts - Book Listing and Filtering CLI
*
* List and filter books from a Calibre metadata.db database.
*
* Usage:
* deno run -A bc-list-books.ts <metadata.db> [options]
*
* Options:
* --tag <name> Filter by tag name
* --search <query> Search by title
* --author <name> Filter by author name
* --has-tags Only books with tags
* --no-tags Only books without tags
* --limit <n> Limit results
* --format <type> Output format: table (default), json, csv
* --stats Show statistics only
* --tags-list Show all tags with counts
*/
import {
openCalibreDb,
closeCalibreDb,
getBooks,
getAllTags,
getStats,
type CalibreBook,
type GetBooksOptions,
} from "./calibre-db.ts";
function parseArgs(args: string[]): {
dbPath: string;
options: GetBooksOptions;
format: "table" | "json" | "csv";
showStats: boolean;
showTagsList: boolean;
} {
const dbPath = args[0];
if (!dbPath) {
console.error("Error: Database path required");
console.error("Usage: deno run -A bc-list-books.ts <metadata.db> [options]");
console.error("");
console.error("Options:");
console.error(" --tag <name> Filter by tag name");
console.error(" --search <query> Search by title");
console.error(" --author <name> Filter by author name");
console.error(" --has-tags Only books with tags");
console.error(" --no-tags Only books without tags");
console.error(" --limit <n> Limit results");
console.error(" --format <type> Output format: table (default), json, csv");
console.error(" --stats Show statistics only");
console.error(" --tags-list Show all tags with counts");
Deno.exit(1);
}
const options: GetBooksOptions = {};
let format: "table" | "json" | "csv" = "table";
let showStats = false;
let showTagsList = false;
for (let i = 1; i < args.length; i++) {
const arg = args[i];
switch (arg) {
case "--tag":
options.tagFilter = args[++i];
break;
case "--search":
options.titleSearch = args[++i];
break;
case "--author":
options.authorFilter = args[++i];
break;
case "--has-tags":
options.hasTagsOnly = true;
break;
case "--no-tags":
options.noTagsOnly = true;
break;
case "--limit":
options.limit = parseInt(args[++i], 10);
break;
case "--format":
format = args[++i] as "table" | "json" | "csv";
break;
case "--stats":
showStats = true;
break;
case "--tags-list":
showTagsList = true;
break;
}
}
return { dbPath, options, format, showStats, showTagsList };
}
function formatTable(books: CalibreBook[]): void {
console.log("ID\tTitle\tAuthor\tTags");
console.log("---\t-----\t------\t----");
for (const book of books) {
const title = book.title.length > 50 ? book.title.slice(0, 47) + "..." : book.title;
const author = book.author.length > 25 ? book.author.slice(0, 22) + "..." : book.author;
const tags = book.tags.slice(0, 3).join(", ") + (book.tags.length > 3 ? "..." : "");
console.log(`${book.id}\t${title}\t${author}\t${tags}`);
}
console.log(`\n${books.length} books`);
}
function formatJson(books: CalibreBook[]): void {
console.log(JSON.stringify(books, null, 2));
}
function formatCsv(books: CalibreBook[]): void {
console.log("id,title,author,tags,description,publisher,isbn,series");
for (const book of books) {
const escapeCsv = (s: string | undefined) => {
if (!s) return "";
if (s.includes(",") || s.includes('"') || s.includes("\n")) {
return `"${s.replace(/"/g, '""')}"`;
}
return s;
};
console.log(
[
book.id,
escapeCsv(book.title),
escapeCsv(book.author),
escapeCsv(book.tags.join("; ")),
escapeCsv(book.description?.slice(0, 200)),
escapeCsv(book.publisher),
escapeCsv(book.isbn),
escapeCsv(book.series),
].join(",")
);
}
}
if (import.meta.main) {
const { dbPath, options, format, showStats, showTagsList } = parseArgs(Deno.args);
const db = openCalibreDb(dbPath);
try {
if (showStats) {
const stats = getStats(db);
console.log("Database Statistics:");
console.log(` Total books: ${stats.totalBooks}`);
console.log(` Books with tags: ${stats.booksWithTags}`);
console.log(` Books without tags: ${stats.booksWithoutTags}`);
console.log(` Total unique tags: ${stats.totalTags}`);
console.log(` Total authors: ${stats.totalAuthors}`);
} else if (showTagsList) {
const tags = getAllTags(db);
if (format === "json") {
console.log(JSON.stringify(tags, null, 2));
} else if (format === "csv") {
console.log("id,name,count");
for (const tag of tags) {
console.log(`${tag.id},${tag.name},${tag.count}`);
}
} else {
console.log("ID\tTag\tCount");
console.log("---\t---\t-----");
for (const tag of tags) {
console.log(`${tag.id}\t${tag.name}\t${tag.count}`);
}
console.log(`\n${tags.length} tags`);
}
} else {
const books = getBooks(db, options);
switch (format) {
case "json":
formatJson(books);
break;
case "csv":
formatCsv(books);
break;
default:
formatTable(books);
}
}
} finally {
closeCalibreDb(db);
}
}
#!/usr/bin/env -S deno run -A
/**
* bc-map-tags.ts - Book Classification via Tag Mapping
*
* Reads books from Calibre metadata.db and classifies them using tag mapping rules.
* Books that can be classified are written to classified-by-tags.json.
* Books that need LLM classification are written to needs-llm.json.
*
* Usage:
* deno run -A bc-map-tags.ts <metadata.db> [--output-dir <dir>]
*/
import { openCalibreDb, closeCalibreDb, getBooks, type CalibreBook } from "./calibre-db.ts";
// === TYPES ===
type Category = "fiction" | "cookbooks" | "technical" | "business" | "self_help" | "other_nonfiction";
interface TagMappingRules {
_meta: {
description: string;
version: string;
last_updated: string;
categories: Category[];
};
priority_order: Category[];
excluded_tags: string[];
category_rules: {
[key in Category]: {
exact_matches: string[];
pattern_matches: string[];
};
};
confidence_rules: {
single_definitive_tag: number;
multiple_same_category: number;
primary_clear_secondary_different: number;
ambiguous_resolved_by_priority: number;
very_ambiguous: number;
};
}
interface ClassifiedBook {
book_id: number;
title: string;
author: string;
category: Category;
confidence: number;
classification_source: "tag_mapping";
source_tags: string[];
matched_rules: string[];
classified_at: string;
}
interface UnclassifiedBook {
book_id: number;
title: string;
author: string;
description?: string;
source_tags: string[];
reason: "no_tags" | "excluded_tags_only" | "no_matching_rules";
}
// === CLASSIFICATION LOGIC ===
function loadMappingRules(path: string): TagMappingRules {
const text = Deno.readTextFileSync(path);
return JSON.parse(text);
}
function matchTag(tag: string, category: string, rules: TagMappingRules): boolean {
const categoryRules = rules.category_rules[category as Category];
if (!categoryRules) return false;
// Check exact matches (case-insensitive)
if (categoryRules.exact_matches.some((m) => m.toLowerCase() === tag.toLowerCase())) {
return true;
}
// Check pattern matches
for (const pattern of categoryRules.pattern_matches) {
const regex = new RegExp(pattern, "i");
if (regex.test(tag)) {
return true;
}
}
return false;
}
function classifyBook(
book: CalibreBook,
rules: TagMappingRules
): { classified: ClassifiedBook } | { unclassified: UnclassifiedBook } {
// Filter out excluded tags
const validTags = book.tags.filter(
(tag) => !rules.excluded_tags.some((ex) => ex.toLowerCase() === tag.toLowerCase())
);
// No tags at all
if (book.tags.length === 0) {
return {
unclassified: {
book_id: book.id,
title: book.title,
author: book.author,
description: book.description,
source_tags: [],
reason: "no_tags",
},
};
}
// Only excluded tags
if (validTags.length === 0) {
return {
unclassified: {
book_id: book.id,
title: book.title,
author: book.author,
description: book.description,
source_tags: book.tags,
reason: "excluded_tags_only",
},
};
}
// Find matching categories for each tag
const categoryMatches: Map<Category, string[]> = new Map();
const matchedRules: string[] = [];
for (const tag of validTags) {
for (const category of rules.priority_order) {
if (matchTag(tag, category, rules)) {
if (!categoryMatches.has(category)) {
categoryMatches.set(category, []);
}
categoryMatches.get(category)!.push(tag);
matchedRules.push(`${tag} -> ${category}`);
break; // Tag matches first priority category only
}
}
}
// No matching rules
if (categoryMatches.size === 0) {
return {
unclassified: {
book_id: book.id,
title: book.title,
author: book.author,
description: book.description,
source_tags: book.tags,
reason: "no_matching_rules",
},
};
}
// Select category based on priority order
let selectedCategory: Category | null = null;
for (const category of rules.priority_order) {
if (categoryMatches.has(category)) {
selectedCategory = category;
break;
}
}
if (!selectedCategory) {
// Shouldn't happen, but fallback
selectedCategory = "other_nonfiction";
}
// Calculate confidence
let confidence: number;
if (categoryMatches.size === 1) {
const matchCount = categoryMatches.get(selectedCategory)!.length;
if (matchCount === 1 && validTags.length === 1) {
confidence = rules.confidence_rules.single_definitive_tag;
} else {
confidence = rules.confidence_rules.multiple_same_category;
}
} else if (categoryMatches.size === 2) {
confidence = rules.confidence_rules.primary_clear_secondary_different;
} else {
confidence = rules.confidence_rules.ambiguous_resolved_by_priority;
}
return {
classified: {
book_id: book.id,
title: book.title,
author: book.author,
category: selectedCategory,
confidence,
classification_source: "tag_mapping",
source_tags: book.tags,
matched_rules: matchedRules,
classified_at: new Date().toISOString(),
},
};
}
// === MAIN ===
if (import.meta.main) {
const dbPath = Deno.args[0];
let outputDir = ".";
// Parse args
for (let i = 1; i < Deno.args.length; i++) {
if (Deno.args[i] === "--output-dir" && Deno.args[i + 1]) {
outputDir = Deno.args[++i];
}
}
if (!dbPath) {
console.error("Usage: deno run -A bc-map-tags.ts <metadata.db> [--output-dir <dir>]");
Deno.exit(1);
}
// Load rules
const scriptDir = new URL(".", import.meta.url).pathname;
const rulesPath = `${scriptDir}../data/tag-mapping-rules.json`;
console.log(`Loading rules from: ${rulesPath}`);
const rules = loadMappingRules(rulesPath);
// Open database
console.log(`Opening database: ${dbPath}`);
const db = openCalibreDb(dbPath);
try {
// Get all books
console.log("Fetching all books...");
const books = getBooks(db);
console.log(`Found ${books.length} books`);
// Classify books
const classified: ClassifiedBook[] = [];
const needsLlm: UnclassifiedBook[] = [];
const categoryStats: Map<Category, number> = new Map();
const unclassifiedReasons: Map<string, number> = new Map();
for (const book of books) {
const result = classifyBook(book, rules);
if ("classified" in result) {
classified.push(result.classified);
categoryStats.set(
result.classified.category,
(categoryStats.get(result.classified.category) || 0) + 1
);
} else {
needsLlm.push(result.unclassified);
unclassifiedReasons.set(
result.unclassified.reason,
(unclassifiedReasons.get(result.unclassified.reason) || 0) + 1
);
}
}
// Write outputs
const classifiedPath = `${outputDir}/classified-by-tags.json`;
const needsLlmPath = `${outputDir}/needs-llm.json`;
Deno.writeTextFileSync(classifiedPath, JSON.stringify(classified, null, 2));
console.log(`Wrote ${classified.length} classified books to ${classifiedPath}`);
Deno.writeTextFileSync(needsLlmPath, JSON.stringify(needsLlm, null, 2));
console.log(`Wrote ${needsLlm.length} books needing LLM classification to ${needsLlmPath}`);
// Print summary
console.log("\n=== Classification Summary ===");
console.log(`Total books: ${books.length}`);
console.log(`Classified by tags: ${classified.length}`);
console.log(`Needs LLM classification: ${needsLlm.length}`);
console.log("\nCategory distribution:");
for (const category of rules.priority_order) {
const count = categoryStats.get(category) || 0;
console.log(` ${category}: ${count}`);
}
console.log("\nUnclassified reasons:");
for (const [reason, count] of unclassifiedReasons) {
console.log(` ${reason}: ${count}`);
}
} finally {
closeCalibreDb(db);
}
}
#!/usr/bin/env -S deno run --allow-read --allow-write
/**
* bulk-preprocess.ts - Bulk Ebook Preprocessing
*
* Runs deterministic (non-LLM) processing on the ebook collection,
* generating parsed JSON files ready for agent analysis.
*
* Usage:
* deno run -A bulk-preprocess.ts # Process all books
* deno run -A bulk-preprocess.ts --category other_nonfiction
* deno run -A bulk-preprocess.ts --book-id 74 # Single book
* deno run -A bulk-preprocess.ts --resume # Skip already processed
* deno run -A bulk-preprocess.ts --dry-run # Show what would be processed
*/
import {
openCalibreDb,
closeCalibreDb,
getBookById,
type CalibreBook,
} from "./calibre-db.ts";
import { parseBook, type ParsedBook } from "./ea-parse.ts";
// === CONFIGURATION ===
// Set these paths via environment variables or modify defaults for your setup.
const EBOOKS_ROOT = Deno.env.get("EBOOKS_ROOT") || ".";
const BOOKS_DIR = Deno.env.get("BOOKS_DIR") || `${EBOOKS_ROOT}/books`;
const PREPROCESSED_DIR = Deno.env.get("PREPROCESSED_DIR") || `${EBOOKS_ROOT}/preprocessed`;
const CLASSIFICATION_INDEX = Deno.env.get("CLASSIFICATION_INDEX") || `${EBOOKS_ROOT}/book-classification-index.json`;
const CALIBRE_DB = Deno.env.get("CALIBRE_DB") || `${BOOKS_DIR}/metadata.db`;
const DEFAULT_CHUNK_SIZE = 1500;
const DEFAULT_OVERLAP = 150;
const DEFAULT_PARALLEL = 4;
// === INTERFACES ===
interface ClassificationEntry {
book_id: number;
title: string;
author: string;
category: string;
confidence: number;
classification_source: string;
llm_reasoning?: string;
source_tags?: string[];
matched_rules?: string[];
classified_at: string;
}
interface ClassificationIndex {
version: string;
generated: string;
collection_path: string;
categories: Record<string, { name: string; description: string }>;
statistics: {
total_books: number;
classified_from_tags: number;
classified_by_llm: number;
category_counts: Record<string, number>;
};
books: ClassificationEntry[];
}
interface BookMetadataOutput {
book_id: number;
title: string;
author: string;
category: string;
confidence: number;
tags: string[];
description?: string;
isbn?: string;
publisher?: string;
series?: string;
series_index?: number;
formats: string[];
paths: Record<string, string>;
}
interface BookStatsOutput {
book_id: number;
word_count: number;
character_count: number;
chapter_count: number;
chunk_count: number;
avg_chunk_size: number;
preprocessed_at: string;
}
interface ManifestEntry {
status: "processed" | "failed" | "skipped";
title?: string;
author?: string;
category?: string;
processed_at?: string;
error?: string;
}
interface Manifest {
version: string;
generated_at: string;
last_updated: string;
total_books: number;
processed: number;
failed: number;
skipped: number;
books: Record<string, ManifestEntry>;
}
interface ProcessingOptions {
category?: string;
bookId?: number;
resume: boolean;
dryRun: boolean;
chunkSize: number;
overlap: number;
parallel: number;
preferFormat: "txt" | "epub";
}
// === HELPERS ===
async function ensureDir(path: string): Promise<void> {
try {
await Deno.mkdir(path, { recursive: true });
} catch (e) {
if (!(e instanceof Deno.errors.AlreadyExists)) {
throw e;
}
}
}
async function fileExists(path: string): Promise<boolean> {
try {
await Deno.stat(path);
return true;
} catch {
return false;
}
}
async function findBookFiles(
bookPath: string
): Promise<Record<string, string>> {
const result: Record<string, string> = {};
const fullPath = `${BOOKS_DIR}/${bookPath}`;
try {
for await (const entry of Deno.readDir(fullPath)) {
if (entry.isFile) {
const ext = entry.name.split(".").pop()?.toLowerCase();
if (ext && ["txt", "epub", "azw3", "mobi", "pdf"].includes(ext)) {
result[ext] = `${fullPath}/${entry.name}`;
}
}
}
} catch {
// Directory doesn't exist or can't be read
}
return result;
}
function countWords(text: string): number {
return text.split(/\s+/).filter((w) => w.length > 0).length;
}
// === MANIFEST MANAGEMENT ===
async function loadManifest(): Promise<Manifest> {
const manifestPath = `${PREPROCESSED_DIR}/_manifest.json`;
if (await fileExists(manifestPath)) {
const content = await Deno.readTextFile(manifestPath);
return JSON.parse(content);
}
return {
version: "1.0.0",
generated_at: new Date().toISOString(),
last_updated: new Date().toISOString(),
total_books: 0,
processed: 0,
failed: 0,
skipped: 0,
books: {},
};
}
async function saveManifest(manifest: Manifest): Promise<void> {
manifest.last_updated = new Date().toISOString();
// Recalculate counts
manifest.processed = Object.values(manifest.books).filter(
(b) => b.status === "processed"
).length;
manifest.failed = Object.values(manifest.books).filter(
(b) => b.status === "failed"
).length;
manifest.skipped = Object.values(manifest.books).filter(
(b) => b.status === "skipped"
).length;
await ensureDir(PREPROCESSED_DIR);
await Deno.writeTextFile(
`${PREPROCESSED_DIR}/_manifest.json`,
JSON.stringify(manifest, null, 2)
);
}
// === PROCESSING ===
async function processBook(
classification: ClassificationEntry,
calibreBook: CalibreBook,
options: ProcessingOptions
): Promise<{ success: boolean; error?: string }> {
const bookDir = `${PREPROCESSED_DIR}/${classification.book_id}`;
// Find text file
const files = await findBookFiles(calibreBook.path);
const textFile =
files[options.preferFormat] || files.txt || files.epub;
if (!textFile) {
return {
success: false,
error: `No ${options.preferFormat}/txt/epub file found`,
};
}
// Parse the book
let parsed: ParsedBook;
try {
parsed = await parseBook(textFile, {
chunkSize: options.chunkSize,
overlap: options.overlap,
});
} catch (e) {
return {
success: false,
error: `Parse error: ${e instanceof Error ? e.message : String(e)}`,
};
}
// Ensure output directory
await ensureDir(bookDir);
// Build metadata output (merging Calibre + classification data)
const metadata: BookMetadataOutput = {
book_id: classification.book_id,
title: calibreBook.title,
author: calibreBook.author,
category: classification.category,
confidence: classification.confidence,
tags: calibreBook.tags,
description: calibreBook.description,
isbn: calibreBook.isbn,
publisher: calibreBook.publisher,
series: calibreBook.series,
series_index: calibreBook.series_index,
formats: Object.keys(files),
paths: files,
};
// Calculate stats
const fullText = parsed.chunks.map((c) => c.text).join("");
const stats: BookStatsOutput = {
book_id: classification.book_id,
word_count: countWords(fullText),
character_count: parsed.metadata.total_characters,
chapter_count: parsed.chapters.length,
chunk_count: parsed.chunks.length,
avg_chunk_size: Math.round(
parsed.metadata.total_characters / Math.max(parsed.chunks.length, 1)
),
preprocessed_at: new Date().toISOString(),
};
// Build parsed output with book_id added
const parsedOutput = {
book_id: classification.book_id,
source_file: textFile,
metadata: parsed.metadata,
chapters: parsed.chapters,
chunks: parsed.chunks,
total_characters: parsed.metadata.total_characters,
};
// Write output files
await Deno.writeTextFile(
`${bookDir}/parsed.json`,
JSON.stringify(parsedOutput, null, 2)
);
await Deno.writeTextFile(
`${bookDir}/metadata.json`,
JSON.stringify(metadata, null, 2)
);
await Deno.writeTextFile(
`${bookDir}/stats.json`,
JSON.stringify(stats, null, 2)
);
return { success: true };
}
// === MAIN ===
async function main(): Promise<void> {
const args = Deno.args;
// Help
if (args.includes("--help") || args.includes("-h")) {
console.log(`bulk-preprocess.ts - Bulk Ebook Preprocessing
Usage:
deno run -A bulk-preprocess.ts [options]
Options:
--category <cat> Filter by category (fiction, other_nonfiction, etc.)
--book-id <id> Process single book by Calibre ID
--resume Skip books already in preprocessed/
--dry-run Show what would be processed, don't execute
--chunk-size <n> Characters per chunk (default: ${DEFAULT_CHUNK_SIZE})
--overlap <n> Overlap between chunks (default: ${DEFAULT_OVERLAP})
--parallel <n> Concurrent books to process (default: ${DEFAULT_PARALLEL})
--format <fmt> Prefer txt or epub (default: txt)
Examples:
deno run -A bulk-preprocess.ts --category other_nonfiction --dry-run
deno run -A bulk-preprocess.ts --book-id 74
deno run -A bulk-preprocess.ts --resume --parallel 8
`);
Deno.exit(0);
}
// Parse options
const options: ProcessingOptions = {
category: undefined,
bookId: undefined,
resume: args.includes("--resume"),
dryRun: args.includes("--dry-run"),
chunkSize: DEFAULT_CHUNK_SIZE,
overlap: DEFAULT_OVERLAP,
parallel: DEFAULT_PARALLEL,
preferFormat: "txt",
};
const categoryIdx = args.indexOf("--category");
if (categoryIdx !== -1) {
options.category = args[categoryIdx + 1];
}
const bookIdIdx = args.indexOf("--book-id");
if (bookIdIdx !== -1) {
options.bookId = parseInt(args[bookIdIdx + 1]);
}
const chunkSizeIdx = args.indexOf("--chunk-size");
if (chunkSizeIdx !== -1) {
options.chunkSize = parseInt(args[chunkSizeIdx + 1]);
}
const overlapIdx = args.indexOf("--overlap");
if (overlapIdx !== -1) {
options.overlap = parseInt(args[overlapIdx + 1]);
}
const parallelIdx = args.indexOf("--parallel");
if (parallelIdx !== -1) {
options.parallel = parseInt(args[parallelIdx + 1]);
}
const formatIdx = args.indexOf("--format");
if (formatIdx !== -1) {
options.preferFormat = args[formatIdx + 1] as "txt" | "epub";
}
// Load classification index
console.log("Loading classification index...");
const indexContent = await Deno.readTextFile(CLASSIFICATION_INDEX);
const classificationIndex: ClassificationIndex = JSON.parse(indexContent);
// Filter books
let books = classificationIndex.books;
if (options.bookId !== undefined) {
books = books.filter((b) => b.book_id === options.bookId);
}
if (options.category) {
books = books.filter((b) => b.category === options.category);
}
console.log(`Found ${books.length} books to process`);
if (books.length === 0) {
console.log("No books match the criteria");
Deno.exit(0);
}
// Load manifest
const manifest = await loadManifest();
manifest.total_books = classificationIndex.statistics.total_books;
// Filter out already processed if --resume
if (options.resume) {
const before = books.length;
books = books.filter((b) => {
const existing = manifest.books[String(b.book_id)];
return !existing || existing.status !== "processed";
});
console.log(`Resuming: ${before - books.length} already processed, ${books.length} remaining`);
}
// Dry run - just show what would be processed
if (options.dryRun) {
console.log("\n=== DRY RUN ===");
console.log(`Would process ${books.length} books:`);
for (const book of books.slice(0, 20)) {
console.log(` [${book.book_id}] ${book.title} by ${book.author} (${book.category})`);
}
if (books.length > 20) {
console.log(` ... and ${books.length - 20} more`);
}
console.log("\nCategory breakdown:");
const byCategory: Record<string, number> = {};
for (const book of books) {
byCategory[book.category] = (byCategory[book.category] || 0) + 1;
}
for (const [cat, count] of Object.entries(byCategory)) {
console.log(` ${cat}: ${count}`);
}
Deno.exit(0);
}
// Open Calibre DB
console.log("Opening Calibre database...");
const db = openCalibreDb(CALIBRE_DB);
// Process books
await ensureDir(PREPROCESSED_DIR);
let processed = 0;
let failed = 0;
let skipped = 0;
const startTime = Date.now();
// Process in batches for controlled concurrency
for (let i = 0; i < books.length; i += options.parallel) {
const batch = books.slice(i, i + options.parallel);
const results = await Promise.all(
batch.map(async (classification) => {
const bookId = classification.book_id;
const calibreBook = getBookById(db, bookId);
if (!calibreBook) {
return {
bookId,
classification,
success: false,
error: "Book not found in Calibre DB",
};
}
const result = await processBook(classification, calibreBook, options);
return { bookId, classification, calibreBook, ...result };
})
);
// Update manifest and print progress
for (const result of results) {
const entry: ManifestEntry = {
status: result.success ? "processed" : "failed",
title: result.classification.title,
author: result.classification.author,
category: result.classification.category,
};
if (result.success) {
entry.processed_at = new Date().toISOString();
processed++;
} else {
entry.error = result.error;
failed++;
}
manifest.books[String(result.bookId)] = entry;
}
// Progress update
const elapsed = (Date.now() - startTime) / 1000;
const rate = (processed + failed) / elapsed;
const remaining = books.length - (processed + failed + skipped);
const eta = remaining / rate;
console.log(
`Progress: ${processed + failed}/${books.length} ` +
`(${processed} ok, ${failed} failed) ` +
`[${rate.toFixed(1)}/s, ETA: ${Math.round(eta)}s]`
);
// Save manifest periodically
if ((i + options.parallel) % 50 === 0 || i + options.parallel >= books.length) {
await saveManifest(manifest);
}
}
// Final manifest save
await saveManifest(manifest);
// Close database
closeCalibreDb(db);
// Summary
const elapsed = (Date.now() - startTime) / 1000;
console.log("\n=== COMPLETE ===");
console.log(`Processed: ${processed}`);
console.log(`Failed: ${failed}`);
console.log(`Skipped: ${skipped}`);
console.log(`Time: ${elapsed.toFixed(1)}s`);
console.log(`Rate: ${((processed + failed) / elapsed).toFixed(1)} books/s`);
console.log(`\nOutput: ${PREPROCESSED_DIR}/`);
console.log(`Manifest: ${PREPROCESSED_DIR}/_manifest.json`);
}
main();
/**
* calibre-db.ts - Calibre Database Utility Module
*
* Provides functions for reading Calibre's metadata.db SQLite database.
* This module handles all database access for book metadata, tags, authors, etc.
*
* Usage:
* import { openCalibreDb, getBooks, getAllTags } from "./calibre-db.ts";
* const db = openCalibreDb("/path/to/metadata.db");
* const books = getBooks(db);
*/
import { Database } from "jsr:@db/sqlite@0.12";
// === INTERFACES ===
export interface CalibreBook {
id: number;
title: string;
author: string;
path: string;
tags: string[];
description?: string;
publisher?: string;
pubdate?: string;
isbn?: string;
series?: string;
series_index?: number;
rating?: number;
timestamp?: string;
last_modified?: string;
}
export interface CalibreTag {
id: number;
name: string;
count: number;
}
export interface GetBooksOptions {
limit?: number;
offset?: number;
tagFilter?: string;
authorFilter?: string;
titleSearch?: string;
hasTagsOnly?: boolean;
noTagsOnly?: boolean;
}
// === DATABASE CONNECTION ===
export function openCalibreDb(path: string): Database {
return new Database(path, { readonly: true });
}
export function closeCalibreDb(db: Database): void {
db.close();
}
// === HELPER FUNCTIONS ===
function queryAll<T>(db: Database, sql: string, params: unknown[] = []): T[] {
const stmt = db.prepare(sql);
return stmt.all(...params) as T[];
}
function queryOne<T>(db: Database, sql: string, params: unknown[] = []): T | undefined {
const stmt = db.prepare(sql);
return stmt.get(...params) as T | undefined;
}
// === BOOK QUERIES ===
interface BookRow {
id: number;
title: string;
path: string;
pubdate: string | null;
timestamp: string | null;
last_modified: string | null;
series_index: number | null;
authors: string | null;
}
export function getBooks(db: Database, options: GetBooksOptions = {}): CalibreBook[] {
const books: CalibreBook[] = [];
// Base query to get books with author
let query = `
SELECT
b.id,
b.title,
b.path,
b.pubdate,
b.timestamp,
b.last_modified,
b.series_index,
GROUP_CONCAT(DISTINCT a.name) as authors
FROM books b
LEFT JOIN books_authors_link bal ON b.id = bal.book
LEFT JOIN authors a ON bal.author = a.id
`;
const conditions: string[] = [];
const params: unknown[] = [];
// Tag filter
if (options.tagFilter) {
query = `
SELECT
b.id,
b.title,
b.path,
b.pubdate,
b.timestamp,
b.last_modified,
b.series_index,
GROUP_CONCAT(DISTINCT a.name) as authors
FROM books b
LEFT JOIN books_authors_link bal ON b.id = bal.book
LEFT JOIN authors a ON bal.author = a.id
JOIN books_tags_link btl ON b.id = btl.book
JOIN tags t ON btl.tag = t.id
`;
conditions.push("t.name = ?");
params.push(options.tagFilter);
}
// Has tags only
if (options.hasTagsOnly) {
conditions.push("b.id IN (SELECT DISTINCT book FROM books_tags_link)");
}
// No tags only
if (options.noTagsOnly) {
conditions.push("b.id NOT IN (SELECT DISTINCT book FROM books_tags_link)");
}
// Title search
if (options.titleSearch) {
conditions.push("b.title LIKE ?");
params.push(`%${options.titleSearch}%`);
}
// Author filter
if (options.authorFilter) {
conditions.push("a.name LIKE ?");
params.push(`%${options.authorFilter}%`);
}
if (conditions.length > 0) {
query += " WHERE " + conditions.join(" AND ");
}
query += " GROUP BY b.id ORDER BY b.title";
if (options.limit) {
query += ` LIMIT ${options.limit}`;
if (options.offset) {
query += ` OFFSET ${options.offset}`;
}
}
const rows = queryAll<BookRow>(db, query, params);
for (const row of rows) {
const book: CalibreBook = {
id: row.id,
title: row.title,
author: row.authors || "Unknown",
path: row.path,
tags: [],
pubdate: row.pubdate || undefined,
timestamp: row.timestamp || undefined,
last_modified: row.last_modified || undefined,
series_index: row.series_index || undefined,
};
// Get tags for this book
book.tags = getBookTags(db, row.id);
// Get additional metadata
const extras = getBookExtras(db, row.id);
book.description = extras.description;
book.publisher = extras.publisher;
book.isbn = extras.isbn;
book.series = extras.series;
book.rating = extras.rating;
books.push(book);
}
return books;
}
export function getBookById(db: Database, bookId: number): CalibreBook | null {
const row = queryOne<BookRow>(
db,
`
SELECT
b.id,
b.title,
b.path,
b.pubdate,
b.timestamp,
b.last_modified,
b.series_index,
GROUP_CONCAT(DISTINCT a.name) as authors
FROM books b
LEFT JOIN books_authors_link bal ON b.id = bal.book
LEFT JOIN authors a ON bal.author = a.id
WHERE b.id = ?
GROUP BY b.id
`,
[bookId]
);
if (!row) return null;
const book: CalibreBook = {
id: row.id,
title: row.title,
author: row.authors || "Unknown",
path: row.path,
tags: getBookTags(db, row.id),
pubdate: row.pubdate || undefined,
timestamp: row.timestamp || undefined,
last_modified: row.last_modified || undefined,
series_index: row.series_index || undefined,
};
const extras = getBookExtras(db, row.id);
book.description = extras.description;
book.publisher = extras.publisher;
book.isbn = extras.isbn;
book.series = extras.series;
book.rating = extras.rating;
return book;
}
function getBookExtras(
db: Database,
bookId: number
): {
description?: string;
publisher?: string;
isbn?: string;
series?: string;
rating?: number;
} {
const result: {
description?: string;
publisher?: string;
isbn?: string;
series?: string;
rating?: number;
} = {};
// Description from comments table
const comment = queryOne<{ text: string }>(
db,
"SELECT text FROM comments WHERE book = ?",
[bookId]
);
if (comment) {
result.description = comment.text;
}
// Publisher
const publisher = queryOne<{ name: string }>(
db,
`
SELECT p.name FROM publishers p
JOIN books_publishers_link bpl ON p.id = bpl.publisher
WHERE bpl.book = ?
`,
[bookId]
);
if (publisher) {
result.publisher = publisher.name;
}
// ISBN from identifiers
const identifier = queryOne<{ val: string }>(
db,
"SELECT val FROM identifiers WHERE book = ? AND type = 'isbn'",
[bookId]
);
if (identifier) {
result.isbn = identifier.val;
}
// Series
const series = queryOne<{ name: string }>(
db,
`
SELECT s.name FROM series s
JOIN books_series_link bsl ON s.id = bsl.series
WHERE bsl.book = ?
`,
[bookId]
);
if (series) {
result.series = series.name;
}
// Rating
const rating = queryOne<{ rating: number }>(
db,
`
SELECT r.rating FROM ratings r
JOIN books_ratings_link brl ON r.id = brl.rating
WHERE brl.book = ?
`,
[bookId]
);
if (rating) {
result.rating = rating.rating;
}
return result;
}
// === TAG QUERIES ===
export function getBookTags(db: Database, bookId: number): string[] {
const rows = queryAll<{ name: string }>(
db,
`
SELECT t.name FROM tags t
JOIN books_tags_link btl ON t.id = btl.tag
WHERE btl.book = ?
ORDER BY t.name
`,
[bookId]
);
return rows.map((row) => row.name);
}
export function getAllTags(db: Database): CalibreTag[] {
const rows = queryAll<{ id: number; name: string; count: number }>(
db,
`
SELECT t.id, t.name, COUNT(btl.book) as count
FROM tags t
LEFT JOIN books_tags_link btl ON t.id = btl.tag
GROUP BY t.id
ORDER BY count DESC, t.name
`
);
return rows;
}
// === SEARCH ===
export function searchBooks(db: Database, query: string): CalibreBook[] {
return getBooks(db, { titleSearch: query });
}
// === STATISTICS ===
export function getStats(db: Database): {
totalBooks: number;
booksWithTags: number;
booksWithoutTags: number;
totalTags: number;
totalAuthors: number;
} {
const totalBooks =
queryOne<{ count: number }>(db, "SELECT COUNT(*) as count FROM books")?.count || 0;
const booksWithTags =
queryOne<{ count: number }>(db, "SELECT COUNT(DISTINCT book) as count FROM books_tags_link")
?.count || 0;
const totalTags =
queryOne<{ count: number }>(db, "SELECT COUNT(*) as count FROM tags")?.count || 0;
const totalAuthors =
queryOne<{ count: number }>(db, "SELECT COUNT(*) as count FROM authors")?.count || 0;
return {
totalBooks,
booksWithTags,
booksWithoutTags: totalBooks - booksWithTags,
totalTags,
totalAuthors,
};
}
// === CLI ENTRY POINT (for testing) ===
if (import.meta.main) {
const dbPath = Deno.args[0];
if (!dbPath) {
console.error("Usage: deno run --allow-read --allow-ffi calibre-db.ts <metadata.db>");
Deno.exit(1);
}
const db = openCalibreDb(dbPath);
const stats = getStats(db);
console.log("Database Statistics:");
console.log(` Total books: ${stats.totalBooks}`);
console.log(` Books with tags: ${stats.booksWithTags}`);
console.log(` Books without tags: ${stats.booksWithoutTags}`);
console.log(` Total unique tags: ${stats.totalTags}`);
console.log(` Total authors: ${stats.totalAuthors}`);
closeCalibreDb(db);
}
#!/usr/bin/env -S deno run --allow-read --allow-write
/**
* ea-classify.ts - Concept Classification (LLM-Assisted)
*
* Presents extracted concepts to the LLM for type and layer classification.
* This script handles I/O and structure; the LLM provides judgment.
*
* Usage:
* deno run --allow-read --allow-write scripts/ea-classify.ts <concepts.json>
* deno run --allow-read --allow-write scripts/ea-classify.ts concepts.json --output classified.json
*/
// === INTERFACES ===
interface ExtractedConcept {
id: string;
name: string;
exact_quote: string;
context_before?: string;
context_after?: string;
start_position: number;
end_position: number;
chunk_id: string;
chapter_number?: number;
chapter_title?: string;
extraction_notes?: string;
requires_review: boolean;
extraction_date: string;
}
interface ExtractionResult {
source: {
title: string;
author: string;
file_path: string;
};
extraction_date: string;
concepts: ExtractedConcept[];
chunks_processed: number;
total_chunks: number;
}
type ConceptType = "principle" | "mechanism" | "pattern" | "strategy" | "tactic";
interface ClassifiedConcept extends ExtractedConcept {
type: ConceptType;
layer: number; // 0-4
type_confidence: number; // 0-1
layer_confidence: number; // 0-1
classification_notes?: string;
classification_date: string;
}
interface ClassificationResult {
source: {
title: string;
author: string;
file_path: string;
};
classification_date: string;
concepts: ClassifiedConcept[];
type_distribution: Record<ConceptType, number>;
layer_distribution: Record<number, number>;
}
// === CLASSIFICATION PROMPT GENERATION ===
function generateClassificationPrompt(concept: ExtractedConcept, bookTitle: string): string {
return `## Concept Classification Task
**Source:** "${bookTitle}"
**Concept ID:** ${concept.id}
**Chapter:** ${concept.chapter_title || concept.chapter_number || "Unknown"}
### Concept to Classify:
**Name:** ${concept.name}
**Exact Quote:**
"${concept.exact_quote}"
**Context Before:** ${concept.context_before || "(none)"}
**Context After:** ${concept.context_after || "(none)"}
### Classification Instructions:
Classify this concept by TYPE and LAYER.
**TYPES (choose one):**
- **principle** - Foundational truth or axiom (e.g., "Communities form around shared identity")
- **mechanism** - How something works (e.g., "Reciprocity creates social bonds by triggering obligation")
- **pattern** - Recurring structure or framework (e.g., "The community lifecycle: formation, growth, maturation")
- **strategy** - High-level approach (e.g., "Build trust before asking for contribution")
- **tactic** - Specific actionable technique (e.g., "Send welcome emails within 24 hours")
**LAYERS (choose 0-4):**
- **0** - Foundational (universal truths about human nature)
- **1** - Theoretical (domain-specific theory)
- **2** - Strategic (frameworks and approaches)
- **3** - Tactical (specific methods, tool-agnostic)
- **4** - Specific (concrete implementations, named tools)
### Output Format:
\`\`\`
TYPE: [principle|mechanism|pattern|strategy|tactic]
TYPE_CONFIDENCE: [0.0-1.0]
LAYER: [0-4]
LAYER_CONFIDENCE: [0.0-1.0]
NOTES: [Brief explanation of classification reasoning]
\`\`\`
`;
}
// === BATCH CLASSIFICATION PROMPT ===
function generateBatchClassificationPrompt(concepts: ExtractedConcept[], bookTitle: string): string {
let prompt = `## Batch Concept Classification
**Source:** "${bookTitle}"
**Concepts to classify:** ${concepts.length}
### Classification Guide:
**TYPES:**
- principle - Foundational truth ("X is essential for Y")
- mechanism - How it works ("X causes Y by Z")
- pattern - Recurring structure (named frameworks, stages)
- strategy - High-level approach ("Focus on X before Y")
- tactic - Specific action ("Do X to achieve Y")
**LAYERS (0-4):**
- 0: Universal human truths
- 1: Domain-specific theory
- 2: Frameworks and approaches
- 3: Tool-agnostic methods
- 4: Specific implementations
### Concepts:
`;
for (const concept of concepts) {
prompt += `---
**ID:** ${concept.id}
**Name:** ${concept.name}
**Quote:** "${concept.exact_quote.slice(0, 200)}${concept.exact_quote.length > 200 ? "..." : ""}"
`;
}
prompt += `### Output Format (one block per concept):
\`\`\`
ID: [concept-id]
TYPE: [principle|mechanism|pattern|strategy|tactic]
LAYER: [0-4]
NOTES: [Brief reasoning]
\`\`\`
`;
return prompt;
}
// === CLASSIFICATION PARSING ===
function parseClassification(
llmResponse: string,
concept: ExtractedConcept
): ClassifiedConcept {
// Parse single concept classification
const typeMatch = llmResponse.match(/TYPE:\s*(principle|mechanism|pattern|strategy|tactic)/i);
const typeConfMatch = llmResponse.match(/TYPE_CONFIDENCE:\s*([\d.]+)/i);
const layerMatch = llmResponse.match(/LAYER:\s*(\d)/i);
const layerConfMatch = llmResponse.match(/LAYER_CONFIDENCE:\s*([\d.]+)/i);
const notesMatch = llmResponse.match(/NOTES:\s*(.+?)(?:\n|$)/is);
const type = (typeMatch?.[1]?.toLowerCase() || "principle") as ConceptType;
const typeConfidence = typeConfMatch ? parseFloat(typeConfMatch[1]) : 0.7;
const layer = layerMatch ? parseInt(layerMatch[1]) : 2;
const layerConfidence = layerConfMatch ? parseFloat(layerConfMatch[1]) : 0.7;
const notes = notesMatch?.[1]?.trim();
return {
...concept,
type,
layer,
type_confidence: typeConfidence,
layer_confidence: layerConfidence,
classification_notes: notes,
classification_date: new Date().toISOString(),
};
}
function parseBatchClassifications(
llmResponse: string,
concepts: ExtractedConcept[]
): ClassifiedConcept[] {
const classified: ClassifiedConcept[] = [];
const conceptMap = new Map(concepts.map((c) => [c.id, c]));
// Parse batch response - look for ID blocks
const blockPattern = /ID:\s*(concept-\d+)\s*\nTYPE:\s*(principle|mechanism|pattern|strategy|tactic)\s*\nLAYER:\s*(\d)\s*\nNOTES:\s*(.+?)(?=\n\nID:|$)/gis;
let match;
while ((match = blockPattern.exec(llmResponse)) !== null) {
const id = match[1];
const type = match[2].toLowerCase() as ConceptType;
const layer = parseInt(match[3]);
const notes = match[4].trim();
const concept = conceptMap.get(id);
if (concept) {
classified.push({
...concept,
type,
layer,
type_confidence: 0.8, // Batch mode uses default confidence
layer_confidence: 0.8,
classification_notes: notes,
classification_date: new Date().toISOString(),
});
conceptMap.delete(id);
}
}
// Handle any unclassified concepts
for (const [, concept] of conceptMap) {
classified.push({
...concept,
type: "principle",
layer: 2,
type_confidence: 0.3,
layer_confidence: 0.3,
classification_notes: "Not classified - using defaults",
classification_date: new Date().toISOString(),
});
}
return classified;
}
// === MAIN ===
async function main(): Promise<void> {
const args = Deno.args;
// Help
if (args.includes("--help") || args.includes("-h") || args.length === 0) {
console.log(`ea-classify.ts - Concept Classification (LLM-Assisted)
Usage:
deno run --allow-read --allow-write scripts/ea-classify.ts <concepts.json> [options]
Arguments:
concepts.json Output from ea-extract.ts
Options:
--output <file> Write results to file
--batch Output single batch prompt (faster, less accurate)
--concept-id <id> Classify only specific concept
--interactive Process concepts with manual LLM input
Examples:
# Generate classification prompts
deno run --allow-read scripts/ea-classify.ts concepts.json > prompts.txt
# Batch mode (single prompt for all)
deno run --allow-read scripts/ea-classify.ts concepts.json --batch
# Interactive mode
deno run --allow-read --allow-write scripts/ea-classify.ts concepts.json --interactive --output classified.json
`);
Deno.exit(0);
}
// Parse arguments
const outputIdx = args.indexOf("--output");
const outputFile = outputIdx !== -1 ? args[outputIdx + 1] : null;
const conceptIdIdx = args.indexOf("--concept-id");
const specificConceptId = conceptIdIdx !== -1 ? args[conceptIdIdx + 1] : null;
const batch = args.includes("--batch");
const interactive = args.includes("--interactive");
// Find input file
const skipIndices = new Set<number>();
[outputIdx, conceptIdIdx].forEach((idx) => {
if (idx !== -1) {
skipIndices.add(idx);
skipIndices.add(idx + 1);
}
});
let inputFile: string | null = null;
for (let i = 0; i < args.length; i++) {
if (!args[i].startsWith("--") && !skipIndices.has(i)) {
inputFile = args[i];
break;
}
}
if (!inputFile) {
console.error("Error: No input file specified");
Deno.exit(1);
}
// Load extracted concepts
let extracted: ExtractionResult;
try {
const content = await Deno.readTextFile(inputFile);
extracted = JSON.parse(content);
} catch (e) {
console.error(`Error reading input file: ${e}`);
Deno.exit(1);
}
// Filter concepts
let conceptsToClassify = extracted.concepts;
if (specificConceptId) {
conceptsToClassify = conceptsToClassify.filter((c) => c.id === specificConceptId);
if (conceptsToClassify.length === 0) {
console.error(`Concept not found: ${specificConceptId}`);
Deno.exit(1);
}
}
// Process
if (batch) {
// Batch mode - single prompt for all concepts
console.log(generateBatchClassificationPrompt(conceptsToClassify, extracted.source.title));
} else if (interactive) {
// Interactive mode
const classifiedConcepts: ClassifiedConcept[] = [];
console.log("Interactive classification mode.");
console.log("For each concept, paste LLM response and type END when done.\n");
for (const concept of conceptsToClassify) {
console.log("=".repeat(80));
console.log(generateClassificationPrompt(concept, extracted.source.title));
console.log("=".repeat(80));
console.log("\nPaste LLM response (type END on new line when done):");
// Read response
const lines: string[] = [];
const decoder = new TextDecoder();
const buf = new Uint8Array(1024);
while (true) {
const n = await Deno.stdin.read(buf);
if (n === null) break;
const text = decoder.decode(buf.subarray(0, n));
lines.push(text);
if (text.trim().endsWith("END")) {
break;
}
}
const response = lines.join("");
const classified = parseClassification(response, concept);
classifiedConcepts.push(classified);
console.log(`Classified: ${concept.name} as ${classified.type} (Layer ${classified.layer})\n`);
}
// Build result
const typeDistribution: Record<ConceptType, number> = {
principle: 0,
mechanism: 0,
pattern: 0,
strategy: 0,
tactic: 0,
};
const layerDistribution: Record<number, number> = { 0: 0, 1: 0, 2: 0, 3: 0, 4: 0 };
for (const c of classifiedConcepts) {
typeDistribution[c.type]++;
layerDistribution[c.layer]++;
}
const result: ClassificationResult = {
source: extracted.source,
classification_date: new Date().toISOString(),
concepts: classifiedConcepts,
type_distribution: typeDistribution,
layer_distribution: layerDistribution,
};
const output = JSON.stringify(result, null, 2);
if (outputFile) {
await Deno.writeTextFile(outputFile, output);
console.log(`\nClassification results written to: ${outputFile}`);
} else {
console.log("\n" + output);
}
} else {
// Default: output individual prompts
console.log(`# Classification Prompts for: ${extracted.source.title}`);
console.log(`# Concepts: ${conceptsToClassify.length}\n`);
for (const concept of conceptsToClassify) {
console.log("=".repeat(80));
console.log(generateClassificationPrompt(concept, extracted.source.title));
console.log("");
}
}
}
main();
#!/usr/bin/env -S deno run --allow-read --allow-write
/**
* kb-generate-index.ts - Knowledge Base Entity Index Generator
*
* Scans the knowledge base directory, extracts metadata from entity files,
* and generates a searchable _entities.json index for entity resolution.
*
* Usage:
* deno run --allow-read --allow-write kb-generate-index.ts [knowledge-dir]
* deno run --allow-read --allow-write kb-generate-index.ts /path/to/knowledge
*
* Output:
* Creates/updates _entities.json in the knowledge directory root
*/
import { walk } from "https://deno.land/std@0.208.0/fs/walk.ts";
import { parse as parsePath } from "https://deno.land/std@0.208.0/path/mod.ts";
interface EntityRecord {
name: string;
path: string;
domain: string;
type: string;
status: string;
aliases: string[];
lastUpdated: string;
}
interface EntityIndex {
generated: string;
entityCount: number;
domains: Record<string, number>;
entities: EntityRecord[];
}
/**
* Parse entity metadata from markdown file content
*/
function parseEntityFile(content: string, filePath: string): EntityRecord | null {
const lines = content.split("\n");
// Extract name from H1
const nameMatch = lines[0]?.match(/^#\s+(.+)$/);
if (!nameMatch) return null;
const name = nameMatch[1].trim();
// Extract metadata fields
let type = "";
let status = "";
let aliases: string[] = [];
let lastUpdated = "";
for (const line of lines.slice(1, 20)) { // Check first 20 lines for metadata
const typeMatch = line.match(/^\*\*Type:\*\*\s*(.+)$/);
if (typeMatch) type = typeMatch[1].trim();
const statusMatch = line.match(/^\*\*Status:\*\*\s*(.+)$/);
if (statusMatch) status = statusMatch[1].trim();
const aliasMatch = line.match(/^\*\*Aliases:\*\*\s*(.+)$/);
if (aliasMatch) {
aliases = aliasMatch[1].split(",").map(a => a.trim()).filter(a => a);
}
const dateMatch = line.match(/^\*\*Last Updated:\*\*\s*(.+)$/);
if (dateMatch) lastUpdated = dateMatch[1].trim();
}
// Extract domain and type from path
// e.g., "nonfiction/frameworks/kind-vs-wicked.md" -> domain: "nonfiction", type: "frameworks"
const pathParts = filePath.split("/");
const knowledgeIdx = pathParts.findIndex(p => p === "knowledge");
const domain = knowledgeIdx >= 0 && pathParts[knowledgeIdx + 1]
? pathParts[knowledgeIdx + 1]
: "unknown";
// Get relative path from knowledge directory
const relativePath = knowledgeIdx >= 0
? pathParts.slice(knowledgeIdx + 1).join("/")
: filePath;
return {
name,
path: relativePath,
domain,
type: type || "unknown",
status: status || "unknown",
aliases,
lastUpdated,
};
}
/**
* Scan knowledge directory and build entity index
*/
async function generateIndex(knowledgeDir: string): Promise<EntityIndex> {
const entities: EntityRecord[] = [];
const domains: Record<string, number> = {};
// Walk the knowledge directory
for await (const entry of walk(knowledgeDir, {
exts: [".md"],
skip: [/WORKFLOW\.md$/, /_entities\.json$/], // Skip workflow and JSON index
})) {
if (!entry.isFile) continue;
// Skip files directly in knowledge root (only process domain subdirectories)
const relativePath = entry.path.replace(knowledgeDir + "/", "");
if (!relativePath.includes("/")) continue;
// Skip helper files (those starting with underscore)
// These include: _index.md, _quotes.md, _inventory.md, etc.
const fileName = parsePath(entry.path).base;
if (fileName.startsWith("_")) continue;
try {
const content = await Deno.readTextFile(entry.path);
const entity = parseEntityFile(content, entry.path);
if (entity) {
entities.push(entity);
domains[entity.domain] = (domains[entity.domain] || 0) + 1;
}
} catch (err) {
console.error(`Error processing ${entry.path}:`, err);
}
}
// Sort entities by name
entities.sort((a, b) => a.name.localeCompare(b.name));
return {
generated: new Date().toISOString(),
entityCount: entities.length,
domains,
entities,
};
}
/**
* Main function
*/
async function main() {
const knowledgeDir = Deno.args[0] || "./knowledge";
console.log(`Scanning knowledge base: ${knowledgeDir}`);
const index = await generateIndex(knowledgeDir);
console.log(`Found ${index.entityCount} entities across domains:`, index.domains);
// Write index file
const outputPath = `${knowledgeDir}/_entities.json`;
await Deno.writeTextFile(outputPath, JSON.stringify(index, null, 2));
console.log(`Index written to: ${outputPath}`);
// Also output a summary
console.log("\nEntities indexed:");
for (const entity of index.entities) {
const aliasCount = entity.aliases.length;
const aliasNote = aliasCount > 0 ? ` (+${aliasCount} aliases)` : "";
console.log(` - ${entity.name} [${entity.type}]${aliasNote}`);
}
}
main();
Related skills
How it compares
Pick ebook-analysis for citation-traceable thematic extraction from books; pick generic summarization skills when provenance and entity taxonomy are not required.
FAQ
What extraction modes does ebook-analysis support?
ebook-analysis provides Concept Extraction for ideas classified by abstraction level and Entity Extraction for named studies, researchers, frameworks, and anecdotes, both requiring citation traceability to exact sources.
Why does ebook-analysis require citations?
ebook-analysis treats citation traceability as non-negotiable, preferring smaller extractions with full provenance over larger summaries that cannot be traced back to specific ebook or PDF passages.