
Paper Workbench
- 29 installs
- 16 repo stars
- Updated August 3, 2026
- bahayonghang/my-claude-code-settings
paper-workbench is a Claude Code skill that normalizes papers into a paper-record and routes them into scan, deep-read, synthesis, and review modes for literature work.
About
paper-workbench is a literature workbench that normalizes papers from arxiv, DOI, PDF, or landing pages into a machine-readable paper-record, then routes them into reading modes like scan, deep-read, card, synthesis, and review. A researcher uses it to skim, deeply read, compare, or build a literature review across many papers. It keeps normalization separate from analysis and never invents page numbers or quotes.
- Single entrypoint for paper intake: scan, deep-read, card, synthesis, review, plus json/interpret/xray modes
- Normalizes arxiv, DOI, PDF, and landing-page sources into a reusable paper-record via normalize_paper.py
- Drives multi-paper synthesis and literature-review outlines against a researcher-profile
Paper Workbench by the numbers
- 29 all-time installs (skills.sh)
- Ranked #939 of 1,879 Documentation skills by installs in the Skillselion catalog
- Data as of Aug 4, 2026 (Skillselion catalog sync)
paper-workbench capabilities & compatibility
- Capabilities
- paper interpretation · paper interpretation flow · research
- Use cases
- research · documentation
What paper-workbench says it does
Unified entrypoint for paper intake, strategic reading, multi-paper synthesis, and review construction.
Do not use this skill when the primary job is to implement a paper from its methods into working code. That implementation work is out of scope for this skill.
npx skills add https://github.com/bahayonghang/my-claude-code-settings --skill paper-workbenchAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 29 |
|---|---|
| repo stars | ★ 16 |
| Last updated | August 3, 2026 |
| Repository | bahayonghang/my-claude-code-settings ↗ |
What it does
Skim, deep-read, compare, or synthesize a set of papers and build a literature-review outline or gap map.
Who is it for?
Reading, comparing, and synthesizing multiple papers into a literature review or gap map.
Skip if: Implementing a paper's method as working code, which it says is out of scope.
When should I use this skill?
When you want to skim, deep-read, card, compare, synthesize, or build a literature review from papers, DOIs, PDFs, or existing paper JSON.
What you get
Normalized paper-record artifacts plus mode outputs like deep reads, cross-paper synthesis, and review outlines.
- normalized paper-record JSON
- literature-synthesis artifact
- review-outline artifact
By the numbers
- 8 routing modes (scan, deep-read, card, synthesis, review, json, interpret, xray)
- bundled scripts: normalize_paper.py, workbench_io.py, xray_io.py
Files
Paper Workbench
Unified entrypoint for paper intake, strategic reading, multi-paper synthesis, and review construction.
Keep paper-record as the normalization layer. Do not merge high-level analysis back into the normalized record.
In thepythoncommands below,<skill-dir>is this skill's base directory,
announced when the skill loads. Substitute that literal path; it is not an
environment variable. Bundled scripts self-locate, so only the path needs to
resolve.
When to use
Use this skill when the job is to:
- read one paper quickly
- deeply deconstruct one paper
- compare or synthesize multiple papers
- build a review outline or gap map
- normalize paper sources into reusable machine-readable artifacts
Do not use this skill when the primary job is to implement a paper from its methods into working code. That implementation work is out of scope for this skill.
Public interfaces
paper-record— normalized single-paper factsresearcher-profile— user research anchorpaper-deep-read— single-paper strategic analysis artifactliterature-synthesis— cross-paper integration artifactreview-outline— literature-review planning artifact
Accepted inputs
- arXiv IDs and arXiv URLs
- AlphaXiv URLs
- DOI strings or
doi.org/...URLs - local academic PDFs or text files
- remote PDF URLs
- paper landing pages that expose a PDF
- existing
paper-recordJSON - existing
researcher-profile,paper-deep-read,literature-synthesis, or
review-outline JSON
Routing workflow
1. Resolve the input class from $ARGUMENTS, the latest user message, or a pasted JSON artifact. 2. If the request is paper-level and not already normalized, run scripts/normalize_paper.py first. 3. Determine the mode from explicit user intent or the defaulting rules below. 4. If the chosen mode is profile-sensitive, load the supplied researcher-profile or collect only the missing fields. 5. Produce the requested mode output. 6. Persist artifacts only when the user asked to save them.
Mode quick guide
Single-paper modes
scan- Use for “先快速扫一下”, “预判”, or fast worth-reading decisions
deep-read- Use for “精读这篇”, “深度阅读”, “解构这篇”
card- Use for “只做卡片”
interpret- Compatibility path for a lightweight explanation
xray- Compatibility path for compact critique
json- Return the normalized
paper-record
Cross-paper modes
synthesis- Use for “整合这几篇”, “对比分析”, “找研究空白”
review- Use for “搭综述框架”, “写这一段”
Defaulting rules
- If the user explicitly asks for a machine-readable or saved schema artifact,
default to json
- If the user provides a single paper and asks to read or analyze it without a
more specific mode, default to scan
- If the user provides 3 or more papers and asks for integration, default to
synthesis
- If the user provides exactly 2 papers and asks for integration, run a
comparison-oriented synthesis and mark any gap mapping as provisional
Normalize first
For any paper-like input, run:
python "<skill-dir>/scripts/normalize_paper.py" \
--source "<paper-source>" \
--lang "<lang>" \
--fulltext "<auto|prefer|never>"Use --save only when the user asked to persist the normalized JSON.
Profile workflow
Before deep-read, card, synthesis, or review, prefer a researcher-profile.
If missing, collect only these fields:
research_fieldcore_questionthesis(optional)target_tierstage
If the user clearly wants no back-and-forth, proceed with a generic profile-light analysis and explicitly mark that personalization is limited.
If the user wants persistence, create or update the profile with:
python "<skill-dir>/scripts/workbench_io.py" init-profile \
--path "<profile-path>" \
--research-field "<field>" \
--core-question "<question>" \
--thesis "<optional-thesis>" \
--target-tier "<target-tier>" \
--stage "<stage>"Artifact persistence
When the user asks to save a deep read, synthesis, or review plan, write a JSON artifact plus an optional Markdown or Org sidecar:
python "<skill-dir>/scripts/workbench_io.py" save-artifact \
--workspace "<workspace-dir>" \
--artifact-type "<paper-deep-read|literature-synthesis|review-outline>" \
--title "<artifact-title>" \
--payload-file "<json-payload-file>" \
--profile-path "<optional-profile-path>" \
--source-record "<path-to-paper-record>" \
--sidecar-file "<optional-md-or-org>"Output rules
- Separate
作者观点from系统分析 - Never invent page numbers, quotations, or empirical details
- If a requested quote or page anchor is missing, use
[信息待核实] synthesisandreviewmust integrate arguments across papers rather than
serially summarizing each paper
reviewparagraphs must use PEEL as a micro-argument structure, not a
citation list
- If the input evidence is too thin for the requested mode, downgrade the claim
strength instead of pretending full coverage
Edge cases
- Mixed raw sources + existing JSON artifacts:
- normalize raw sources first, then merge at the artifact layer
- More than one paper but user asks for
deep-read: - either choose the clearly primary paper or ask which one to focus on
- DOI metadata only and no reachable full text:
- return the strongest metadata available and mark missing full-text facts
References
references/routing.md— source classification and routing logicreferences/schema.md— canonicalpaper-recordcontractreferences/artifacts.md—researcher-profileand higher-level artifactsreferences/migration.md— compatibility and alias mappingreferences/ANALYSIS_FRAMEWORK.md— x-ray five-dimension critique frameworkreferences/template-paper.org— Org sidecar template for deep-read / interpret outputreferences/template-xray.org— Org sidecar template for x-ray critique outputreferences/modes/json.md— machine-readable output rulesreferences/modes/interpret.md— lightweight explanation pathreferences/modes/xray.md— compact critique pathreferences/modes/scan.md— single-paper quick triagereferences/modes/deep-read.md— full single-paper deconstructionreferences/modes/card.md— literature card onlyreferences/modes/synthesis.md— cross-paper integrationreferences/modes/review.md— literature-review planning and writing
{
"skill_name": "paper-workbench",
"evals": [
{
"id": 1,
"prompt": "我刚上传了一篇 PDF,先快速扫一下值不值得精读,重点看它和我的研究问题“平台化协作如何改变学术写作劳动”有没有关系。",
"expected_output": "Runs scan mode, gives the five quick-triage fields, and evaluates relevance against the research question.",
"files": [],
"assertions": [
"Normalizes the PDF into a paper-record first",
"Returns the scan quick-triage fields",
"Assesses relevance against the stated research question"
]
},
{
"id": 2,
"prompt": "精读这篇文献。我做科技社会学,核心问题是平台化协作如何改变学术写作劳动,现在在文献梳理阶段,目标投 SSCI Q1。",
"expected_output": "Runs deep-read mode, includes quick scan, academic deconstruction, strategic relevance, and a reusable literature card.",
"files": [],
"assertions": [
"Uses the supplied researcher-profile fields (field, question, stage, target tier)",
"Separates 作者观点 from 系统分析",
"Produces a reusable literature card"
]
},
{
"id": 3,
"prompt": "只做卡片,不要长报告。我要一个后面写综述能直接复用的版本。",
"expected_output": "Runs card mode only and returns a concise literature card plus a short critical summary.",
"files": [],
"assertions": [
"Returns only a literature card, not a full report",
"Card is reusable for a later review"
]
},
{
"id": 4,
"prompt": "我这里已经有四篇 related papers 了,帮我整合这几篇,顺便找研究空白,最好指出我的题目更适合补哪类空白。",
"expected_output": "Runs synthesis mode with concept map, debate spectrum, method matrix, gap map, and relation network anchored to the user's research question.",
"files": [],
"assertions": [
"Integrates arguments across papers instead of summarizing each serially",
"Produces a gap map anchored to the user's question"
]
},
{
"id": 5,
"prompt": "基于刚才那组整合结果,帮我搭一个综述框架,再示范写一段,不要逐篇罗列文献。",
"expected_output": "Runs review mode, recommends a narrative strategy, outputs a structured outline, and drafts a PEEL-style paragraph.",
"files": [],
"assertions": [
"Outputs a structured review outline",
"Drafts a PEEL-style paragraph, not a citation list"
]
},
{
"id": 6,
"prompt": "用我的 Zotero 库,像导师一样逐图带我读这篇论文,每张图停下来等我。",
"expected_output": "Routing-negative: Zotero-first interactive per-figure tutoring should route to literature-mentor, not paper-workbench.",
"files": [],
"assertions": [
"Does NOT run paper-workbench's profile-driven deep-read",
"Defers to literature-mentor for the interactive tutor read"
]
},
{
"id": 7,
"prompt": "帮我调研 2026 年 RAG 的最新进展,要带网页来源的综述。",
"expected_output": "Routing-negative: current-topic web research with citations should route to deep-research-pro, not paper-workbench.",
"files": [],
"assertions": [
"Does NOT treat this as paper intake/normalization",
"Defers to deep-research-pro for cited web research"
]
}
]
}
X-Ray Paper Analysis Framework
认知提取算法
去噪
- 忽略背景介绍、客套话和通用的已知知识
- 跳过冗长的 Related Work(除非有关键对比)
- 过滤掉"为了发表而写"的填充内容
提取
- 锁定论文的核心贡献(Delta)
- 识别作者的"灵光一闪"时刻
- 找出决定成败的 1-2 个关键操作
批判
- 寻找逻辑漏洞或边界条件
- 识别隐形假设
- 标记未解决的问题
五维分析框架
【1. 核心痛点】
- 一句话定义: 这篇论文试图解决什么具体的、困难的问题?
- 前人困境: 在它之前,为什么别人解决不了?
【2. 解题机制】
- 核心直觉: 作者那个"灵光一闪"的想法(用最直白的语言)
- 关键步骤: 只列决定成败的 1-2 个"神来之笔"
【3. 创新增量】
- 对比 SOTA: 相比当前最佳,具体提升在哪?
- 新拼图: 为人类知识库增加了哪块具体的新拼图?
【4. 批判性边界】
- 隐形假设: 作者在什么条件下才能成功?
- 未解之谜: 论文没解决什么?带来了什么新问题?
【5. 一言以蔽之】
- 餐巾纸图: 如果要画一个图,画什么?
- 餐巾纸公式: 如果只能写一句公式,写什么?
输出质量标准
- 高密度: 使用列表和关键词,不写长段落
- 直白: 用最简单的语言解释复杂概念
- 批判: 必须指出至少一个隐形假设或未解问题
- ASCII Art: 仅用纯 ASCII 基础符号(+, -, |, >, <, /, \, *, =, .)
- 餐巾纸图/公式: 必须一眼能懂
Workbench artifacts
paper-workbench keeps paper-record as the normalization contract and uses separate artifacts for higher-level analysis.
researcher-profile
Use this as the strategic anchor for deep-read, synthesis, and review.
{
"artifact_type": "researcher-profile",
"version": "1.0",
"created_at": "...",
"updated_at": "...",
"research_field": "科技社会学",
"core_question": "平台化协作如何改变学术写作劳动?",
"thesis": "可选",
"target_tier": "SSCI Q1 / CSSCI / 顶会",
"stage": "选题构思 / 文献梳理 / 研究设计 / 初稿撰写"
}If the user has not given these facts, ask only for the missing fields before starting a profile-sensitive mode.
paper-deep-read
Single-paper artifact generated by deep-read or card.
{
"artifact_type": "paper-deep-read",
"version": "1.0",
"created_at": "...",
"updated_at": "...",
"title": "...",
"profile_path": "optional",
"source_records": ["path/to/paper-record.json"],
"payload": {
"quick_scan": {},
"academic_deconstruction": {},
"strategic_relevance": {},
"literature_card": {}
}
}literature-synthesis
Cross-paper artifact produced from 3+ papers or 3+ paper-deep-read artifacts. When there are only 2 inputs, downgrade to comparison mode and mark the output as limited.
{
"artifact_type": "literature-synthesis",
"version": "1.0",
"created_at": "...",
"updated_at": "...",
"title": "...",
"profile_path": "optional",
"source_records": ["paper-a.json", "paper-b.json", "paper-c.json"],
"payload": {
"concept_map": {},
"debate_spectrum": {},
"method_matrix": {},
"gap_map": {},
"relation_network": {}
}
}review-outline
Review-planning artifact produced from a literature-synthesis artifact, or from 3+ paper-deep-read artifacts when no synthesis artifact exists yet.
{
"artifact_type": "review-outline",
"version": "1.0",
"created_at": "...",
"updated_at": "...",
"title": "...",
"profile_path": "optional",
"source_records": ["synthesis.json"],
"payload": {
"narrative_strategy": {},
"outline": [],
"paragraph_engine": {}
}
}Workspace persistence
Use scripts/workbench_io.py when the user asks to save or reuse workbench state.
init-profile— create or update aresearcher-profileJSON fileresolve-artifact— compute the next timestamped save pathsave-artifact— write the JSON artifact and optionally copy a Markdown or
Org sidecar next to it
Recommended layout:
workspace/
├── researcher_profile.json
├── paper_deep_read/
├── literature_synthesis/
└── review_outline/Migration
paper-workbench remains the only public paper-analysis entrypoint, but it is no longer just a thin json / interpret / xray router.
Stable contracts
paper-recordremains the canonical normalization schemajson,interpret, andxrayremain valid public modesnormalize_paper.pyremains the single-paper normalization helper
New contracts
researcher-profilepaper-deep-readliterature-synthesisreview-outline
Alias mapping
- “先快速扫一下” / “预判” →
scan - “精读这篇” / “解构这篇” →
deep-read - “只做卡片” →
card - “整合这几篇” / “对比分析” / “找研究空白” →
synthesis - “搭综述框架” / “写这一段” →
review
Backward-compatibility rule
When an old workflow only expects a normalized machine-readable artifact, stay in json mode and do not silently upgrade to a higher-level analysis mode.
Card mode
Card mode is the minimum reusable output for a single paper.
Required input
- one normalized
paper-record - optional
researcher-profile
Output shape
Return only:
文献快照 / Snapshot标准引用研究类型核心主张理论透镜关键发现与我的关联待用要点主要局限- one 200-300字批判性摘要
Rules
- Do not include the full deep-read report
- Keep the summary reusable inside a literature review body paragraph
- If strategic relevance depends on profile data that is missing, mark it as
provisional
Deep-read mode
Deep-read mode is the main single-paper workflow. It turns a normalized paper-record plus an optional researcher-profile into a strategic analysis artifact.
Required input
- one normalized
paper-record - use a
researcher-profilewhen available; if not, ask for missing profile
fields before producing the “与我研究的战略关联” section
Output shape
1. 快速预判
Use the same five fields as scan mode.
2. 思辨性深度解构
A1. 理论坐标定位A2. 论证逻辑拆解A3. 方法论审视A4. 学术贡献与潜在影响
3. 与我研究的战略关联
B1. 理论工具B2. 方法启示B3. 证据支持B4. 对话定位
4. 可直接复用的输出
C1. 文献摘要卡片C2. 批判性摘要
Rules
- Clearly separate
作者观点from系统分析 B3. 证据支持may cite page anchors only when they are grounded in
content.page_chunks; otherwise use [信息待核实]
- The final card must be reusable on its own without the rest of the report
Interpret mode
Interpret mode consumes a normalized paper-record payload as its fact source. It is the lightweight, profile-optional explanation path.
Handoff fields
Prefer these normalized fields:
bibliography.titlebibliography.authorsbibliography.venuebibliography.abstractcontent.summarycontent.problemcontent.methodcontent.resultscontent.sections
Output shape
- 1 short orientation paragraph
- 3-5 bullets on problem, method, findings, and why the paper matters
- explicit limitations when normalized facts are sparse
Rules
- Do not re-parse the original PDF or URL when the normalized record already has
enough facts
- Keep the interpretation grounded in the normalized record
- If important fields are missing, state the limitation instead of inventing
details
JSON mode
Use JSON mode when the user wants a structured record, reusable machine-readable facts, a saved artifact, or an input for another skill.
Behavior
1. Normalize first 2. Return the exact paper-record payload 3. Prefer a short status line plus one fenced json block
Save behavior
If the user asks to save the normalized JSON, pass --save PATH to normalize_paper.py and then return the same payload unless they explicitly asked for silent save only.
Review mode
Review mode converts integrated literature analysis into a literature-review plan or paragraph draft.
Required input
- prefer one
literature-synthesisartifact - otherwise accept 3+
paper-deep-readartifacts plus aresearcher-profile
Output shape
1. 综述叙事策略选择
Choose one:
- 主题聚类式
- 论辩推进式
- 发展脉络式
- 方法检视式
- 缺口导向式
Explain why it fits the current material.
2. 结构化大纲
- at most 3 heading levels
- each section includes
论证目标,拟引用文献,过渡逻辑
3. 段落写作引擎
Use PEEL:
PointEvidenceEvaluationLink
Rules
- A review paragraph is a micro-argument, not a pile of paper summaries
- Do not cite unsupported page numbers or verbatim quotes
- If the user asks to “写这一段”, keep the paragraph anchored in the chosen
section goal and explicit literature positions
Scan mode
Scan mode is the quick triage step for a single paper.
Required input
- one normalized
paper-record - optional
researcher-profile
Output shape
Return exactly these five fields, using 3-5 sentences total:
文章类别学术意图核心主张相关度评估阅读建议
Rules
- This is a triage pass, not a full summary
- If profile data exists, relevance must be evaluated against the user’s own
research question
- If source fidelity is low, say so instead of pretending the paper was deeply
read
Synthesis mode
Synthesis mode is the multi-paper integration workflow.
Required input
- prefer 3 or more
paper-recordorpaper-deep-readinputs - use a
researcher-profilewhen available
Output shape
Full synthesis (3+ inputs)
概念图谱争论光谱方法论矩阵研究空白地图文献关系网
Comparison-only fallback (2 inputs)
核心概念对照立场差异方法差异对我的研究意味着什么
Rules
- Never turn this into five serial paper summaries
- The “研究空白地图” section must classify gaps as phenomenon, context,
theory, or method
- When there are only 2 inputs, explicitly label the gap map as provisional or
omit it
- Tie each gap back to the user’s research question when a profile exists
X-Ray mode
X-ray mode consumes a normalized paper-record payload as its fact source. This is the compact critique path for users who want logic, assumptions, and boundaries rather than a full literature-workbench output.
Handoff fields
Prefer these normalized fields:
bibliography.titlebibliography.authorsbibliography.abstractcontent.summarycontent.problemcontent.methodcontent.resultscontent.sectionscontent.page_chunksarxiv_enhancement.key_insightsarxiv_enhancement.citations
Output shape
Follow references/ANALYSIS_FRAMEWORK.md:
- core pain point
- solving mechanism
- delta / novelty
- hidden assumptions and open problems
- napkin formula / sketch
Rules
- Deconstruct the paper's logic model, not just the abstract wording
- Use the normalized record as the source of truth
- Call out when critique confidence is limited by sparse source fidelity
- If a requested page or quotation cannot be grounded in
page_chunks, output
[信息待核实]
Routing
Source classes
Treat the input as one of:
paper-recordJSONresearcher-profileJSONpaper-deep-read/literature-synthesis/review-outlineJSON- arXiv ID
- arXiv URL
- AlphaXiv URL
- DOI string
- DOI URL
- local PDF / text file
- remote PDF URL
- generic landing page with a PDF link
Normalize-first rule
Before any single-paper analysis mode, normalize the paper source into the shared paper-record schema.
Do not let downstream modes re-parse the original PDF or URL when a normalized record already exists.
Profile rule
deep-read, card, synthesis, and review should use a researcher-profile when available.
If a profile-sensitive mode is requested and the user has not given the five profile fields, ask only for the missing facts:
- research field
- core question
- thesis (optional)
- target tier
- current stage
Mode routing
Single-paper modes
scan- Single paper
- 3-5 sentence quick triage
deep-read- Single paper
- Full academic deconstruction + strategic relevance + literature card
card- Single paper
- Only output the reusable card and short critical summary
interpret- Single paper
- Lightweight explanation grounded in normalized facts
xray- Single paper
- Compact logic/assumption critique using the x-ray framework
json- Return the canonical normalized record
Cross-paper modes
synthesis- Prefer 3+
paper-recordorpaper-deep-readinputs - If there are only 2 inputs, degrade to comparison mode and say the gap map
is provisional
review- Prefer a
literature-synthesisartifact - Otherwise accept 3+
paper-deep-readartifacts plus a profile
Compatibility aliases
jsonkeeps the old machine-readable route unchangedinterpretkeeps the old lightweight human-readable routexraykeeps the old compact critique route- Old “single paper with no explicit mode” behavior should default to:
jsonfor tool-facing / save-facing / schema-facing requestsscanfor human reading requests
Fallbacks
- arXiv / AlphaXiv → prefer AlphaXiv structured metadata, then arXiv PDF
- DOI → prefer Crossref metadata; if only metadata is available, return partial
metadata and mark missing facts instead of inventing full text
- landing page → resolve a PDF link when possible
- existing normalized JSON → trust it and reuse it directly
paper-record schema
Canonical top-level fields:
{
"schema_version": "paper-record",
"status": "resolved|partial|unresolved",
"source": {},
"document": {},
"bibliography": {},
"content": {},
"arxiv_enhancement": {},
"provenance": {},
"errors": []
}Required shape
source
inputinput_kindresolved_pdf_urlcanonical_url
document
document_typedegree_levellanguage
bibliography
titleauthorsyearvenuepublisherdoiabstractkeywords
content
summaryproblemmethodresultssectionspage_chunksfull_text_markdownfull_text_included
content.page_chunks
Optional page-level anchors for downstream citation or quote lookup.
{
"anchor": "p3",
"page_start": 3,
"page_end": 3,
"label": "p3",
"excerpt": "Short preview of the page text.",
"text": "Full page text when full_text_included=true, otherwise null."
}Rules:
page_chunksmay be empty when the source only resolved to metadata.textmay benullwhen full text retention is disabled.- Downstream modes must output
[信息待核实]instead of inventing page numbers or verbatim quotations when the needed anchor is missing.
arxiv_enhancement
arxiv_idalphaxiv_availableintermediate_reportkey_insightscitations
provenance
metadata_sourcescontent_sourceswarningsconfidence
Status semantics
resolved— enough structured facts exist for downstream analysispartial— useful metadata exists, but important fields are missingunresolved— the source could not be converted into a usable paper record
#+title: paper-{简短标题}
#+date: [{YYYY-MM-DD Day HH:MM}]
#+filetags: :paper:
#+identifier: {YYYYMMDDTHHMMSS}
#+source: {URL 或来源描述}
#+authors: {作者列表}
#+venue: {发表场所/年份}
* 问题
{真实的困境是什么?用一个让外行也好奇的问题开头。}
{来龙去脉:从什么问题出发,之前试了什么,卡在哪里,这篇论文找到了什么路。}
* 翻译
{把核心想法讲到外行能跟上。形式自由——类比、图、例子、递进讲解,用最适合的方式。}
{需要覆盖:怎么做的、效果如何、理解全文的钥匙概念。子标题按内容需要组织,不必固定。}
* 核心概念
{挑出 1-3 个理解这篇论文必须过的概念,逐个拆解。}
{每个概念:一句话说清是什么、一个类比或例子让人秒懂、为什么少了它就跟不上后面的内容。}
* 洞见
{一句话:这篇论文真正找到的思想结晶。}
{脱离论文上下文,这句话本身还有没有力量?不要复述结论,要提炼你自己看到的东西。没有就直说没有。}
* 博导审稿
{像带了二十年研究生的博导在办公室跟学生聊。}
{选题眼光、方法成熟度、实验诚意、写作功力。}
{判决:strong accept / weak accept / borderline / weak reject / strong reject,一句话理由。}
* 启发
{迁移/混搭/反转三个视角试探,命中展开,全没命中说「没有」。}
#+title: xray-{简短标题}
#+date: [{YYYY-MM-DD Day HH:MM}]
#+filetags: :read:xray:paper:
#+identifier: {YYYYMMDDTHHMMSS}
#+source: {论文链接}
#+authors: {作者}
#+venue: {会议/期刊}
* NAPKIN FORMULA
#+begin_example
+----------------------------------------------------------+
| |
| {餐巾纸公式} |
| |
+----------------------------------------------------------+
#+end_example
{一句话解释公式含义}
* PROBLEM
**痛点定义**: {一句话定义问题}
**前人困境**: {为什么之前解决不了}
* INSIGHT
**核心直觉**: {作者的灵光一闪,用最直白的语言}
**关键步骤**:
1. {神来之笔1}
2. {神来之笔2}
* DELTA
**vs SOTA**: {相比当前最佳的具体提升}
**新拼图**: {为人类知识库增加了什么}
* CRITIQUE
**隐形假设**:
- {假设1}
- {假设2}
**未解之谜**:
- {遗留问题1}
- {遗留问题2}
* LOGIC FLOW
#+begin_example
{纯 ASCII 逻辑结构图: 问题 --> 洞见 --> 方法 --> 结果}
#+end_example
* NAPKIN SKETCH
#+begin_example
{餐巾纸图: 用 ASCII 绘制核心概念}
#+end_example
#!/usr/bin/env python3
"""Normalize arXiv inputs and generic academic PDFs into a shared JSON schema."""
from __future__ import annotations
import argparse
import importlib.util
import json
import pathlib
import re
import sys
import tempfile
import urllib.error
import urllib.parse
import urllib.request
from html import unescape
from typing import Any
SCHEMA_VERSION = "paper-record"
USER_AGENT = "paper-workbench/1.0 (+https://example.invalid)"
ARXIV_ID_RE = re.compile(
r"(?P<id>(?:\d{4}\.\d{4,5}|[a-z\-]+(?:\.[A-Z]{2})?/\d{7})(?:v\d+)?)",
re.IGNORECASE,
)
DOI_RE = re.compile(r"(?P<doi>10\.\d{4,9}/[-._;()/:A-Z0-9]+)", re.IGNORECASE)
URL_RE = re.compile(r"^https?://", re.IGNORECASE)
PDF_LINK_RE = re.compile(r'href=["\']([^"\']+\.pdf(?:\?[^"\']*)?)["\']', re.IGNORECASE)
META_PDF_RE = re.compile(
r'<meta[^>]+(?:name|property)=["\'](?:citation_pdf_url|pdf_url)["\'][^>]+content=["\']([^"\']+)["\']',
re.IGNORECASE,
)
TITLE_META_RE = re.compile(
r'<meta[^>]+(?:name|property)=["\'](?:citation_title|og:title|dc\.title)["\'][^>]+content=["\']([^"\']+)["\']',
re.IGNORECASE,
)
SECTION_PATTERNS = [
re.compile(r"^(abstract|introduction|background|method|methods|approach|experiment|experiments|results|discussion|conclusion|limitations|references)\b", re.IGNORECASE),
re.compile(r"^\d+(?:\.\d+)*\s+\S"),
re.compile(r"^[IVXLC]+\.\s+\S", re.IGNORECASE),
re.compile(r"^(摘\s*要|关键词|引言|绪论|研究方法|实验|结果|讨论|结论|参考文献)\b"),
]
ABSTRACT_START_PATTERNS = [
re.compile(r"^abstract\b", re.IGNORECASE),
re.compile(r"^摘\s*要\b"),
]
ABSTRACT_END_PATTERNS = [
re.compile(r"^keywords?\b", re.IGNORECASE),
re.compile(r"^index terms\b", re.IGNORECASE),
re.compile(r"^关键词[::]?", re.IGNORECASE),
re.compile(r"^\d+(?:\.\d+)*\s+[A-Z]"),
re.compile(r"^[IVXLC]+\.\s+[A-Z]", re.IGNORECASE),
re.compile(r"^(introduction|background|method|methods|approach|引言|绪论|研究方法)\b", re.IGNORECASE),
]
KEYWORDS_PATTERNS = [
re.compile(r"^keywords?[::]?\s*(.+)$", re.IGNORECASE),
re.compile(r"^index terms[::]?\s*(.+)$", re.IGNORECASE),
re.compile(r"^关键词[::]?\s*(.+)$", re.IGNORECASE),
]
THESIS_PATTERNS = {
"doctor": [
re.compile(r"博士学位论文"),
re.compile(r"doctoral dissertation", re.IGNORECASE),
re.compile(r"ph\.?d\.?\s+thesis", re.IGNORECASE),
],
"master": [
re.compile(r"硕士学位论文"),
re.compile(r"master(?:'s)? thesis", re.IGNORECASE),
re.compile(r"master dissertation", re.IGNORECASE),
],
}
def load_xray_io_module():
script_path = pathlib.Path(__file__).resolve().parent / "xray_io.py"
spec = importlib.util.spec_from_file_location("xray_io", script_path)
if spec is None or spec.loader is None:
raise RuntimeError(f"Unable to load xray_io helper from {script_path}")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
XRAY_IO = load_xray_io_module()
def make_base_record(source: str, input_kind: str) -> dict[str, Any]:
return {
"schema_version": SCHEMA_VERSION,
"status": "unresolved",
"source": {
"input": source,
"input_kind": input_kind,
"resolved_pdf_url": None,
"canonical_url": None,
},
"document": {
"document_type": "unknown",
"degree_level": None,
"language": None,
},
"bibliography": {
"title": None,
"authors": [],
"year": None,
"venue": None,
"publisher": None,
"doi": None,
"abstract": None,
"keywords": [],
},
"content": {
"summary": None,
"problem": None,
"method": None,
"results": None,
"sections": [],
"page_chunks": [],
"full_text_markdown": None,
"full_text_included": False,
},
"arxiv_enhancement": {
"arxiv_id": None,
"alphaxiv_available": False,
"intermediate_report": None,
"key_insights": [],
"citations": [],
},
"provenance": {
"metadata_sources": [],
"content_sources": [],
"warnings": [],
"confidence": "low",
},
"errors": [],
}
def is_url(value: str) -> bool:
return bool(URL_RE.match(value.strip()))
def extract_arxiv_id(source: str) -> str | None:
parsed = urllib.parse.urlparse(source)
if parsed.scheme and parsed.netloc:
candidate = urllib.parse.unquote(parsed.path)
for part in candidate.split("/"):
match = ARXIV_ID_RE.fullmatch(part)
if match:
return match.group("id")
if parsed.netloc.lower().endswith("alphaxiv.org"):
query_id = urllib.parse.parse_qs(parsed.query).get("id")
if query_id:
match = ARXIV_ID_RE.search(query_id[0])
if match:
return match.group("id")
match = ARXIV_ID_RE.fullmatch(source.strip())
if match:
return match.group("id")
match = ARXIV_ID_RE.search(source)
return match.group("id") if match else None
def extract_doi(source: str) -> str | None:
parsed = urllib.parse.urlparse(source)
if parsed.scheme and parsed.netloc:
candidates = [urllib.parse.unquote(parsed.path), urllib.parse.unquote(parsed.query)]
for candidate in candidates:
match = DOI_RE.search(candidate)
if match:
return match.group("doi").rstrip(").,;]")
match = DOI_RE.search(source.strip())
if match:
return match.group("doi").rstrip(").,;]")
return None
def detect_input_kind(source: str) -> str:
path = pathlib.Path(source).expanduser()
if path.exists():
suffix = path.suffix.lower()
if suffix == ".pdf":
return "local_pdf"
if suffix == ".json":
return "normalized_json"
return "local_text"
lowered = source.lower().strip()
arxiv_id = extract_arxiv_id(source)
doi = extract_doi(source)
if arxiv_id and ("arxiv.org" in lowered or "alphaxiv.org" in lowered or ARXIV_ID_RE.fullmatch(source.strip())):
if "alphaxiv.org" in lowered:
return "alphaxiv_url"
if "arxiv.org" in lowered:
return "arxiv_url"
return "arxiv_id"
if doi:
return "doi"
if is_url(source):
if lowered.endswith(".pdf") or "/pdf/" in lowered:
return "web_pdf"
return "generic_paper_url"
return "generic_text"
def read_json_file(path: pathlib.Path) -> dict[str, Any]:
with path.open(encoding="utf-8") as handle:
payload = json.load(handle)
schema_version = payload.get("schema_version")
if schema_version == SCHEMA_VERSION:
return payload
raise ValueError("Unsupported normalized JSON schema")
def http_get(url: str, *, binary: bool = False) -> tuple[bytes | str, str, str]:
req = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(req, timeout=30) as response:
final_url = response.geturl()
content_type = response.headers.get_content_type()
data = response.read()
if binary:
return data, content_type, final_url
return data.decode("utf-8", errors="replace"), content_type, final_url
def fetch_json(url: str) -> dict[str, Any] | None:
try:
text, _, _ = http_get(url)
except urllib.error.HTTPError as exc:
if exc.code == 404:
return None
raise
assert isinstance(text, str)
return json.loads(text)
def fetch_text(url: str) -> str | None:
try:
text, _, _ = http_get(url)
except urllib.error.HTTPError as exc:
if exc.code == 404:
return None
raise
assert isinstance(text, str)
return text
def fetch_binary(url: str) -> tuple[bytes, str, str]:
data, content_type, final_url = http_get(url, binary=True)
assert isinstance(data, bytes)
return data, content_type, final_url
def fetch_crossref_json(doi: str) -> dict[str, Any] | None:
encoded = urllib.parse.quote(doi, safe="")
try:
text, _, _ = http_get(f"https://api.crossref.org/works/{encoded}")
except urllib.error.HTTPError as exc:
if exc.code == 404:
return None
raise
assert isinstance(text, str)
payload = json.loads(text)
message = payload.get("message")
return message if isinstance(message, dict) else None
def crossref_authors(message: dict[str, Any]) -> list[dict[str, Any]]:
authors = []
for author in message.get("author") or []:
if not isinstance(author, dict):
continue
given = clean_line(str(author.get("given") or ""))
family = clean_line(str(author.get("family") or ""))
name = clean_line(" ".join(part for part in [given, family] if part))
if not name:
name = clean_line(str(author.get("name") or ""))
affiliation = None
raw_affiliations = author.get("affiliation") or []
if raw_affiliations and isinstance(raw_affiliations, list):
first = raw_affiliations[0]
if isinstance(first, dict):
affiliation = clean_line(str(first.get("name") or "")) or None
if name:
authors.append({"name": name, "affiliation": affiliation})
return authors
def first_crossref_value(value: Any) -> str | None:
if isinstance(value, list):
for item in value:
if isinstance(item, str) and clean_line(item):
return clean_line(item)
return None
if isinstance(value, str):
return clean_line(value)
return None
def normalize_doi_source(source: str, *, fulltext_mode: str) -> dict[str, Any]:
record = make_base_record(source, "doi")
doi = extract_doi(source)
record["bibliography"]["doi"] = doi
record["source"]["canonical_url"] = f"https://doi.org/{doi}" if doi else None
if not doi:
record["errors"].append("Could not extract a DOI from the source.")
return record
message = fetch_crossref_json(doi)
if not message:
record["errors"].append("Crossref did not return metadata for this DOI.")
return record
record["provenance"]["metadata_sources"].append("crossref")
record["bibliography"]["title"] = first_crossref_value(message.get("title"))
record["bibliography"]["authors"] = crossref_authors(message)
issued = message.get("issued", {}).get("date-parts", [])
if issued and isinstance(issued, list) and issued[0]:
first_year = issued[0][0]
if isinstance(first_year, int):
record["bibliography"]["year"] = first_year
record["bibliography"]["venue"] = first_crossref_value(message.get("container-title"))
record["bibliography"]["publisher"] = clean_line(str(message.get("publisher") or "")) or None
record["bibliography"]["abstract"] = clean_line(str(message.get("abstract") or "")) or None
record["bibliography"]["keywords"] = [clean_line(str(item)) for item in (message.get("subject") or []) if clean_line(str(item))]
landing_page = first_crossref_value(message.get("URL"))
raw_resource = message.get("resource")
resource = raw_resource if isinstance(raw_resource, dict) else {}
primary_url = first_crossref_value(resource.get("primary")) or landing_page
if primary_url:
record["source"]["canonical_url"] = primary_url
record["document"]["document_type"] = infer_document_type("", "doi", record["bibliography"]["venue"])
summary, problem, method, results = summarize_from_abstract(record["bibliography"]["abstract"])
record["content"]["summary"] = summary
record["content"]["problem"] = problem
record["content"]["method"] = method
record["content"]["results"] = results
record["content"]["sections"] = []
record["content"]["full_text_included"] = False
record["content"]["full_text_markdown"] = None
record["provenance"]["confidence"] = "medium" if record["content"]["summary"] else "low"
record["status"] = "resolved" if record["bibliography"]["title"] and record["content"]["summary"] else "partial"
if fulltext_mode == "prefer":
record["provenance"]["warnings"].append("DOI resolution returned metadata only; no direct full text was fetched.")
return record
def resolve_pdf_from_html(url: str, html: str) -> tuple[str | None, str | None]:
meta_match = META_PDF_RE.search(html)
if meta_match:
return urllib.parse.urljoin(url, unescape(meta_match.group(1).strip())), extract_html_title(html)
link_match = PDF_LINK_RE.search(html)
if link_match:
return urllib.parse.urljoin(url, unescape(link_match.group(1).strip())), extract_html_title(html)
return None, extract_html_title(html)
def extract_html_title(html: str) -> str | None:
meta_match = TITLE_META_RE.search(html)
if meta_match:
return clean_line(meta_match.group(1))
title_match = re.search(r"<title>(.*?)</title>", html, re.IGNORECASE | re.DOTALL)
if title_match:
return clean_line(unescape(title_match.group(1)))
return None
def clean_line(value: str) -> str:
value = unescape(value)
value = re.sub(r"\s+", " ", value).strip(" \t\r\n-:|")
return value
def build_page_chunks(
raw_chunks: list[dict[str, Any]] | None,
*,
include_text: bool,
) -> list[dict[str, Any]]:
if not raw_chunks:
return []
normalized: list[dict[str, Any]] = []
for index, chunk in enumerate(raw_chunks, start=1):
text = str(chunk.get("text") or "").strip()
if not text:
continue
anchor = str(chunk.get("anchor") or f"p{index}").strip() or f"p{index}"
page_start = chunk.get("page_start") or index
page_end = chunk.get("page_end") or page_start
label = clean_line(str(chunk.get("label") or anchor)) or anchor
excerpt = clean_line(str(chunk.get("excerpt") or text[:280])) or None
normalized.append(
{
"anchor": anchor,
"page_start": page_start,
"page_end": page_end,
"label": label,
"excerpt": excerpt,
"text": text if include_text else None,
}
)
return normalized
def normalize_whitespace(text: str) -> str:
text = text.replace("\r\n", "\n").replace("\r", "\n")
text = text.replace("\u00a0", " ")
text = re.sub(r"[ \t]+\n", "\n", text)
text = re.sub(r"\n{3,}", "\n\n", text)
return text.strip()
def split_lines(text: str) -> list[str]:
return [line.strip() for line in normalize_whitespace(text).splitlines()]
def is_heading(line: str) -> bool:
if not line:
return False
return any(pattern.search(line) for pattern in SECTION_PATTERNS)
def top_block_lines(text: str, *, max_lines: int = 60) -> list[str]:
lines = split_lines(text)
return [line for line in lines[:max_lines] if line]
def infer_language(text: str) -> str:
sample = text[:6000]
cjk_count = sum(1 for char in sample if "\u4e00" <= char <= "\u9fff")
if cjk_count >= 30 or (sample and cjk_count / max(len(sample), 1) > 0.02):
return "zh"
return "en"
def extract_title_from_label(lines: list[str]) -> str | None:
patterns = [
re.compile(r"^title[::]\s*(.+)$", re.IGNORECASE),
re.compile(r"^题目[::]\s*(.+)$"),
]
for line in lines[:15]:
for pattern in patterns:
match = pattern.match(line)
if match:
return clean_line(match.group(1))
return None
def score_title_candidate(line: str, title_context: bool) -> int:
score = 0
if 12 <= len(line) <= 180:
score += 5
if re.search(r"[A-Za-z\u4e00-\u9fff]", line):
score += 3
if "@" in line or re.search(r"\b(university|department|school|email|advisor)\b", line, re.IGNORECASE):
score -= 6
if re.search(r"\b(arxiv|doi|abstract|authors?|venue|keywords?)\b", line, re.IGNORECASE):
score -= 5
if re.search(r"(硕士学位论文|博士学位论文|master(?:'s)? thesis|doctoral dissertation)", line, re.IGNORECASE):
score -= 4
if title_context:
score += 2
if len(line.split()) >= 3:
score += 2
return score
def extract_title(text: str) -> str | None:
lines = top_block_lines(text)
labelled = extract_title_from_label(lines)
if labelled:
return labelled
best_line = None
best_score = -999
for index, line in enumerate(lines[:20]):
score = score_title_candidate(line, title_context=index < 8)
if score > best_score:
best_score = score
best_line = line
return clean_line(best_line) if best_line and best_score > 2 else None
def parse_author_names(raw: str) -> list[dict[str, Any]]:
raw = clean_line(raw)
raw = re.sub(r"^(authors?|作者)[::]\s*", "", raw, flags=re.IGNORECASE)
separators = [";", ";", ",", ",", " and ", "、"]
normalized = raw
for separator in separators:
normalized = normalized.replace(separator, "|")
names = [clean_line(part) for part in normalized.split("|") if clean_line(part)]
results = []
for name in names:
if re.search(r"(university|department|school|laboratory|email|advisor|指导教师)", name, re.IGNORECASE):
continue
if len(name) > 80:
continue
results.append({"name": name, "affiliation": None})
return results
def extract_authors(text: str, title: str | None = None) -> list[dict[str, Any]]:
lines = top_block_lines(text)
for line in lines[:20]:
if re.match(r"^(authors?|作者)[::]", line, re.IGNORECASE):
authors = parse_author_names(line)
if authors:
return authors
title_seen = title is None
for line in lines[:20]:
if title and clean_line(line) == clean_line(title):
title_seen = True
continue
if not title_seen or len(line) > 160:
continue
english_names = re.findall(r"[A-Z][A-Za-z.'-]+(?:\s+[A-Z][A-Za-z.'-]+){1,3}", line)
if len(english_names) >= 2:
authors = [{"name": clean_line(name), "affiliation": None} for name in english_names]
if authors:
return authors
if re.search(r"作者", line):
authors = parse_author_names(line)
if authors:
return authors
return []
def extract_year(text: str) -> int | None:
top = "\n".join(top_block_lines(text, max_lines=80))
matches = [int(match.group(0)) for match in re.finditer(r"\b(19|20)\d{2}\b", top)]
plausible = [year for year in matches if 1990 <= year <= 2035]
return plausible[0] if plausible else None
def extract_venue_and_publisher(text: str) -> tuple[str | None, str | None]:
lines = top_block_lines(text, max_lines=50)
venue = None
publisher = None
for line in lines:
venue_match = re.match(r"^venue[::]\s*(.+)$", line, re.IGNORECASE)
if venue_match:
venue = clean_line(venue_match.group(1))
break
if venue is None:
for line in lines:
if re.search(r"\b(proceedings|conference|symposium|workshop|transactions on|journal of)\b", line, re.IGNORECASE):
venue = clean_line(line)
break
sample = "\n".join(lines)
if re.search(r"\bIEEE\b", sample):
publisher = "IEEE"
elif re.search(r"\bElsevier\b", sample, re.IGNORECASE):
publisher = "Elsevier"
elif re.search(r"\bACM\b", sample):
publisher = "ACM"
elif re.search(r"\bSpringer\b", sample, re.IGNORECASE):
publisher = "Springer"
return venue, publisher
def collect_block(
lines: list[str],
start_patterns: list[re.Pattern[str]],
end_patterns: list[re.Pattern[str]],
) -> str | None:
start_index = None
for index, line in enumerate(lines):
if any(pattern.match(line) for pattern in start_patterns):
start_index = index
break
if start_index is None:
return None
parts = []
first_line = lines[start_index]
stripped_first = re.sub(r"^[A-Za-z\u4e00-\u9fff ]+[::]?\s*", "", first_line)
if stripped_first and stripped_first != first_line:
parts.append(stripped_first)
for line in lines[start_index + 1 :]:
if any(pattern.match(line) for pattern in end_patterns):
break
parts.append(line)
block = "\n".join(part for part in parts if part).strip()
return block or None
def extract_abstract(text: str) -> str | None:
lines = split_lines(text)
block = collect_block(lines, ABSTRACT_START_PATTERNS, ABSTRACT_END_PATTERNS)
if block:
return normalize_whitespace(block)
return None
def split_keywords(raw: str) -> list[str]:
cleaned = re.sub(r"\s+", " ", raw).strip(" ::;;,.,")
if not cleaned:
return []
normalized = cleaned
for separator in [";", ";", ",", ",", " / ", "|", "·"]:
normalized = normalized.replace(separator, "|")
return [part.strip() for part in normalized.split("|") if part.strip()]
def extract_keywords(text: str) -> list[str]:
for line in split_lines(text):
for pattern in KEYWORDS_PATTERNS:
match = pattern.match(line)
if match:
return split_keywords(match.group(1))
return []
def extract_sections(text: str) -> list[str]:
sections = []
seen = set()
for line in split_lines(text):
candidate = clean_line(line)
if not candidate or len(candidate) > 120:
continue
if is_heading(candidate):
normalized = candidate.lower()
if normalized not in seen:
seen.add(normalized)
sections.append(candidate)
return sections[:20]
def extract_label_value(text: str, labels: list[str]) -> str | None:
pattern = re.compile(
rf"^(?:{'|'.join(labels)})[::]\s*(.+)$",
re.IGNORECASE | re.MULTILINE,
)
match = pattern.search(text)
if match:
return clean_line(match.group(1))
return None
def infer_degree_level(text: str) -> str | None:
for level, patterns in THESIS_PATTERNS.items():
if any(pattern.search(text) for pattern in patterns):
return level
return None
def infer_document_type(text: str, source_kind: str, venue: str | None) -> str:
if source_kind in {"arxiv_id", "arxiv_url", "alphaxiv_url"}:
return "preprint"
degree_level = infer_degree_level(text)
if degree_level:
return "thesis"
venue_text = f"{venue or ''}\n{text[:3000]}"
if re.search(r"\b(proceedings|conference|symposium|workshop)\b", venue_text, re.IGNORECASE):
return "conference-paper"
if re.search(r"\b(journal|transactions on|letters|review)\b", venue_text, re.IGNORECASE):
return "journal-article"
return "unknown"
def sentence_split(text: str) -> list[str]:
text = re.sub(r"\s+", " ", text).strip()
if not text:
return []
pieces = re.split(r"(?<=[.!?。!?])\s+", text)
return [piece.strip() for piece in pieces if piece.strip()]
def find_sentence(text: str, patterns: list[str]) -> str | None:
for sentence in sentence_split(text):
if any(re.search(pattern, sentence, re.IGNORECASE) for pattern in patterns):
return sentence
return None
def summarize_from_abstract(abstract: str | None) -> tuple[str | None, str | None, str | None, str | None]:
if not abstract:
return None, None, None, None
sentences = sentence_split(abstract)
summary = " ".join(sentences[:2]) if sentences else abstract
problem = sentences[0] if sentences else abstract
method = find_sentence(
abstract,
[r"\bwe (propose|present|introduce|develop)\b", r"\b提出\b", r"\b设计\b"],
)
if method is None and len(sentences) >= 2:
method = sentences[1]
results = find_sentence(
abstract,
[r"\b(improv|outperform|achiev|reduce|f1|accuracy|bleu|auc)\b", r"实验结果", r"提升"],
)
return summary, problem, method, results
def should_include_full_text(text: str, mode: str, *, summary_present: bool) -> bool:
if mode == "prefer":
return True
if mode == "never":
return False
return len(text) <= 8000 or not summary_present
def choose_canonical_url(source: str, input_kind: str, arxiv_id: str | None) -> str | None:
if input_kind == "alphaxiv_url":
return source
if arxiv_id:
return f"https://arxiv.org/abs/{arxiv_id}"
return source if is_url(source) else None
def parse_bibtex_value(bibtex: str | None, field: str) -> str | None:
if not bibtex:
return None
match = re.search(rf"{field}\s*=\s*{{(.*?)}}", bibtex, re.IGNORECASE | re.DOTALL)
if match:
return clean_line(match.group(1))
return None
def authors_from_bibtex(bibtex: str | None) -> list[dict[str, Any]]:
raw = parse_bibtex_value(bibtex, "author")
if not raw:
return []
parts = [clean_line(part) for part in re.split(r"\s+and\s+", raw) if clean_line(part)]
return [{"name": part, "affiliation": None} for part in parts]
def enrich_from_text_record(
record: dict[str, Any],
text: str,
*,
fulltext_mode: str,
page_chunks: list[dict[str, Any]] | None = None,
) -> dict[str, Any]:
text = normalize_whitespace(text)
language = infer_language(text)
title = record["bibliography"]["title"] or extract_title(text)
authors = record["bibliography"]["authors"] or extract_authors(text, title)
abstract = record["bibliography"]["abstract"] or extract_abstract(text)
year = record["bibliography"]["year"] or extract_year(text)
venue, publisher = extract_venue_and_publisher(text)
keywords = record["bibliography"]["keywords"] or extract_keywords(text)
sections = extract_sections(text)
summary, problem, method, results = summarize_from_abstract(abstract)
labelled_problem = extract_label_value(text, ["problem"])
labelled_method = extract_label_value(text, ["method", "approach", "solution", "insight"])
labelled_results = extract_label_value(text, ["results?", "delta"])
if summary is None:
summary_parts = [part for part in [labelled_problem, labelled_method, labelled_results] if part]
summary = " ".join(summary_parts) if summary_parts else None
problem = problem or labelled_problem
method = method or labelled_method
results = results or labelled_results
record["document"]["language"] = language
record["document"]["degree_level"] = record["document"]["degree_level"] or infer_degree_level(text)
record["document"]["document_type"] = infer_document_type(
text,
record["source"]["input_kind"],
venue or record["bibliography"]["venue"],
)
record["bibliography"]["title"] = title
record["bibliography"]["authors"] = authors
record["bibliography"]["year"] = year
record["bibliography"]["venue"] = record["bibliography"]["venue"] or venue
record["bibliography"]["publisher"] = record["bibliography"]["publisher"] or publisher
record["bibliography"]["abstract"] = abstract
record["bibliography"]["keywords"] = keywords
record["content"]["summary"] = record["content"]["summary"] or summary
record["content"]["problem"] = record["content"]["problem"] or problem
record["content"]["method"] = record["content"]["method"] or method
record["content"]["results"] = record["content"]["results"] or results
record["content"]["sections"] = record["content"]["sections"] or sections
include_full_text = should_include_full_text(
text,
fulltext_mode,
summary_present=bool(record["content"]["summary"]),
)
record["content"]["page_chunks"] = build_page_chunks(page_chunks, include_text=include_full_text)
record["content"]["full_text_included"] = include_full_text
record["content"]["full_text_markdown"] = text if include_full_text else None
has_content_signal = bool(
record["bibliography"]["abstract"]
or record["content"]["summary"]
or record["content"]["problem"]
or record["content"]["method"]
or record["content"]["results"]
or record["content"]["sections"]
or include_full_text
)
if record["bibliography"]["title"] and has_content_signal:
record["status"] = "resolved"
record["provenance"]["confidence"] = "medium"
elif record["bibliography"]["title"] or record["bibliography"]["authors"]:
record["status"] = "partial"
record["provenance"]["confidence"] = "low"
else:
record["status"] = "unresolved"
record["errors"].append("Could not extract enough metadata from the paper text.")
return record
def extract_text_from_downloaded_pdf(url: str) -> tuple[str, str, list[dict[str, Any]]]:
binary, _, final_url = fetch_binary(url)
with tempfile.NamedTemporaryFile(delete=False, suffix=".pdf") as handle:
handle.write(binary)
temp_path = pathlib.Path(handle.name)
try:
pages = XRAY_IO.extract_pages(str(temp_path))
finally:
temp_path.unlink(missing_ok=True)
text = "\n\n".join(page["text"] for page in pages if page.get("text"))
return text, final_url, pages
def normalize_generic_pdf_source(source: str, *, input_kind: str, fulltext_mode: str) -> dict[str, Any]:
record = make_base_record(source, input_kind)
record["source"]["canonical_url"] = source if is_url(source) else None
if input_kind in {"local_pdf", "local_text"}:
pages = XRAY_IO.extract_pages(source)
text = "\n\n".join(page["text"] for page in pages if page.get("text"))
record["provenance"]["content_sources"].append("local-file")
return enrich_from_text_record(record, text, fulltext_mode=fulltext_mode, page_chunks=pages)
if input_kind == "web_pdf":
text, final_url, pages = extract_text_from_downloaded_pdf(source)
record["source"]["resolved_pdf_url"] = final_url
record["source"]["canonical_url"] = final_url
record["provenance"]["content_sources"].append("remote-pdf")
return enrich_from_text_record(record, text, fulltext_mode=fulltext_mode, page_chunks=pages)
if input_kind == "generic_paper_url":
html, content_type, final_url = http_get(source)
if content_type == "application/pdf":
text, resolved_final_url, pages = extract_text_from_downloaded_pdf(final_url)
record["source"]["resolved_pdf_url"] = resolved_final_url
record["source"]["canonical_url"] = resolved_final_url
record["provenance"]["content_sources"].append("remote-pdf")
return enrich_from_text_record(record, text, fulltext_mode=fulltext_mode, page_chunks=pages)
assert isinstance(html, str)
pdf_url, html_title = resolve_pdf_from_html(final_url, html)
if html_title:
record["bibliography"]["title"] = html_title
record["provenance"]["metadata_sources"].append("page-metadata")
if pdf_url:
record["source"]["resolved_pdf_url"] = pdf_url
text, resolved_final_url, pages = extract_text_from_downloaded_pdf(pdf_url)
record["source"]["resolved_pdf_url"] = resolved_final_url
record["provenance"]["content_sources"].append("resolved-page-pdf")
return enrich_from_text_record(record, text, fulltext_mode=fulltext_mode, page_chunks=pages)
record["status"] = "unresolved"
record["provenance"]["warnings"].append("The page did not expose a PDF link.")
record["errors"].append("Paper landing pages must expose a PDF for generic parsing.")
return record
record["status"] = "unresolved"
record["errors"].append(f"Unsupported generic source kind: {input_kind}")
return record
def normalize_arxiv_source(source: str, *, input_kind: str, lang: str, fulltext_mode: str) -> dict[str, Any]:
record = make_base_record(source, input_kind)
arxiv_id = extract_arxiv_id(source)
record["source"]["canonical_url"] = choose_canonical_url(source, input_kind, arxiv_id)
record["source"]["resolved_pdf_url"] = f"https://arxiv.org/pdf/{arxiv_id}.pdf" if arxiv_id else None
record["arxiv_enhancement"]["arxiv_id"] = arxiv_id
if not arxiv_id:
record["errors"].append("Could not extract an arXiv identifier from the source.")
return record
paper_payload = fetch_json(f"https://api.alphaxiv.org/papers/v3/{arxiv_id}")
if not paper_payload:
record["provenance"]["warnings"].append("AlphaXiv did not return metadata; falling back to the arXiv PDF.")
fallback = normalize_generic_pdf_source(
record["source"]["resolved_pdf_url"],
input_kind="web_pdf",
fulltext_mode=fulltext_mode,
)
fallback["source"]["input"] = source
fallback["source"]["input_kind"] = input_kind
fallback["source"]["canonical_url"] = record["source"]["canonical_url"]
fallback["arxiv_enhancement"]["arxiv_id"] = arxiv_id
return fallback
record["provenance"]["metadata_sources"].append("alphaxiv-paper")
bibtex = paper_payload.get("citationBibtex")
record["bibliography"]["title"] = paper_payload.get("title") or parse_bibtex_value(bibtex, "title")
record["bibliography"]["authors"] = authors_from_bibtex(bibtex)
record["bibliography"]["year"] = int(parse_bibtex_value(bibtex, "year") or 0) or None
record["bibliography"]["venue"] = parse_bibtex_value(bibtex, "booktitle") or parse_bibtex_value(bibtex, "journal")
record["bibliography"]["abstract"] = paper_payload.get("abstract")
record["document"]["language"] = lang
record["document"]["document_type"] = "preprint"
record["provenance"]["confidence"] = "medium"
version_id = paper_payload.get("versionId")
overview_payload = None
if version_id:
overview_payload = fetch_json(f"https://api.alphaxiv.org/papers/v3/{version_id}/overview/{lang}")
if overview_payload:
record["provenance"]["content_sources"].append("alphaxiv-overview")
record["arxiv_enhancement"]["alphaxiv_available"] = True
record["arxiv_enhancement"]["intermediate_report"] = overview_payload.get("intermediateReport")
record["arxiv_enhancement"]["key_insights"] = overview_payload.get("summary", {}).get("keyInsights", [])
record["arxiv_enhancement"]["citations"] = overview_payload.get("citations", [])
summary_payload = overview_payload.get("summary") or {}
record["content"]["summary"] = summary_payload.get("summary")
problems = summary_payload.get("originalProblem") or []
solutions = summary_payload.get("solution") or []
results = summary_payload.get("results") or []
record["content"]["problem"] = " ".join(problems) if isinstance(problems, list) else problems
record["content"]["method"] = " ".join(solutions) if isinstance(solutions, list) else solutions
record["content"]["results"] = " ".join(results) if isinstance(results, list) else results
if record["content"]["summary"]:
record["status"] = "resolved"
record["provenance"]["confidence"] = "high"
alphaxiv_markdown = None
if fulltext_mode == "prefer" or (fulltext_mode == "auto" and not record["content"]["summary"]):
alphaxiv_markdown = fetch_text(f"https://alphaxiv.org/abs/{arxiv_id}.md")
if alphaxiv_markdown:
record["provenance"]["content_sources"].append("alphaxiv-markdown")
synthetic_page = [{"anchor": "p1", "page_start": 1, "page_end": 1, "label": "p1", "excerpt": alphaxiv_markdown[:280], "text": alphaxiv_markdown}]
enrich_from_text_record(record, alphaxiv_markdown, fulltext_mode="prefer", page_chunks=synthetic_page)
else:
summary, problem, method, results = summarize_from_abstract(record["bibliography"]["abstract"])
record["content"]["summary"] = record["content"]["summary"] or summary
record["content"]["problem"] = record["content"]["problem"] or problem
record["content"]["method"] = record["content"]["method"] or method
record["content"]["results"] = record["content"]["results"] or results
record["content"]["page_chunks"] = []
record["content"]["full_text_markdown"] = None
record["content"]["full_text_included"] = False
if record["bibliography"]["title"] and record["content"]["summary"]:
record["status"] = "resolved"
elif record["bibliography"]["title"]:
record["status"] = "partial"
if record["status"] == "unresolved" and record["bibliography"]["title"]:
record["status"] = "partial"
return record
def normalize_source(source: str, *, lang: str = "en", fulltext_mode: str = "auto") -> dict[str, Any]:
input_kind = detect_input_kind(source)
if input_kind == "normalized_json":
return read_json_file(pathlib.Path(source).expanduser())
if input_kind in {"arxiv_id", "arxiv_url", "alphaxiv_url"}:
return normalize_arxiv_source(source, input_kind=input_kind, lang=lang, fulltext_mode=fulltext_mode)
if input_kind == "doi":
return normalize_doi_source(source, fulltext_mode=fulltext_mode)
if input_kind in {"local_pdf", "local_text", "web_pdf", "generic_paper_url"}:
return normalize_generic_pdf_source(source, input_kind=input_kind, fulltext_mode=fulltext_mode)
record = make_base_record(source, input_kind)
record["errors"].append("Unsupported paper source. Provide an arXiv reference, DOI, or a paper PDF.")
return record
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--source", required=True, help="Local path, URL, or arXiv identifier.")
parser.add_argument("--lang", default="en", help="Language code for AlphaXiv summary fetches.")
parser.add_argument(
"--fulltext",
default="auto",
choices=["auto", "prefer", "never"],
help="Whether to include extracted full text in the JSON output.",
)
parser.add_argument("--save", help="Optional path to write the JSON payload.")
return parser.parse_args()
def main() -> int:
args = parse_args()
try:
payload = normalize_source(args.source, lang=args.lang, fulltext_mode=args.fulltext)
except Exception as exc: # pragma: no cover - CLI error path
print(str(exc), file=sys.stderr)
return 1
rendered = json.dumps(payload, ensure_ascii=False, indent=2)
if args.save:
target = pathlib.Path(args.save).expanduser()
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(rendered + "\n", encoding="utf-8")
print(rendered)
return 0
if __name__ == "__main__":
raise SystemExit(main())
#!/usr/bin/env python3
"""Helpers for researcher profiles and reusable workbench artifacts."""
from __future__ import annotations
import argparse
import json
import pathlib
import re
import shutil
import sys
from datetime import datetime, timezone
from typing import Any
ARTIFACT_VERSION = "1.0"
SUPPORTED_ARTIFACT_TYPES = {
"researcher-profile",
"paper-deep-read",
"literature-synthesis",
"review-outline",
}
def now_iso() -> str:
return datetime.now(timezone.utc).isoformat(timespec="seconds")
def slugify(value: str) -> str:
words = re.findall(r"[a-z0-9]+", value.lower())
if not words:
return "artifact"
return "-".join(words[:6])
def read_json(path: pathlib.Path) -> dict[str, Any]:
with path.open(encoding="utf-8") as handle:
return json.load(handle)
def write_json(path: pathlib.Path, payload: dict[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def build_researcher_profile(
*,
research_field: str | None = None,
core_question: str | None = None,
thesis: str | None = None,
target_tier: str | None = None,
stage: str | None = None,
) -> dict[str, Any]:
timestamp = now_iso()
return {
"artifact_type": "researcher-profile",
"version": ARTIFACT_VERSION,
"created_at": timestamp,
"updated_at": timestamp,
"research_field": research_field,
"core_question": core_question,
"thesis": thesis,
"target_tier": target_tier,
"stage": stage,
}
def update_researcher_profile(path: pathlib.Path, updates: dict[str, Any]) -> dict[str, Any]:
if path.exists():
payload = read_json(path)
else:
payload = build_researcher_profile()
payload["created_at"] = now_iso()
if payload.get("artifact_type") != "researcher-profile":
raise ValueError(f"Profile file does not contain a researcher-profile artifact: {path}")
for key, value in updates.items():
if value is not None:
payload[key] = value
payload["updated_at"] = now_iso()
return payload
def build_artifact(
*,
artifact_type: str,
title: str,
payload: dict[str, Any],
profile_path: str | None = None,
source_records: list[str] | None = None,
) -> dict[str, Any]:
if artifact_type not in SUPPORTED_ARTIFACT_TYPES - {"researcher-profile"}:
raise ValueError(f"Unsupported artifact type: {artifact_type}")
timestamp = now_iso()
return {
"artifact_type": artifact_type,
"version": ARTIFACT_VERSION,
"created_at": timestamp,
"updated_at": timestamp,
"title": title,
"profile_path": profile_path,
"source_records": source_records or [],
"payload": payload,
}
def artifact_directory_name(artifact_type: str) -> str:
return artifact_type.replace("-", "_")
def resolve_artifact_path(workspace: pathlib.Path, artifact_type: str, title: str) -> pathlib.Path:
if artifact_type not in SUPPORTED_ARTIFACT_TYPES - {"researcher-profile"}:
raise ValueError(f"Unsupported artifact type: {artifact_type}")
timestamp = datetime.now().strftime("%Y%m%dT%H%M%S")
base_name = f"{timestamp}--{artifact_type}--{slugify(title)}.json"
return workspace / artifact_directory_name(artifact_type) / base_name
def read_payload_file(path: str | None) -> dict[str, Any]:
if path is None or path == "-":
return json.load(sys.stdin)
payload_path = pathlib.Path(path).expanduser()
return read_json(payload_path)
def save_sidecar(artifact_path: pathlib.Path, sidecar_file: str | None, sidecar_ext: str | None) -> pathlib.Path | None:
if not sidecar_file:
return None
extension = sidecar_ext or pathlib.Path(sidecar_file).suffix.lstrip(".") or "md"
sidecar_path = artifact_path.with_suffix(f".{extension}")
source_path = pathlib.Path(sidecar_file).expanduser()
sidecar_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(source_path, sidecar_path)
return sidecar_path
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
subparsers = parser.add_subparsers(dest="command", required=True)
profile_parser = subparsers.add_parser("init-profile")
profile_parser.add_argument("--path", required=True)
profile_parser.add_argument("--research-field")
profile_parser.add_argument("--core-question")
profile_parser.add_argument("--thesis")
profile_parser.add_argument("--target-tier")
profile_parser.add_argument("--stage")
show_profile_parser = subparsers.add_parser("show-profile")
show_profile_parser.add_argument("--path", required=True)
resolve_parser = subparsers.add_parser("resolve-artifact")
resolve_parser.add_argument("--workspace", required=True)
resolve_parser.add_argument("--artifact-type", required=True)
resolve_parser.add_argument("--title", required=True)
save_parser = subparsers.add_parser("save-artifact")
save_parser.add_argument("--workspace", required=True)
save_parser.add_argument("--artifact-type", required=True)
save_parser.add_argument("--title", required=True)
save_parser.add_argument("--profile-path")
save_parser.add_argument("--payload-file", default="-")
save_parser.add_argument("--sidecar-file")
save_parser.add_argument("--sidecar-ext")
save_parser.add_argument("--source-record", action="append", default=[])
return parser.parse_args()
def main() -> int:
args = parse_args()
try:
if args.command == "init-profile":
profile_path = pathlib.Path(args.path).expanduser()
payload = update_researcher_profile(
profile_path,
{
"research_field": args.research_field,
"core_question": args.core_question,
"thesis": args.thesis,
"target_tier": args.target_tier,
"stage": args.stage,
},
)
write_json(profile_path, payload)
print(json.dumps(payload, ensure_ascii=False, indent=2))
return 0
if args.command == "show-profile":
profile_path = pathlib.Path(args.path).expanduser()
payload = read_json(profile_path)
print(json.dumps(payload, ensure_ascii=False, indent=2))
return 0
if args.command == "resolve-artifact":
workspace = pathlib.Path(args.workspace).expanduser()
print(resolve_artifact_path(workspace, args.artifact_type, args.title))
return 0
if args.command == "save-artifact":
workspace = pathlib.Path(args.workspace).expanduser()
artifact_path = resolve_artifact_path(workspace, args.artifact_type, args.title)
artifact = build_artifact(
artifact_type=args.artifact_type,
title=args.title,
payload=read_payload_file(args.payload_file),
profile_path=args.profile_path,
source_records=args.source_record,
)
write_json(artifact_path, artifact)
sidecar_path = save_sidecar(artifact_path, args.sidecar_file, args.sidecar_ext)
response = {
"artifact_path": str(artifact_path),
"sidecar_path": str(sidecar_path) if sidecar_path else None,
"artifact": artifact,
}
print(json.dumps(response, ensure_ascii=False, indent=2))
return 0
except Exception as exc: # pragma: no cover - CLI error path
print(str(exc), file=sys.stderr)
return 1
print(f"Unknown command: {args.command}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
#!/usr/bin/env python3
"""Utilities for local paper extraction and save path resolution."""
from __future__ import annotations
import argparse
import pathlib
import re
import sys
from datetime import datetime
from typing import Any
TEXT_SUFFIXES = {".txt", ".md", ".org"}
def slugify_title(title: str) -> str:
words = re.findall(r"[a-z0-9]+", title.lower())
if not words:
return "paper"
return "-".join(words[:5])
def normalize_text(value: str) -> str:
return re.sub(r"\s+", " ", value).strip()
def make_chunk(
*,
anchor: str,
text: str,
page_number: int,
) -> dict[str, Any]:
clean_text = text.strip()
excerpt = normalize_text(clean_text)[:280]
return {
"anchor": anchor,
"page_start": page_number,
"page_end": page_number,
"label": anchor,
"excerpt": excerpt or None,
"text": clean_text,
}
def extract_pages(source: str) -> list[dict[str, Any]]:
path = pathlib.Path(source).expanduser()
if not path.exists():
raise FileNotFoundError(f"Local source does not exist: {path}")
suffix = path.suffix.lower()
if suffix in TEXT_SUFFIXES:
text = path.read_text(encoding="utf-8").strip()
if not text:
raise RuntimeError(f"No extractable text found in text file: {path}")
return [make_chunk(anchor="p1", text=text, page_number=1)]
if suffix != ".pdf":
raise ValueError(
f"Unsupported local file type: {suffix or '<no suffix>'}. "
"Supported: .pdf, .txt, .md, .org"
)
try:
import pymupdf # type: ignore
except ImportError:
try:
import fitz as pymupdf # type: ignore
except ImportError as err:
raise RuntimeError(
"PyMuPDF is required for local PDF extraction. "
"Install `pymupdf` or provide pasted text / a local text file."
) from err
doc = pymupdf.open(path)
try:
pages = []
for page_number, page in enumerate(doc, start=1):
page_text = page.get_text("text").strip()
if page_text:
pages.append(make_chunk(anchor=f"p{page_number}", text=page_text, page_number=page_number))
finally:
doc.close()
if not pages:
raise RuntimeError(f"No extractable text found in PDF: {path}")
return pages
def extract_text(source: str) -> str:
pages = extract_pages(source)
text = "\n\n".join(page["text"] for page in pages if page.get("text"))
if not text.strip():
raise RuntimeError(f"No extractable text found in file: {source}")
return text
def resolve_save_path(save_path: str, title: str) -> pathlib.Path:
target = pathlib.Path(save_path).expanduser()
timestamp = datetime.now().strftime("%Y%m%dT%H%M%S")
default_name = f"{timestamp}--xray-{slugify_title(title)}__read.org"
if target.exists() and target.is_dir():
return target / default_name
if target.suffix:
return target
return target / default_name
def main() -> int:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest="command", required=True)
extract_parser = subparsers.add_parser("extract")
extract_parser.add_argument("--source", required=True)
save_parser = subparsers.add_parser("resolve-save")
save_parser.add_argument("--save-path", required=True)
save_parser.add_argument("--title", required=True)
args = parser.parse_args()
try:
if args.command == "extract":
sys.stdout.write(extract_text(args.source))
return 0
if args.command == "resolve-save":
resolved = resolve_save_path(args.save_path, args.title)
resolved.parent.mkdir(parents=True, exist_ok=True)
sys.stdout.write(str(resolved))
return 0
except Exception as exc: # pragma: no cover - CLI error path
print(str(exc), file=sys.stderr)
return 1
print(f"Unknown command: {args.command}", file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
Title: Robust Memory Routing for Long-Horizon Fault Diagnosis
Author: Mei Chen
Doctoral Dissertation
School of Mechanical Engineering
2023
Abstract
Fault diagnosis in long-horizon industrial monitoring streams is difficult because useful evidence is sparse and delayed. We develop a routing-based memory architecture that compresses repeated observations and preserves fault-relevant context. Experiments show consistent gains in F1 and latency over transformer and recurrent baselines.
Keywords: fault diagnosis; memory routing; industrial AI
1 Introduction
2 Method
3 Results
4 Discussion
References
题目:面向工业日志异常检测的稀疏记忆路由方法
作者:张三
学校:华东某大学
硕士学位论文
2024年
摘要
工业日志中的异常事件通常稀疏且容易被重复的正常模式淹没。本文提出一种稀疏记忆路由模型,先将长日志压缩为记忆槽,再选择与故障相关的槽输入分类器。实验结果表明,该方法在两个制造场景数据集上均优于 Transformer 基线,并降低了推理成本。
关键词:工业日志;异常检测;记忆路由
1 绪论
2 方法
3 实验结果
4 结论
参考文献
#!/usr/bin/env python3
"""Tests for the paper-pdf-normalizer helper script."""
from __future__ import annotations
import importlib.util
import json
import subprocess
import sys
from pathlib import Path
SKILL_DIR = Path(__file__).parent.parent
SCRIPTS_DIR = SKILL_DIR / "scripts"
SCRIPT = SCRIPTS_DIR / "normalize_paper.py"
WORKBENCH_IO_SCRIPT = SCRIPTS_DIR / "workbench_io.py"
FIXTURES_DIR = Path(__file__).parent / "fixtures"
SAMPLE_LOCAL_PAPER = FIXTURES_DIR / "sample_local_paper.pdf"
spec = importlib.util.spec_from_file_location("normalize_paper", SCRIPT)
assert spec is not None and spec.loader is not None
normalize_paper = importlib.util.module_from_spec(spec)
spec.loader.exec_module(normalize_paper)
def run_python_script(script: Path, *args: str) -> subprocess.CompletedProcess[str]:
return subprocess.run(
[sys.executable, str(script), *args],
capture_output=True,
text=True,
check=True,
)
def load_payload(*args: str) -> dict:
completed = run_python_script(SCRIPT, *args)
return json.loads(completed.stdout)
def test_local_pdf_normalizes_sample_fixture():
payload = load_payload(
"--source",
str(SAMPLE_LOCAL_PAPER),
"--fulltext",
"never",
)
assert payload["schema_version"] == "paper-record"
assert payload["status"] == "resolved"
assert payload["source"]["input_kind"] == "local_pdf"
assert payload["bibliography"]["title"] == "Sparse Memory Routing for Industrial Anomaly Detection"
assert [author["name"] for author in payload["bibliography"]["authors"]] == ["Lin Qiao", "Mira Patel"]
assert payload["bibliography"]["venue"] == "Workshop on Reliable Industrial AI 2025"
assert payload["bibliography"]["doi"] is None
assert payload["document"]["document_type"] == "conference-paper"
assert payload["content"]["full_text_included"] is False
assert payload["content"]["page_chunks"]
assert payload["content"]["page_chunks"][0]["anchor"] == "p1"
assert payload["content"]["page_chunks"][0]["text"] is None
def test_master_thesis_fixture_sets_thesis_type():
payload = normalize_paper.normalize_source(
str(FIXTURES_DIR / "master_thesis.txt"),
fulltext_mode="never",
)
assert payload["schema_version"] == "paper-record"
assert payload["status"] == "resolved"
assert payload["document"]["document_type"] == "thesis"
assert payload["document"]["degree_level"] == "master"
assert payload["document"]["language"] == "zh"
assert payload["bibliography"]["title"] == "面向工业日志异常检测的稀疏记忆路由方法"
assert payload["bibliography"]["keywords"] == ["工业日志", "异常检测", "记忆路由"]
assert any("绪论" in section for section in payload["content"]["sections"])
def test_doctoral_thesis_fixture_sets_doctor_level():
payload = normalize_paper.normalize_source(
str(FIXTURES_DIR / "doctor_thesis.txt"),
fulltext_mode="never",
)
assert payload["schema_version"] == "paper-record"
assert payload["status"] == "resolved"
assert payload["document"]["document_type"] == "thesis"
assert payload["document"]["degree_level"] == "doctor"
assert payload["bibliography"]["authors"][0]["name"] == "Mei Chen"
assert payload["content"]["summary"]
def test_normalized_json_is_passthrough(tmp_path: Path):
existing = normalize_paper.normalize_source(
str(FIXTURES_DIR / "doctor_thesis.txt"),
fulltext_mode="never",
)
json_path = tmp_path / "normalized.json"
json_path.write_text(json.dumps(existing, ensure_ascii=False), encoding="utf-8")
payload = normalize_paper.normalize_source(str(json_path))
assert payload == existing
def test_fulltext_prefer_keeps_page_chunk_text():
payload = normalize_paper.normalize_source(
str(SAMPLE_LOCAL_PAPER),
fulltext_mode="prefer",
)
assert payload["content"]["full_text_included"] is True
assert payload["content"]["page_chunks"][0]["text"]
def test_doi_source_uses_crossref_metadata(monkeypatch):
def fake_fetch_crossref_json(doi):
_ = doi
return {
"title": ["Attention Is All You Need"],
"author": [
{"given": "Ashish", "family": "Vaswani"},
{"given": "Noam", "family": "Shazeer"},
],
"issued": {"date-parts": [[2017, 6, 12]]},
"container-title": ["Neural Information Processing Systems"],
"publisher": "Curran Associates, Inc.",
"abstract": "We propose the Transformer architecture.",
"subject": ["Machine Learning", "Attention"],
"URL": "https://doi.org/10.5555/3295222.3295349",
"resource": {"primary": {"URL": "https://papers.example.org/transformer"}},
}
monkeypatch.setattr(normalize_paper, "fetch_crossref_json", fake_fetch_crossref_json)
payload = normalize_paper.normalize_source("10.5555/3295222.3295349", fulltext_mode="never")
assert payload["schema_version"] == "paper-record"
assert payload["source"]["input_kind"] == "doi"
assert payload["bibliography"]["doi"] == "10.5555/3295222.3295349"
assert payload["bibliography"]["title"] == "Attention Is All You Need"
assert [author["name"] for author in payload["bibliography"]["authors"]] == ["Ashish Vaswani", "Noam Shazeer"]
assert payload["bibliography"]["year"] == 2017
assert payload["bibliography"]["venue"] == "Neural Information Processing Systems"
assert payload["source"]["canonical_url"] == "https://doi.org/10.5555/3295222.3295349"
assert payload["status"] in {"resolved", "partial"}
def test_arxiv_source_uses_alphaxiv_when_available(monkeypatch):
def fake_fetch_json(url: str):
if url.endswith("/papers/v3/1706.03762"):
return {
"title": "Attention Is All You Need",
"abstract": "We propose the Transformer architecture.",
"citationBibtex": (
"@Article{Vaswani2017AttentionIA,\n"
" author = {Ashish Vaswani and Noam Shazeer},\n"
" booktitle = {Neural Information Processing Systems},\n"
" title = {Attention Is All You Need},\n"
" year = {2017}\n"
"}\n"
),
"versionId": "version-123",
}
if url.endswith("/papers/v3/version-123/overview/en"):
return {
"summary": {
"summary": "The Transformer replaces recurrence with attention.",
"originalProblem": ["RNNs train slowly."],
"solution": ["Use attention-only encoder-decoder stacks."],
"keyInsights": ["Parallel attention works."],
"results": ["Better BLEU with less training cost."],
},
"intermediateReport": "structured report",
"citations": [{"title": "Seq2Seq"}],
}
raise AssertionError(f"Unexpected URL: {url}")
def fake_fetch_text(url):
_ = url
return None
monkeypatch.setattr(normalize_paper, "fetch_json", fake_fetch_json)
monkeypatch.setattr(normalize_paper, "fetch_text", fake_fetch_text)
payload = normalize_paper.normalize_source("1706.03762", fulltext_mode="never")
assert payload["status"] == "resolved"
assert payload["source"]["input_kind"] == "arxiv_id"
assert payload["document"]["document_type"] == "preprint"
assert payload["arxiv_enhancement"]["alphaxiv_available"] is True
assert payload["content"]["summary"] == "The Transformer replaces recurrence with attention."
assert payload["bibliography"]["venue"] == "Neural Information Processing Systems"
def test_generic_page_without_pdf_returns_unresolved(monkeypatch):
def fake_http_get(url, binary=False):
_ = binary
return ("<html><title>Paper</title></html>", "text/html", url)
monkeypatch.setattr(normalize_paper, "http_get", fake_http_get)
payload = normalize_paper.normalize_source("https://example.com/paper", fulltext_mode="never")
assert payload["status"] == "unresolved"
assert payload["bibliography"]["title"] == "Paper"
assert payload["errors"]
def test_init_profile_writes_researcher_profile(tmp_path: Path):
profile_path = tmp_path / "researcher-profile.json"
payload = json.loads(
run_python_script(
WORKBENCH_IO_SCRIPT,
"init-profile",
"--path",
str(profile_path),
"--research-field",
"科技社会学",
"--core-question",
"平台化协作如何改变学术写作劳动?",
"--stage",
"文献梳理",
).stdout
)
assert payload["artifact_type"] == "researcher-profile"
assert payload["research_field"] == "科技社会学"
assert payload["core_question"] == "平台化协作如何改变学术写作劳动?"
assert profile_path.exists()
def test_save_artifact_writes_json_and_sidecar(tmp_path: Path):
workspace = tmp_path / "workspace"
payload_path = tmp_path / "payload.json"
payload_path.write_text(
json.dumps({"summary": "deep read result"}, ensure_ascii=False),
encoding="utf-8",
)
sidecar_path = tmp_path / "artifact.md"
sidecar_path.write_text("# Deep Read\n", encoding="utf-8")
result = json.loads(
run_python_script(
WORKBENCH_IO_SCRIPT,
"save-artifact",
"--workspace",
str(workspace),
"--artifact-type",
"paper-deep-read",
"--title",
"Attention Is All You Need",
"--payload-file",
str(payload_path),
"--sidecar-file",
str(sidecar_path),
"--sidecar-ext",
"md",
"--source-record",
"records/paper.json",
).stdout
)
artifact_path = Path(result["artifact_path"])
generated_sidecar = Path(result["sidecar_path"])
saved_payload = json.loads(artifact_path.read_text(encoding="utf-8"))
assert artifact_path.exists()
assert generated_sidecar.exists()
assert saved_payload["artifact_type"] == "paper-deep-read"
assert saved_payload["source_records"] == ["records/paper.json"]
assert saved_payload["payload"]["summary"] == "deep read result"
Related skills
FAQ
What modes does paper-workbench support?
Single-paper scan, deep-read, card, interpret, xray, json, and cross-paper synthesis and review modes.
What sources can it ingest?
arXiv IDs and URLs, AlphaXiv URLs, DOIs, local or remote PDFs, landing pages exposing a PDF, and existing paper-record or artifact JSON.