
Xiaoyuzhou Asr
- 1 installs
- 1 repo stars
- Updated May 24, 2026
- worldwonderer/xiaoyuzhou-asr
Transcribe 小宇宙 (Xiaoyuzhou) podcast episodes to text using local GPU-accelerated Qwen3-ASR plus a companion xyz API server.
About
Fetches Xiaoyuzhou podcast metadata and audio via an xyz API server, then transcribes episodes locally with Qwen3-ASR (Metal/CUDA). A developer uses it to download and transcribe one or many 小宇宙 podcast episodes.
- Requires a running ultrazg/xyz API server plus ffmpeg and the Qwen3-ASR model
- Supports single-episode, search, and batch transcription workflows
Xiaoyuzhou Asr by the numbers
- 1 all-time installs (skills.sh)
- Ranked #1,983 of 2,715 Automation & Workflows skills by installs in the Skillselion catalog
- Data as of Jul 27, 2026 (Skillselion catalog sync)
npx skills add https://github.com/worldwonderer/xiaoyuzhou-asr --skill xiaoyuzhou-asrAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 1 |
|---|---|
| repo stars | ★ 1 |
| Last updated | May 24, 2026 |
| Repository | worldwonderer/xiaoyuzhou-asr ↗ |
What it does
Transcribe 小宇宙 (Xiaoyuzhou) podcast episodes to text using local GPU-accelerated Qwen3-ASR plus a companion xyz API server.
Files
xiaoyuzhou-asr
Transcribe 小宇宙 podcast episodes to text using local Qwen3-ASR (Metal/CUDA accelerated).
Required service: this skill does not call 小宇宙 directly. It requires a compatible
ultrazg/xyz API server to be installed and running.
Default base URL ishttp://localhost:23020; override it withXYZ_BASE_URL. Without
this service, login, search, episode lookup, and audio URL retrieval will not work.
Prerequisites
1. xyz API server running — fetches episode data and audio URLs from 小宇宙
git clone https://github.com/ultrazg/xyz.git && cd xyz && go run .
# Default port: 23020, change with -p2. ffmpeg — audio format conversion (brew install ffmpeg) 3. Qwen3-ASR model — download (HF Hub does NOT ship tokenizer.json):
python3 -c "
from huggingface_hub import snapshot_download
snapshot_download('Qwen/Qwen3-ASR-0.6B', local_dir='models/0.6B')
"4. qwen3-asr-rs — build from source:
git clone https://github.com/alan890104/qwen3-asr-rs.git && cd qwen3-asr-rs
cargo build --release --example local_transcribe5. tokenizer.json — auto-generated by the transcription script on first run (from vocab.json + merges.txt). No manual step needed.
Quick Start
# 1. Login (first time only, saves to ~/.xiaoyuzhou-asr.json)
python3 scripts/transcribe_podcast.py --login
# 2. Check all dependencies
python3 scripts/transcribe_podcast.py --check-env
# 3. Transcribe a single episode
python3 scripts/transcribe_podcast.py --keyword "早咖啡" -o output.md
# Or transcribe a shared episode URL
python3 scripts/transcribe_podcast.py --url "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID" -o output.mdCLI Commands
Authentication
# Interactive login — sends verification code to phone, saves tokens
python3 scripts/transcribe_podcast.py --loginDiscovery
# Search podcasts and show PID (for batch mode)
python3 scripts/transcribe_podcast.py --podcast-info --keyword "声动早咖啡"
# List recent episodes of a podcast
python3 scripts/transcribe_podcast.py --list-episodes --pid PODCAST_ID --count 10Transcription
# Single episode by keyword (picks first result)
python3 scripts/transcribe_podcast.py --keyword "关键词" -o output.md
# Single episode by EID
python3 scripts/transcribe_podcast.py --eid EPISODE_ID -o output.md
# Single episode by Xiaoyuzhou URL
python3 scripts/transcribe_podcast.py --url "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID" -o output.md
# Batch: transcribe 5 latest episodes of a podcast
python3 scripts/transcribe_podcast.py --pid PODCAST_ID --count 5 -o ./transcripts/
# With specific format
python3 scripts/transcribe_podcast.py --eid EPISODE_ID --format srt -o output.srtDiagnostics
# Check all dependencies (ffmpeg, xyz API, token, ASR binary, model)
python3 scripts/transcribe_podcast.py --check-envOutput Formats
| Format | Flag | Description |
|---|---|---|
| Markdown | --format markdown (default) | Metadata header + transcript |
| SRT | --format srt | Subtitles with estimated timestamps |
| Plain text | --format txt | Minimal header + transcript |
| JSON | --format json | Metadata + transcript as JSON |
Batch Mode
- Transcribes the N most recent episodes of a podcast (
--pid --count N) - Saves each episode as a separate file in the output directory
- Checkpoint/resume: skips episodes that already exist in the output directory
Configuration
Settings are resolved in priority order: CLI argument > Environment variable > Config file.
Config File (~/.xiaoyuzhou-asr.json)
Auto-created by --login. Can also store paths:
{
"token": "x-jike-access-token",
"refresh_token": "x-jike-refresh-token",
"model_dir": "/path/to/models/0.6B",
"asr_bin": "/path/to/local_transcribe"
}Environment Variables
| Variable | Description | Default |
|---|---|---|
XYZ_ACCESS_TOKEN | x-jike access token | — (required) |
XYZ_REFRESH_TOKEN | Refresh token for auto-renewal | — (optional) |
XYZ_BASE_URL | xyz API base URL | http://localhost:23020 |
XYZ_HTTP_TIMEOUT | xyz API request timeout in seconds | 15 |
XYZ_DOWNLOAD_TIMEOUT | Audio download timeout in seconds | 120 |
QWEN3_ASR_MODEL_DIR | Qwen3-ASR model directory | auto-detect |
QWEN3_ASR_BIN | local_transcribe binary path | auto-detect |
Token Management
--loginsaves tokens to config file automatically- If API returns 401, auto-refresh using refresh token
- Prompt user to login if no valid token
References
- xyz API endpoints and auth: references/xyz-api.md
- Qwen3-ASR usage and performance: references/qwen3-asr.md
Constraints
- MUST split audio into ≤3-minute segments for Metal GPU stability (auto-handled by script)
- Audio must be WAV 16kHz mono (auto-converted by script)
- tokenizer.json auto-generated on first run (from vocab.json + merges.txt)
- xyz API requires Chinese phone number (+86) login
- All processing is local — audio never leaves the machine
- Download retries up to 3 times on network failure
Script Reuse
This is a skill project, not a packaged Python library. Prefer the CLI above. Other scripts in this repository can still import scripts/transcribe_podcast.py directly:
from transcribe_podcast import (
search_episodes, transcribe_episode, format_output,
TranscriptionError, ApiError, TokenExpiredError,
)
try:
episodes, _ = search_episodes(token, "早咖啡")
episode, transcript, timings = transcribe_episode(token, eid, model_dir, asr_bin)
output = format_output(episode, transcript)
except TranscriptionError as e:
print(f"Error: {e}")name: Test
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: pip install pytest pytest-mock mypy
- name: Syntax check
run: python3 -m py_compile scripts/transcribe_podcast.py scripts/batch_transcribe.py tests/test_transcribe.py
- name: Run mypy
run: python3 -m mypy scripts/transcribe_podcast.py scripts/batch_transcribe.py tests/test_transcribe.py --ignore-missing-imports
- name: Run tests
run: python3 -m pytest tests -v
.omc
__pycache__/
*.pyc
.mypy_cache/
.pytest_cache/
*.egg-info/
dist/
build/
Changelog
v2.0.0 (2026-05-05)
Major feature release with significant improvements across all areas.
New Features
- Interactive login (
--login): phone number → verification code → save tokens to config file - Batch transcription (
--pid + --count): transcribe N most recent episodes of a podcast - Output formats (
--format markdown|srt|txt): SRT subtitles with segment-aware timestamps - Podcast discovery (
--podcast-info): search podcasts and display PID, subscriptions, episode count - Episode listing (
--list-episodes): browse recent episodes before batch transcription - Environment check (
--check-env): validate all dependencies (ffmpeg, xyz API, token, ASR, model) - Config file (
~/.xiaoyuzhou-asr.json): persistent token and path settings - Version flag (
--version)
Improvements
- Custom exception hierarchy (TranscriptionError/ApiError/TokenExpiredError/DependencyError/AudioError)
- Auto-detect ASR binary and model paths with env var override
- Download retry (3 attempts) for network resilience
- Batch mode checkpoint/resume (skips already-transcribed episodes)
- SRT timestamps use actual segment durations with character-proportional timing
- Transcription progress percentage display
- Cross-platform safe filename generation (sanitize_filename)
- Settings resolved from CLI arg > env var > config file
- Auto token refresh on 401
Infrastructure
- 36 unit tests (pytest)
- GitHub Actions CI (Python 3.10-3.13)
- mypy type checking (zero errors)
- pyproject.toml for standard Python packaging
v1.0.0 (2026-05-01)
Initial release.
MIT License
Copyright (c) 2026 pite
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
xiaoyuzhou-asr
小宇宙播客本地转录 Skill,适配 Claude Code / OpenClaw。通过兼容的 xyz API 服务获取节目元数据和音频 URL,使用 Qwen3-ASR 在本地完成语音识别,音频不会上传到外部服务。
重要前提:小宇宙搜索、登录、单集详情和音频 URL 获取都依赖正在运行的 ultrazg/xyz 兼容 API 服务。默认地址是http://localhost:23020,可用XYZ_BASE_URL指向其他地址;如果没有安装并启动该服务,小宇宙转录流程无法使用。
安装
方式一 直接告诉 Claude Code / OpenClaw:
安装这个 skill https://github.com/worldwonderer/xiaoyuzhou-asr方式二 命令行:
# Claude Code
cp -r xiaoyuzhou-asr ~/.claude/skills/xiaoyuzhou-asr
# OpenClaw
npx skills add worldwonderer/xiaoyuzhou-asr -y使用
安装后对 Claude 说:
- 「转录这集小宇宙播客 https://...」
- 「搜索早咖啡最新一期并转录」
- 「把这个单集 ID 转成文字」
自然语言即可触发,不需要记命令。
也可以直接运行脚本:
# 检查环境是否就绪
python3 scripts/transcribe_podcast.py --check-env --token YOUR_TOKEN
# 按小宇宙单集链接转录
python3 scripts/transcribe_podcast.py \
--token YOUR_TOKEN --url "https://www.xiaoyuzhoufm.com/episode/EPISODE_ID" -o output.md
# 按关键词搜索并转录第一个结果
python3 scripts/transcribe_podcast.py \
--token YOUR_TOKEN --keyword "早咖啡" -o output.md脚本也可以被同目录脚本复用(这不是打包发布的 Python library;推荐直接用 CLI):
from transcribe_podcast import search_episodes, run_transcription
episodes, _ = search_episodes(token, "早咖啡")
# 所有异常使用 TranscriptionError 体系,不会 sys.exitSkill 组成
| 文件 | 说明 |
|---|---|
SKILL.md | Skill 入口,定义触发条件和工作流 |
scripts/transcribe_podcast.py | 全流程脚本:搜索 → 下载 → 转换 → 分割 → 转录 → 输出 |
scripts/batch_transcribe.py | 批量发现并转录播客单集的辅助脚本 |
references/xyz-api.md | xyz API 端点、认证、响应格式参考 |
references/qwen3-asr.md | Qwen3-ASR 模型使用、音频要求、长音频处理参考 |
依赖
| 依赖 | 用途 | 安装 |
|---|---|---|
| ultrazg/xyz | 小宇宙 API 服务(需 +86 手机号登录) | git clone → go run . |
| Qwen3-ASR-0.6B | 语音识别模型(约 1.8GB) | huggingface_hub.snapshot_download |
| qwen3-asr-rs | Rust ASR 推理引擎 | cargo build --release --example local_transcribe |
| ffmpeg | 音频格式转换 | brew install ffmpeg |
详细安装步骤见 references/qwen3-asr.md。
输出格式
| Format | 参数 | 说明 |
|---|---|---|
| Markdown | --format markdown(默认) | 元数据标题 + 转录文本 |
| SRT | --format srt | 字幕文件,按分段时间估算句子时间戳 |
| TXT | --format txt | 简洁标题 + 转录文本 |
| JSON | --format json | 元数据和转录文本 JSON |
配置
常用环境变量:
| 变量 | 说明 | 默认值 |
|---|---|---|
XYZ_BASE_URL | xyz API 服务地址 | http://localhost:23020 |
XYZ_ACCESS_TOKEN | 小宇宙 access token | — |
XYZ_REFRESH_TOKEN | 自动刷新 token 用 | — |
XYZ_HTTP_TIMEOUT | xyz API 请求超时秒数 | 15 |
XYZ_DOWNLOAD_TIMEOUT | 音频下载超时秒数 | 120 |
QWEN3_ASR_MODEL_DIR | Qwen3-ASR 模型目录 | 自动探测 |
QWEN3_ASR_BIN | local_transcribe 路径 | 自动探测 |
平台支持
| 平台 | GPU 加速 | 备注 |
|---|---|---|
| Apple Silicon (M1/M2/M3/M4) | Metal | 音频超过 3 分钟会挂起,脚本自动分割为 ≤180s 片段 |
| NVIDIA (CUDA) | CUDA | 无时长限制 |
转录效果
以下是一期 11 分钟的中文播客(声动早咖啡)的实际转录结果,未经人工修改:
<details> <summary>展开查看完整转录</summary>
# 美军在对伊空袭中使用 AI 工具,泡泡玛特起诉 3D 打印公司拓竹科技
**节目**: 声动早咖啡
**日期**: 2026-03-02
**时长**: 11分14秒
**播放量**: 186,128
---
## 转录文本
用声音碰撞世界,生动活泼。嗨,我是早咖啡的兼职泽林,我们节目组正在寻找新伙伴。如果你对商业世界好奇,也喜欢声音这个媒介,欢迎去单击介绍里点招聘入口看看。那我们接下来就进入今天的节目吧。生动早咖啡与你轻松同步日常生活与商业世界。嗨,各位早上好呀,今天是二零二六年的三月三号,星期二。这里是生动早咖啡,我是来自生动活泼的梦一。美军在军事活动中是如何使用AI工具的?小米为什么不会量产自己的超跑概念车?泡泡玛特为什么起诉了三D打印公司拓竹科技?今天的早咖啡,我们将会为你带来这些问题的答案。海湾地区经济活动遭受冲击。根据路透社三月一号的报道,在美国和以色列对伊朗发动袭击之后,伊朗实施报复性攻击,波及了大部分海湾地区国家。当地的经济活动正在遭遇自新冠疫情以来最严重的冲击...</details>
# 美军在对伊空袭中使用 AI 工具,泡泡玛特起诉 3D 打印公司拓竹科技
**节目**: 声动早咖啡
**日期**: 2026-03-02
**时长**: 11分14秒
**播放量**: 186,128
---
## 转录文本
美军在军事活动中是如何使用AI工具的?小米为什么不会量产自己的
超跑概念车?泡泡玛特为什么起诉了三D打印公司拓竹科技?今天的
早咖啡,我们将会为你带来这些问题的答案。海湾地区经济活动遭受
冲击。根据路透社三月一号的报道,在美国和以色列对伊朗发动袭击
之后,伊朗实施报复性攻击,波及了大部分海湾地区国家。当地的经济
活动正在遭遇自新冠疫情以来最严重的冲击,机场被迫关闭,港口停运,
金融市场剧烈震荡...致谢
- ultrazg/xyz — 感谢该项目提供小宇宙 FM 非官方 API 能力,本 skill 通过运行兼容的 xyz API 服务获取节目元数据和音频 URL
- qwen3-asr-rs — Qwen3 ASR Rust 推理引擎(candle 框架)
Qwen3-ASR Reference (qwen3-asr-rs)
Pure-Rust speech-to-text engine for Qwen3-ASR models with Metal/CUDA acceleration.
Source: https://github.com/alan890104/qwen3-asr-rs | Crates.io: qwen3-asr
Table of Contents
Setup
Model Download
pip install huggingface_hub
# 0.6B — 1.7GB, fast, recommended for real-time
huggingface-cli download Qwen/Qwen3-ASR-0.6B --local-dir models
# 1.7B — 4.5GB, higher accuracy
huggingface-cli download Qwen/Qwen3-ASR-1.7B --local-dir models_1.7bBuild
git clone https://github.com/alan890104/qwen3-asr-rs.git
cd qwen3-asr-rs
# macOS (Metal GPU, default)
cargo build --release
# Linux/Windows (NVIDIA CUDA)
cargo build --release --no-default-features --features cuda
# CPU only
cargo build --release --no-default-featuresAs Rust Dependency
# macOS Metal
[dependencies]
qwen3-asr = "0.2"
# NVIDIA CUDA
[dependencies]
qwen3-asr = { version = "0.2", default-features = false, features = ["cuda"] }
# CPU
[dependencies]
qwen3-asr = { version = "0.2", default-features = false }Batch Transcription
use qwen3_asr::{AsrInference, TranscribeOptions, best_device};
let device = best_device(); // auto: CUDA → Metal → CPU
let engine = AsrInference::load("models/", device)?;
let result = engine.transcribe("audio.wav", TranscribeOptions::default())?;
println!("Language: {}", result.language);
println!("Text: {}", result.text);Auto-download from HuggingFace (with hub feature)
let engine = AsrInference::from_pretrained(
"Qwen/Qwen3-ASR-0.6B",
Path::new("models/"),
device,
)?;Streaming Transcription
For real-time low-latency transcription (~2s latency):
use qwen3_asr::StreamingOptions;
let mut state = engine.init_streaming(StreamingOptions::default());
for chunk in mic_chunks { // 16kHz f32 samples
if let Some(result) = engine.feed_audio(&mut state, &chunk)? {
println!("Live: {}", result.text);
}
}
let final_result = engine.finish_streaming(&mut state)?;
println!("Final: {}", final_result.text);Audio Requirements
- Format: WAV (via
houndcrate) or raw f32 samples - Sample rate: 16 kHz (resampled automatically via
rubato) - Channels: Mono
- Input: Local file path or
&[f32]sample array
Convert with ffmpeg:
ffmpeg -i input.m4a -ar 16000 -ac 1 output.wavPerformance
Apple Mac mini M4 (16GB), Metal backend:
| Model | Avg RTF | Load Time | Memory |
|---|---|---|---|
| 0.6B BF16 | 0.230 | 489ms | 1.9GB |
| 1.7B BF16 | 0.319 | 4250ms | 4.6GB |
RTF < 1.0 = faster than real-time. Both models run 3-7x faster than real-time on M4.
Long Audio Handling
Constraints
- Qwen3-ASR officially supports single speech up to 20 minutes
- Streaming mode: memory/latency grows with session duration
- <2 min: smooth
- ~10 min: ~1s/step, acceptable
- ~20 min: ~3-5s/step, upper limit
- \>20 min: not feasible
Strategy: Split and Batch
For podcast episodes (often 30-120 min), split audio at silence boundaries and batch-transcribe:
// Split with ffmpeg first, then:
for seg_file in segment_files {
let result = engine.transcribe(&seg_file, TranscribeOptions::default())?;
transcript.push(result.text);
}
let full_text = transcript.join("\n\n");Split with ffmpeg:
# Detect silence points
ffmpeg -i episode.wav -af "silencedetect=noise=-30dB:d=2" -f null - 2>&1 | grep silence_end
# Split at specific times
ffmpeg -i episode.wav -f segment -segment_times 120.5,240.3,360.1 \
-ar 16000 -ac 1 segment_%03d.wavStrategy: Streaming with Session Reset
For long-running streams, reset sessions at silence boundaries:
let mut state = engine.init_streaming(StreamingOptions::default());
loop {
let chunk = read_audio();
if vad_detects_silence(&chunk) {
let result = engine.finish_streaming(&mut state)?;
save(&result);
// Pass last ~200 chars as context for continuity
let ctx = result.text.chars().rev().take(200).collect::<String>();
let mut opts = StreamingOptions::default().with_initial_text(ctx);
state = engine.init_streaming(opts);
} else {
engine.feed_audio(&mut state, &chunk)?;
}
}xyz API Reference (小宇宙FM API)
Base URL: http://localhost:23020 (default port, configurable via -p flag)
Source: https://github.com/ultrazg/xyz
Table of Contents
Authentication
Send Verification Code
POST /sendCode
Content-Type: application/json
{"mobilePhoneNumber": "13111111111", "areaCode": "+86"}Login
POST /login
Content-Type: application/json
{"mobilePhoneNumber": "13111111111", "verifyCode": "1234", "areaCode": "+86"}Response contains x-jike-access-token and x-jike-refresh-token in data.*. Save both.
Refresh Token
POST /refresh_token
Content-Type: application/json
{"x-jike-access-token": "OLD_TOKEN", "x-jike-refresh-token": "OLD_REFRESH"}Call when any authenticated endpoint returns 401.
Search
POST /search
x-jike-access-token: TOKEN
Content-Type: application/json
{
"keyword": "search term",
"type": "ALL | PODCAST | EPISODE | USER",
"pid": "optional podcast id for searching within a podcast",
"loadMoreKey": {"loadMoreKey": 20, "searchId": "..."}
}Returns array of items with type field distinguishing PODCAST, EPISODE, USER results.
Episode Endpoints
All require x-jike-access-token header.
Episode Detail
POST /episode_detail
{"eid": "EPISODE_ID"}Key fields: title, description, shownotes, media.source.url, duration (seconds), podcast.title, pubDate, playCount, commentCount.
Episode List (by podcast)
POST /episode_list
{"pid": "PODCAST_ID", "order": "asc | desc", "loadMoreKey": {"pubDate":"...","id":"...","direction":"NEXT"}}Returns 20 episodes per page. Use loadMoreKey/loadNextKey from response for pagination.
Popular Episodes
POST /episode_list_by_filter
{"pid": "PODCAST_ID"}Playback Progress
POST /episode_play_progress
{"eid": "EPISODE_ID"}
POST /episode_play_progress_update
{"eid": "EPISODE_ID", "progress": 120.5}Podcast Endpoints
All require x-jike-access-token header.
Podcast Detail
POST /podcast_detail
{"pid": "PODCAST_ID"}Key fields: title, description, subscriptionCount, episodeCount, podcasters[], image.picUrl.
Podcast Info
POST /podcast_get_info
{"pid": "PODCAST_ID"}Related Podcasts
POST /podcast_related
{"pid": "PODCAST_ID"}Podcast Bulletin
POST /podcast_bulletin
{"pid": "PODCAST_ID"}Podcast Honor List
POST /podcast_honor_list
{"pid": "PODCAST_ID"}Common Response Fields
Episode Object
| Field | Type | Description |
|---|---|---|
eid | string | Episode ID |
pid | string | Parent podcast ID |
title | string | Episode title |
description | string | Plain text description |
shownotes | string | HTML formatted show notes |
duration | number | Duration in seconds |
media.source.url | string | Audio download URL (m4a) |
media.size | number | File size in bytes |
media.mimeType | string | e.g. "audio/mp4" |
pubDate | string | ISO 8601 date |
playCount | number | Play count |
commentCount | number | Comment count |
clapCount | number | Clap/like count |
podcast | object | Parent podcast info |
isFavorited | boolean | Favorited by user |
payType | string | "FREE" or paid |
Podcast Object
| Field | Type | Description |
|---|---|---|
pid | string | Podcast ID |
title | string | Podcast title |
author | string | Author name |
description | string | Description |
subscriptionCount | number | Subscribers |
episodeCount | number | Total episodes |
podcasters | array | Host info with uid, nickname, avatar |
image.picUrl | string | Cover image URL |
#!/usr/bin/env python3
"""
批量转录编排脚本
搜索小宇宙上 LLM/AI 相关播客,自动下载转录。
用法:
python3 scripts/batch_transcribe.py # 完整流程
python3 scripts/batch_transcribe.py --discover-only # 只搜索,不转录
python3 scripts/batch_transcribe.py --resume # 从检查点恢复
python3 scripts/batch_transcribe.py --min-plays 5000 # 降低播放量阈值
"""
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from transcribe_podcast import (
TranscriptionError,
TokenExpiredError,
episode_file_stem,
format_output,
get_all_episodes,
resolve_setting,
search_all_podcasts,
transcribe_episode,
_detect_asr_bin,
_detect_model_dir,
)
# --- Config ---
DEFAULT_KEYWORDS = [
"LLM",
"AI Agent",
"大模型",
"GPT",
"Claude",
"ChatGPT",
"OpenAI",
"Anthropic",
"AI应用",
"AI创业",
]
# Skip podcasts whose title matches these (traditional ML/DL)
EXCLUDE_PATTERNS = ["机器学习", "深度学习", "Deep Learning", "Machine Learning", "ML入门"]
DEFAULT_MIN_PLAYS = 10000
DEFAULT_OUTPUT_DIR = Path.home() / "xiaoyuzhou-transcripts"
CHECKPOINT_FILE = "batch-checkpoint.json"
def default_checkpoint() -> dict:
return {"discovered_podcasts": [], "discovered_episodes": [], "completed": [], "failed": []}
def _atomic_write(path: Path, data: dict) -> None:
"""Atomic JSON write: write to tmp, fsync, rename."""
tmp = path.with_suffix(".tmp")
tmp.write_text(json.dumps(data, indent=2, ensure_ascii=False), encoding="utf-8")
tmp.replace(path)
def load_checkpoint(output_dir: Path) -> dict:
path = output_dir / CHECKPOINT_FILE
if path.exists():
data = json.loads(path.read_text(encoding="utf-8"))
if isinstance(data, dict):
checkpoint = default_checkpoint()
checkpoint.update(data)
return checkpoint
return default_checkpoint()
def save_checkpoint(output_dir: Path, checkpoint: dict) -> None:
_atomic_write(output_dir / CHECKPOINT_FILE, checkpoint)
def is_excluded(title: str) -> bool:
lower = title.lower()
return any(p.lower() in lower for p in EXCLUDE_PATTERNS)
def discover_podcasts(token: str, keywords: list[str], min_plays: int) -> list[dict]:
"""Search for podcasts across all keywords, deduplicate, filter."""
seen_pids = set()
all_podcasts = []
for kw in keywords:
print(f"\n搜索关键词: {kw}")
podcasts = search_all_podcasts(token, kw, max_pages=3)
for p in podcasts:
pid = p.get("pid")
if not pid or pid in seen_pids:
continue
title = p.get("title", "")
if is_excluded(title):
print(f" 跳过 (传统ML/DL): {title}")
continue
seen_pids.add(pid)
all_podcasts.append(p)
sub_count = p.get("subscriptionCount", 0)
ep_count = p.get("episodeCount", 0)
print(f" 发现: {title} (订阅: {sub_count:,}, 集数: {ep_count})")
print(f"\n共发现 {len(all_podcasts)} 个播客 (去重后)")
return all_podcasts
def discover_episodes(token: str, podcasts: list[dict], min_plays: int) -> list[dict]:
"""Get all episodes from podcasts, filter by play count."""
all_episodes = []
seen_eids = set()
for p in podcasts:
pid = p.get("pid")
if not isinstance(pid, str) or not pid:
print(f"\n跳过无效播客记录: {p.get('title', '?')}")
continue
title = p.get("title", "")
print(f"\n获取播客单集: {title} ({pid})")
episodes = get_all_episodes(token, pid, max_pages=50)
qualifying = []
for ep in episodes:
eid = ep.get("eid")
if not eid or eid in seen_eids:
continue
seen_eids.add(eid)
play_count = ep.get("playCount", 0)
ep_title = ep.get("title", "")
duration = ep.get("duration", 0)
if play_count < min_plays:
continue
if duration < 60:
continue
qualifying.append(ep)
print(f" 共 {len(episodes)} 集, {len(qualifying)} 集符合条件 (>={min_plays:,} 播放, >1min)")
all_episodes.extend(qualifying)
all_episodes.sort(key=lambda e: e.get("playCount", 0), reverse=True)
print(f"\n总计 {len(all_episodes)} 集待转录")
return all_episodes
def batch_transcribe(
token: str,
episodes: list[dict],
checkpoint: dict,
output_dir: Path,
model_dir: str,
asr_bin: str,
keep_audio: bool = False,
) -> None:
"""Transcribe episodes with checkpoint resume."""
completed_eids = set(checkpoint.get("completed", []))
failed = checkpoint.get("failed", [])
remaining = [e for e in episodes if e["eid"] not in completed_eids]
if not remaining:
print("所有单集已完成转录。")
return
print(f"\n待转录: {len(remaining)} 集 (已完成: {len(completed_eids)}, 失败: {len(failed)})")
for i, ep in enumerate(remaining):
eid = ep["eid"]
title = ep.get("title", "未知")
play_count = ep.get("playCount", 0)
print(f"\n{'='*50}")
print(f"[{i+1}/{len(remaining)}] {title}")
print(f" 播放: {play_count:,}")
print(f"{'='*50}")
safe = episode_file_stem(ep, fallback=eid)
out_path = output_dir / f"{safe}.md"
if out_path.exists() and out_path.stat().st_size > 0:
print(f" 跳过 (文件已存在): {out_path}")
checkpoint["completed"].append(eid)
save_checkpoint(output_dir, checkpoint)
continue
try:
episode, transcript, timings = transcribe_episode(
token, eid, model_dir, asr_bin, keep_audio,
)
output = format_output(episode, transcript, timings)
output_dir.mkdir(parents=True, exist_ok=True)
out_path.write_text(output, encoding="utf-8")
print(f" 已保存: {out_path}")
checkpoint["completed"].append(eid)
save_checkpoint(output_dir, checkpoint)
except TokenExpiredError:
print("\n Token 过期,请重新登录后使用 --resume 恢复")
sys.exit(1)
except (TranscriptionError, Exception) as e:
print(f" 失败: {e}")
failed.append({"eid": eid, "title": title, "error": str(e)})
checkpoint["failed"] = failed
save_checkpoint(output_dir, checkpoint)
continue
def main():
parser = argparse.ArgumentParser(description="批量转录小宇宙 LLM/AI 播客")
parser.add_argument("--keywords", nargs="+", default=None,
help="搜索关键词 (默认使用内置列表)")
parser.add_argument("--min-plays", type=int, default=DEFAULT_MIN_PLAYS,
help=f"最低播放量 (默认 {DEFAULT_MIN_PLAYS:,})")
parser.add_argument("--output-dir", type=str, default=str(DEFAULT_OUTPUT_DIR),
help=f"输出目录 (默认 {DEFAULT_OUTPUT_DIR})")
parser.add_argument("--discover-only", action="store_true",
help="只搜索和发现,不转录")
parser.add_argument("--transcribe-only", action="store_true",
help="从检查点恢复,直接开始转录 (跳过搜索)")
parser.add_argument("--resume", action="store_true",
help="从检查点恢复 (等同于 --transcribe-only)")
parser.add_argument("--keep-audio", action="store_true",
help="保留下载的音频文件")
parser.add_argument("--model-dir", help="Qwen3-ASR 模型目录")
parser.add_argument("--asr-bin", help="qwen3-asr-rs 路径")
parser.add_argument("--token", help="Access token")
args = parser.parse_args()
token = resolve_setting(args.token, "XYZ_ACCESS_TOKEN", "token") or ""
if not token:
print("错误: 需要 access token,请先运行: python3 scripts/transcribe_podcast.py --login")
sys.exit(1)
model_dir = resolve_setting(args.model_dir, "QWEN3_ASR_MODEL_DIR", "model_dir") or _detect_model_dir()
asr_bin = resolve_setting(args.asr_bin, "QWEN3_ASR_BIN", "asr_bin") or _detect_asr_bin()
output_dir = Path(args.output_dir)
output_dir.mkdir(parents=True, exist_ok=True)
keywords = args.keywords or DEFAULT_KEYWORDS
print("=" * 50)
print("小宇宙播客批量转录")
print(f"关键词: {', '.join(keywords)}")
print(f"最低播放量: {args.min_plays:,}")
print(f"输出目录: {output_dir}")
print("=" * 50)
checkpoint = load_checkpoint(output_dir)
# --- Discover phase ---
if not args.transcribe_only and not args.resume:
# Step 1: Discover podcasts
if checkpoint.get("discovered_podcasts") and not args.discover_only:
print(f"\n检查点中已有 {len(checkpoint['discovered_podcasts'])} 个播客,跳过搜索")
print(" (使用 --transcribe-only 或删除检查点以重新搜索)")
podcasts = checkpoint["discovered_podcasts"]
else:
podcasts = discover_podcasts(token, keywords, args.min_plays)
checkpoint["discovered_podcasts"] = podcasts
# Step 2: Discover episodes
if checkpoint.get("discovered_episodes") and not args.discover_only:
print(f"\n检查点中已有 {len(checkpoint['discovered_episodes'])} 个单集,跳过枚举")
episodes = checkpoint["discovered_episodes"]
else:
episodes = discover_episodes(token, podcasts, args.min_plays)
checkpoint["discovered_episodes"] = episodes
save_checkpoint(output_dir, checkpoint)
if args.discover_only:
# Print summary
print(f"\n{'='*50}")
print(f"发现摘要:")
print(f" 播客数: {len(podcasts)}")
print(f" 符合条件的单集数: {len(episodes)}")
print(f"\nTop 10 单集 (按播放量):")
for i, ep in enumerate(episodes[:10]):
podcast_title = ep.get("podcast", {}).get("title", "?")
print(f" {i+1}. [{ep.get('playCount', 0):>8,} 播放] {ep.get('title', '?')} — {podcast_title}")
return
# --- Transcribe phase ---
episodes = checkpoint.get("discovered_episodes", [])
if not episodes:
print("没有待转录的单集。请先不带 --resume 运行以搜索。")
sys.exit(1)
batch_transcribe(
token, episodes, checkpoint, output_dir, model_dir, asr_bin, args.keep_audio,
)
# Final summary
checkpoint = load_checkpoint(output_dir)
completed = len(checkpoint.get("completed", []))
failed = len(checkpoint.get("failed", []))
total = len(checkpoint.get("discovered_episodes", []))
print(f"\n{'='*50}")
print(f"完成: {completed}/{total}, 失败: {failed}")
if failed:
print("失败列表:")
for f in checkpoint["failed"]:
print(f" - {f.get('title', '?')}: {f.get('error', '?')[:80]}")
print(f"输出目录: {output_dir}")
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""
小宇宙播客转录工具
从 小宇宙 FM 获取播客音频,使用 Qwen3-ASR 本地转录为文字。
用法:
python3 transcribe_podcast.py --token TOKEN --keyword "关键词"
python3 transcribe_podcast.py --token TOKEN --eid EPISODE_ID
python3 transcribe_podcast.py --token TOKEN --url https://www.xiaoyuzhoufm.com/episode/EPISODE_ID
python3 transcribe_podcast.py --check-env --token TOKEN
"""
import argparse
import json
import os
import re
import shutil
import subprocess
import sys
import tempfile
from pathlib import Path
from typing import Optional
from urllib.parse import parse_qs, unquote, urlparse
BASE_URL = os.environ.get("XYZ_BASE_URL", "http://localhost:23020")
AUDIO_DIR = Path(tempfile.gettempdir()) / "xiaoyuzhou-audio"
MAX_SEGMENT_SEC = 180 # 3 min — Metal GPU hangs on longer segments
HTTP_TIMEOUT_SEC = float(os.environ.get("XYZ_HTTP_TIMEOUT", "15"))
DOWNLOAD_TIMEOUT_SEC = float(os.environ.get("XYZ_DOWNLOAD_TIMEOUT", "120"))
USER_AGENT = "xiaoyuzhou-asr/2.0"
VERSION = "2.0.0"
def _is_local_host(hostname: Optional[str]) -> bool:
"""Return True when a hostname points at the local machine."""
if not hostname:
return False
host = hostname.strip("[]").lower()
return host in {"localhost", "127.0.0.1", "::1", "0.0.0.0"} or host.endswith(".localhost")
def _ensure_local_proxy_bypass(base_url: str) -> None:
"""Make urllib skip HTTP(S)_PROXY for a local xyz API endpoint."""
parsed = urlparse(base_url)
if not _is_local_host(parsed.hostname):
return
existing: list[str] = []
for key in ("NO_PROXY", "no_proxy"):
existing.extend(p.strip() for p in os.environ.get(key, "").split(",") if p.strip())
if "*" in existing:
return
seen = {p.lower() for p in existing}
for host in ("localhost", "127.0.0.1", "::1", "0.0.0.0"):
if host.lower() not in seen:
existing.append(host)
seen.add(host.lower())
value = ",".join(existing)
os.environ["NO_PROXY"] = value
os.environ["no_proxy"] = value
_ensure_local_proxy_bypass(BASE_URL)
# --- Auto-detect paths ---
def _detect_model_dir() -> str:
"""Detect Qwen3-ASR model directory."""
env = os.environ.get("QWEN3_ASR_MODEL_DIR")
if env and Path(env).exists():
return env
candidates = [
Path.home() / "qwen3-asr-models" / "0.6B",
Path.home() / "models" / "0.6B",
Path("/opt/qwen3-asr-models/0.6B"),
]
for c in candidates:
if c.exists():
return str(c)
return str(candidates[0]) # return default even if missing (for --check-env)
def _detect_asr_bin() -> str:
"""Detect qwen3-asr-rs local_transcribe binary."""
env = os.environ.get("QWEN3_ASR_BIN")
if env and Path(env).exists():
return env
# Check PATH
found = shutil.which("local_transcribe")
if found:
return found
candidates = [
Path.home() / "qwen3-asr-rs" / "target" / "release" / "examples" / "local_transcribe",
Path.home() / "src" / "qwen3-asr-rs" / "target" / "release" / "examples" / "local_transcribe",
]
for c in candidates:
if c.exists():
return str(c)
return str(candidates[0])
# --- Custom exceptions ---
class TranscriptionError(Exception):
"""Base exception for transcription errors."""
class ApiError(TranscriptionError):
"""xyz API call failed."""
def __init__(self, message: str, status_code: int = 0):
super().__init__(message)
self.status_code = status_code
class TokenExpiredError(ApiError):
"""Access token has expired and refresh failed."""
class DependencyError(TranscriptionError):
"""Required dependency is missing or misconfigured."""
class AudioError(TranscriptionError):
"""Audio processing error."""
def sanitize_filename(title: str, max_len: int = 50) -> str:
"""Create a safe cross-platform filename from a title."""
# Replace common separators with hyphens
name = re.sub(r"[/\\|:]", "-", title)
# Remove characters unsafe on any platform
name = re.sub(r'[<>"*?\x00-\x1f]', "", name)
# Collapse whitespace and hyphens
name = re.sub(r"[\s_]+", " ", name).strip()
# Truncate to max_len (keeping valid unicode chars)
if len(name) > max_len:
name = name[:max_len].rsplit(" ", 1)[0].rstrip("- ")
return name or "untitled"
def episode_file_stem(episode: dict, fallback: str = "episode") -> str:
"""Create a stable filename stem that avoids collisions across same-title episodes."""
title = str(episode.get("title") or fallback)
eid = str(episode.get("eid") or fallback)
pub_date = (episode.get("pubDate") or "")[:10]
date_part = sanitize_filename(pub_date, max_len=10) if pub_date else ""
eid_part = sanitize_filename(eid, max_len=32) if eid and eid not in title else ""
fixed_len = len(date_part) + len(eid_part)
separator_len = 3 * len([p for p in (date_part, eid_part) if p])
title_budget = max(20, 90 - fixed_len - separator_len)
title_part = sanitize_filename(title, max_len=title_budget)
return " - ".join(p for p in (date_part, title_part, eid_part) if p)
def output_extension(fmt: str) -> str:
"""Return the conventional output extension for a formatter name."""
return {
"markdown": "md",
"srt": "srt",
"txt": "txt",
"json": "json",
}.get(fmt, "txt")
def extract_episode_id_from_url(value: str) -> str:
"""Extract a Xiaoyuzhou episode id from a supported episode URL."""
parsed = urlparse(value.strip())
if not parsed.scheme or not parsed.netloc:
raise TranscriptionError(f"不是有效 URL: {value}")
host = parsed.netloc.lower()
if host != "xiaoyuzhoufm.com" and not host.endswith(".xiaoyuzhoufm.com"):
raise TranscriptionError(f"暂不支持的小宇宙链接域名: {parsed.netloc}")
query = parse_qs(parsed.query)
for key in ("eid", "episode_id", "episodeId"):
candidate = query.get(key, [""])[0].strip()
if candidate:
return candidate
segments = [s for s in parsed.path.split("/") if s]
for i, segment in enumerate(segments):
if segment in {"episode", "episodes"} and i + 1 < len(segments):
return unquote(segments[i + 1])
raise TranscriptionError(
"无法从链接中提取单集 ID;请使用形如 "
"https://www.xiaoyuzhoufm.com/episode/EPISODE_ID 的链接,或直接传 --eid"
)
def validate_asr_environment(model_dir: str, asr_bin: str) -> None:
"""Fail fast before downloading audio if local ASR dependencies are missing."""
if not asr_bin:
raise DependencyError("ASR 二进制路径为空,请设置 --asr-bin 或 QWEN3_ASR_BIN")
if not Path(asr_bin).exists() and not shutil.which(asr_bin):
raise DependencyError(
f"ASR 二进制不存在: {asr_bin}\n"
"请先编译: cd qwen3-asr-rs && cargo build --release --example local_transcribe"
)
model_path = Path(model_dir)
if not model_path.exists():
raise DependencyError(f"模型目录不存在: {model_dir}")
required = ["config.json", "vocab.json", "merges.txt"]
if not (model_path / "tokenizer.json").exists():
required.append("tokenizer_config.json")
missing = [name for name in required if not (model_path / name).exists()]
if missing:
raise DependencyError(f"模型目录不完整,缺少: {', '.join(missing)} ({model_dir})")
if not any(model_path.glob("*.safetensors")):
raise DependencyError(f"模型目录不完整,缺少 *.safetensors 权重文件 ({model_dir})")
CONFIG_PATH = Path.home() / ".xiaoyuzhou-asr.json"
def load_config() -> dict:
"""Load config from ~/.xiaoyuzhou-asr.json."""
if CONFIG_PATH.exists():
try:
return json.loads(CONFIG_PATH.read_text(encoding="utf-8")) # type: ignore[no-any-return]
except (json.JSONDecodeError, OSError):
pass
return {}
def save_config(config: dict) -> None:
"""Save config to ~/.xiaoyuzhou-asr.json."""
CONFIG_PATH.write_text(json.dumps(config, indent=2, ensure_ascii=False), encoding="utf-8")
try:
os.chmod(CONFIG_PATH, 0o600)
except (OSError, TypeError, ValueError):
pass
print(f"配置已保存到 {CONFIG_PATH}")
def read_json_response(resp, context: str) -> dict:
"""Read and validate a JSON object response from urllib."""
status = getattr(resp, "status", 0) or getattr(resp, "code", 0) or 0
body = resp.read()
if not body or not body.strip():
raise ApiError(f"{context} 响应为空 (HTTP {status})", status_code=status)
try:
parsed = json.loads(body)
except json.JSONDecodeError:
preview = body[:200].decode("utf-8", errors="replace")
raise ApiError(f"{context} 返回非 JSON 响应 (HTTP {status}): {preview}", status_code=status)
if not isinstance(parsed, dict):
raise ApiError(f"{context} JSON 响应不是对象 (HTTP {status})", status_code=status)
return parsed
def resolve_setting(cli_value: Optional[str], env_key: str, config_key: str) -> Optional[str]:
"""Resolve a setting from CLI arg > env var > config file."""
if cli_value:
return cli_value
env_val = os.environ.get(env_key)
if env_val:
return env_val
return load_config().get(config_key)
def do_login(base_url: str) -> None:
"""Interactive login flow: send code → verify → save tokens."""
import urllib.request
import urllib.error
phone = input("手机号 (含区号,如 13111111111): ").strip()
if not phone:
print("手机号不能为空")
return
area_code = "+86"
# Send verification code
req = urllib.request.Request(
f"{base_url}/sendCode",
data=json.dumps({"mobilePhoneNumber": phone, "areaCode": area_code}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT_SEC) as resp:
result = read_json_response(resp, "发送验证码")
print("验证码已发送,请查看手机短信")
except urllib.error.HTTPError as e:
body = e.read().decode()
print(f"发送验证码失败: {e.code} {body[:200]}")
return
except urllib.error.URLError as e:
print(f"无法连接 {base_url},请确认 xyz 服务已启动")
return
except ApiError as e:
print(f"发送验证码失败: {e}")
return
code = input("验证码: ").strip()
if not code:
print("验证码不能为空")
return
# Login
req = urllib.request.Request(
f"{base_url}/login",
data=json.dumps({"mobilePhoneNumber": phone, "areaCode": area_code, "verifyCode": code}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT_SEC) as resp:
result = read_json_response(resp, "登录")
except urllib.error.HTTPError as e:
body = e.read().decode()
print(f"登录失败: {e.code} {body[:200]}")
return
except ApiError as e:
print(f"登录失败: {e}")
return
data = result.get("data", {})
access_token = data.get("x-jike-access-token") or data.get("token", "")
refresh_token = data.get("x-jike-refresh-token") or data.get("refreshToken", "")
if not access_token:
print(f"登录响应异常: {json.dumps(result, ensure_ascii=False)[:200]}")
return
config = load_config()
config["token"] = access_token
if refresh_token:
config["refresh_token"] = refresh_token
save_config(config)
print("登录成功!")
# --- API ---
def api(endpoint: str, token: str, payload: dict, _retry: bool = True) -> dict:
"""Call xyz API endpoint with auto token refresh on 401."""
import urllib.request
import urllib.error
data = json.dumps(payload).encode()
req = urllib.request.Request(
f"{BASE_URL}{endpoint}",
data=data,
headers={
"Content-Type": "application/json",
"x-jike-access-token": token,
},
method="POST",
)
try:
with urllib.request.urlopen(req, timeout=HTTP_TIMEOUT_SEC) as resp:
return read_json_response(resp, f"API {endpoint}")
except urllib.error.HTTPError as e:
if e.code == 401 and _retry:
refresh_token = os.environ.get("XYZ_REFRESH_TOKEN") or load_config().get("refresh_token")
if refresh_token:
print(" Token 过期,自动刷新...")
try:
refresh_req = urllib.request.Request(
f"{BASE_URL}/refresh_token",
data=json.dumps({
"x-jike-access-token": token,
"x-jike-refresh-token": refresh_token,
}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(refresh_req, timeout=HTTP_TIMEOUT_SEC) as resp:
result = read_json_response(resp, "刷新 token")
new_token = (result.get("data", {}).get("x-jike-access-token")
or result.get("data", {}).get("token", ""))
if new_token:
os.environ["XYZ_ACCESS_TOKEN"] = new_token
config = load_config()
config["token"] = new_token
new_refresh = result.get("data", {}).get("x-jike-refresh-token")
if new_refresh:
config["refresh_token"] = new_refresh
os.environ["XYZ_REFRESH_TOKEN"] = new_refresh
save_config(config)
return api(endpoint, new_token, payload, _retry=False)
except Exception:
pass
raise TokenExpiredError(
f"Token 过期且刷新失败,请重新登录 (POST {BASE_URL}/login)",
status_code=401,
)
raise ApiError(f"API 错误 {e.code}: {endpoint}", status_code=e.code)
except urllib.error.URLError as e:
raise ApiError(
f"无法连接 xyz API ({BASE_URL}),请确认服务已启动: {e.reason}",
)
# --- Search and Episode ---
def search_episodes(token: str, keyword: str, limit: int = 5, loadMoreKey: Optional[dict] = None) -> tuple:
"""Search episodes by keyword. Returns (episodes, next_loadMoreKey)."""
payload: dict = {"keyword": keyword, "type": "EPISODE"}
if loadMoreKey:
payload["loadMoreKey"] = loadMoreKey
result = api("/search", token, payload)
episodes = []
for item in result.get("data", {}).get("data", []):
if item.get("type") == "EPISODE":
episodes.append(item)
next_key = result.get("data", {}).get("loadMoreKey")
return episodes[:limit], next_key
def search_all_episodes(token: str, keyword: str, max_pages: int = 10) -> list:
"""Search episodes with pagination. Returns all episodes up to max_pages."""
all_episodes = []
next_key = None
for page in range(max_pages):
episodes, next_key = search_episodes(token, keyword, limit=20, loadMoreKey=next_key)
all_episodes.extend(episodes)
if not next_key or not episodes:
break
return all_episodes
def get_episode_detail(token: str, eid: str) -> dict:
"""Get episode detail by eid."""
result = api("/episode_detail", token, {"eid": eid})
return result.get("data", {}).get("data", {}) # type: ignore[no-any-return]
def get_episode_list(token: str, pid: str, count: int = 5, loadMoreKey: Optional[dict] = None) -> tuple:
"""Get recent episodes of a podcast. Returns (episodes, next_loadMoreKey)."""
payload: dict = {"pid": pid, "order": "desc"}
if loadMoreKey:
payload["loadMoreKey"] = loadMoreKey
result = api("/episode_list", token, payload)
episodes = result.get("data", {}).get("data", []) # type: ignore[assignment]
next_key = result.get("data", {}).get("loadMoreKey")
return episodes[:count], next_key # type: ignore[no-any-return]
def get_all_episodes(token: str, pid: str, max_pages: int = 50) -> list:
"""Get all episodes of a podcast with pagination."""
all_episodes = []
next_key = None
for page in range(max_pages):
episodes, next_key = get_episode_list(token, pid, count=50, loadMoreKey=next_key)
all_episodes.extend(episodes)
if not next_key or not episodes:
break
print(f" 分页 {page+1}: 已获取 {len(all_episodes)} 集...")
return all_episodes
def search_podcasts(token: str, keyword: str, limit: int = 5, loadMoreKey: Optional[dict] = None) -> tuple:
"""Search podcasts by keyword. Returns (podcasts, next_loadMoreKey)."""
payload: dict = {"keyword": keyword, "type": "PODCAST"}
if loadMoreKey:
payload["loadMoreKey"] = loadMoreKey
result = api("/search", token, payload)
podcasts = []
for item in result.get("data", {}).get("data", []):
if item.get("type") == "PODCAST":
podcasts.append(item)
next_key = result.get("data", {}).get("loadMoreKey")
return podcasts[:limit], next_key
def search_all_podcasts(token: str, keyword: str, max_pages: int = 5) -> list:
"""Search podcasts with pagination. Returns all podcasts up to max_pages."""
all_podcasts = []
next_key = None
for page in range(max_pages):
podcasts, next_key = search_podcasts(token, keyword, limit=20, loadMoreKey=next_key)
all_podcasts.extend(podcasts)
if not next_key or not podcasts:
break
return all_podcasts
def get_podcast_detail(token: str, pid: str) -> dict:
"""Get podcast detail."""
result = api("/podcast_detail", token, {"pid": pid})
return result.get("data", {}).get("data", {}) # type: ignore[no-any-return]
def show_podcast_info(token: str, keyword: str) -> None:
"""Search and display podcast info for finding PIDs."""
podcasts, _ = search_podcasts(token, keyword)
if not podcasts:
print(f"未找到与 '{keyword}' 相关的播客")
return
print(f"\n找到 {len(podcasts)} 个播客:\n")
for i, p in enumerate(podcasts):
pid = p.get("pid", "?")
title = p.get("title", "未知")
sub_count = p.get("subscriptionCount", 0)
ep_count = p.get("episodeCount", 0)
author = p.get("author", "")
print(f" {i+1}. {title}")
if author:
print(f" 作者: {author}")
print(f" PID: {pid}")
print(f" 订阅: {sub_count:,} | 集数: {ep_count}")
print()
def list_episodes(token: str, pid: str, count: int = 10) -> None:
"""List recent episodes of a podcast."""
episodes, _ = get_episode_list(token, pid, count)
if not episodes:
print(f"播客 {pid} 没有单集")
return
print(f"\n播客 {pid} 最近 {len(episodes)} 集:\n")
for i, ep in enumerate(episodes):
eid = ep.get("eid", "?")
title = ep.get("title", "未知")
pub_date = ep.get("pubDate", "")[:10]
duration = ep.get("duration", 0)
mins, secs = divmod(duration, 60)
play_count = ep.get("playCount", 0)
print(f" {i+1}. {title}")
print(f" EID: {eid} | 日期: {pub_date} | 时长: {int(mins)}:{int(secs):02d}")
if play_count:
print(f" 播放: {play_count:,}")
print()
# --- Audio Processing ---
def download_audio(url: str, output_path: Path, max_retries: int = 3) -> Path:
"""Download audio file with retry."""
import urllib.error
import urllib.request
output_path.parent.mkdir(parents=True, exist_ok=True)
print(f" 下载音频: {url[:80]}...")
for attempt in range(1, max_retries + 1):
try:
tmp_path = output_path.with_suffix(output_path.suffix + ".part")
request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
with urllib.request.urlopen(request, timeout=DOWNLOAD_TIMEOUT_SEC) as resp:
with open(tmp_path, "wb") as f:
shutil.copyfileobj(resp, f)
tmp_path.replace(output_path)
size_mb = output_path.stat().st_size / 1024 / 1024
print(f" 已下载: {size_mb:.1f} MB")
return output_path
except (urllib.error.URLError, TimeoutError, OSError) as e:
if output_path.exists():
output_path.unlink()
part_path = output_path.with_suffix(output_path.suffix + ".part")
if part_path.exists():
part_path.unlink()
if attempt < max_retries:
print(f" 下载失败 (尝试 {attempt}/{max_retries}): {e},重试...")
else:
raise AudioError(f"下载音频失败 ({max_retries} 次尝试): {e}")
raise AudioError(f"下载音频失败: 未预期流程")
def convert_to_wav(input_path: Path, output_path: Path) -> Path:
"""Convert audio to WAV 16kHz mono using ffmpeg."""
if not shutil.which("ffmpeg"):
raise DependencyError("ffmpeg 未安装,请运行: brew install ffmpeg")
print(" 转换为 WAV 16kHz...")
cmd = [
"ffmpeg", "-y", "-i", str(input_path),
"-ar", "16000", "-ac", "1", "-f", "wav", str(output_path),
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode != 0:
raise AudioError(f"ffmpeg 错误: {result.stderr[:200]}")
return output_path
def get_duration_sec(wav_path: Path) -> float:
"""Get WAV duration in seconds using ffmpeg."""
result = subprocess.run(
["ffmpeg", "-i", str(wav_path)],
capture_output=True, text=True,
)
for line in result.stderr.splitlines():
if "Duration:" in line:
# Duration: 01:30:45.12, ...
part = line.split("Duration:")[1].split(",")[0].strip()
h, m, s = part.split(":")
return float(h) * 3600 + float(m) * 60 + float(s)
raise AudioError(f"无法获取音频时长: {wav_path}")
def split_audio(wav_path: Path, output_dir: Path) -> list[Path]:
"""Split audio at silence boundaries if > MAX_SEGMENT_SEC."""
duration = get_duration_sec(wav_path)
if duration <= MAX_SEGMENT_SEC:
print(f" 音频时长: {duration:.0f}s ({duration/60:.1f}min), 无需分割")
return [wav_path]
print(f" 音频时长: {duration:.0f}s ({duration/60:.1f}min), 需要分割")
# Detect silence points
result = subprocess.run(
["ffmpeg", "-i", str(wav_path),
"-af", "silencedetect=noise=-30dB:d=2",
"-f", "null", "-"],
capture_output=True, text=True,
)
ends = re.findall(r"silence_end:\s*([\d.]+)", result.stderr)
MIN_SEGMENT_SEC = 60
split_times: list[float] = []
for t in ends:
sec = float(t)
if sec < MIN_SEGMENT_SEC or sec >= duration - 1:
continue
if not split_times or sec - split_times[-1] >= MAX_SEGMENT_SEC * 0.8:
split_times.append(sec)
# Always fill gaps with evenly-spaced splits
t = MAX_SEGMENT_SEC
while t < duration:
if not any(abs(t - s) < 60 for s in split_times):
split_times.append(t)
t += MAX_SEGMENT_SEC
split_times.sort()
print(f" 分割点: {[f'{t:.0f}s' for t in split_times]}")
output_dir.mkdir(parents=True, exist_ok=True)
cmd = [
"ffmpeg", "-y", "-i", str(wav_path),
"-f", "segment",
"-segment_times", ",".join(f"{t:.3f}" for t in split_times),
"-ar", "16000", "-ac", "1",
str(output_dir / "seg_%03d.wav"),
]
subprocess.run(cmd, capture_output=True, text=True, check=True)
segments = sorted(output_dir.glob("seg_*.wav"))
print(f" 分割为 {len(segments)} 个片段")
return segments
# --- Tokenizer ---
def ensure_tokenizer(model_dir: str) -> None:
"""Build tokenizer.json from vocab.json + merges.txt if missing."""
tok_path = Path(model_dir) / "tokenizer.json"
if tok_path.exists():
return
print(" 生成 tokenizer.json...")
vocab_path = Path(model_dir) / "vocab.json"
merges_path = Path(model_dir) / "merges.txt"
config_path = Path(model_dir) / "tokenizer_config.json"
if not all(p.exists() for p in [vocab_path, merges_path, config_path]):
raise DependencyError(
f"模型目录不完整,缺少 vocab.json / merges.txt / tokenizer_config.json: {model_dir}"
)
with open(vocab_path) as f:
vocab_val = json.load(f)
with open(merges_path) as f:
merges_vec = [l for l in f.read().splitlines() if l and not l.startswith("#")]
with open(config_path) as f:
tok_cfg = json.load(f)
added_tokens = []
if "added_tokens_decoder" in tok_cfg:
entries = sorted(
[(int(k), v) for k, v in tok_cfg["added_tokens_decoder"].items()],
key=lambda x: x[0],
)
for id_, v in entries:
added_tokens.append({
"id": id_, "content": v["content"],
"single_word": False, "lstrip": False, "rstrip": False,
"normalized": False, "special": v.get("special", False),
})
tokenizer_json = {
"version": "1.0", "truncation": None, "padding": None,
"added_tokens": added_tokens,
"normalizer": {"type": "NFC"},
"pre_tokenizer": {"type": "Sequence", "pretokenizers": [
{"type": "Split", "pattern": {"Regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"}, "behavior": "Isolated", "invert": False},
{"type": "ByteLevel", "add_prefix_space": False, "trim_offsets": False, "use_regex": False},
]},
"post_processor": {"type": "ByteLevel", "add_prefix_space": False, "trim_offsets": False, "use_regex": False},
"decoder": {"type": "ByteLevel", "add_prefix_space": False, "trim_offsets": False, "use_regex": False},
"model": {"type": "BPE", "dropout": None, "unk_token": None,
"continuing_subword_prefix": "", "end_of_word_suffix": "",
"fuse_unk": False, "byte_fallback": False, "ignore_merges": False,
"vocab": vocab_val, "merges": merges_vec},
}
with open(tok_path, "w") as f:
json.dump(tokenizer_json, f)
print(" tokenizer.json 已生成")
# --- Transcription ---
_SENTENCE_END = frozenset("。!?.!?")
def stitch_segments(texts: list[str]) -> str:
"""Join segment texts, merging cut sentences at boundaries."""
if len(texts) <= 1:
return texts[0] if texts else ""
result = [texts[0]]
for text in texts[1:]:
prev = result[-1]
# If previous segment doesn't end with sentence punctuation, merge
if prev and prev[-1] not in _SENTENCE_END:
result[-1] = prev + text
else:
result.append(text)
return "\n\n".join(result)
def transcribe_segment(wav_path: Path, model_dir: str, asr_bin: str) -> str:
"""Transcribe a single WAV segment using qwen3-asr-rs."""
cmd = [asr_bin, model_dir, str(wav_path)]
print(f" 转录: {wav_path.name}...")
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=900)
except subprocess.TimeoutExpired:
raise TranscriptionError(f"转录超时 (900s): {wav_path.name}")
if result.returncode != 0:
raise TranscriptionError(f"转录错误: {result.stderr[:200]}")
lines = result.stdout.strip().splitlines()
for line in lines:
if line.startswith("Text :"):
return line[len("Text :"):].strip()
return result.stdout.strip()
def transcribe_segments(segments: list[Path], model_dir: str, asr_bin: str) -> tuple[str, list[tuple[str, float, float]]]:
"""Transcribe multiple segments. Returns (combined_text, segment_timings)."""
texts = []
timings = []
total = len(segments)
offset = 0.0
for i, seg in enumerate(segments):
pct = (i + 1) / total * 100
print(f" [{i+1}/{total}] ({pct:.0f}%)", end=" ")
text = transcribe_segment(seg, model_dir, asr_bin)
seg_dur = get_duration_sec(seg)
if text:
texts.append(text)
timings.append((text, offset, offset + seg_dur))
offset += seg_dur
return stitch_segments(texts), timings
# --- Output ---
def format_output(episode: dict, transcript: str, _segment_timings: Optional[list] = None) -> str:
"""Format transcript as markdown."""
title = episode.get("title", "未知标题")
podcast = episode.get("podcast", {})
podcast_title = podcast.get("title", "未知节目")
pub_date = episode.get("pubDate") or ""
duration = episode.get("duration") or 0
mins, secs = divmod(duration, 60)
play_count = episode.get("playCount") or 0
lines = [
f"# {title}",
"",
f"**节目**: {podcast_title}",
f"**日期**: {pub_date[:10] if pub_date else '未知'}",
f"**时长**: {int(mins)}分{int(secs)}秒",
]
if play_count:
lines.append(f"**播放量**: {play_count:,}")
lines += [
"",
"---",
"",
"## 转录文本",
"",
transcript,
]
return "\n".join(lines)
def _srt_time(sec: float) -> str:
"""Format seconds as SRT timestamp HH:MM:SS,mmm."""
h, remainder = divmod(int(sec), 3600)
m, s = divmod(remainder, 60)
ms = int((sec % 1) * 1000)
return f"{h:02d}:{m:02d}:{s:02d},{ms:03d}"
def format_srt(episode: dict, transcript: str, segment_timings: Optional[list] = None) -> str:
"""Format transcript as SRT subtitles with segment-aware timestamps."""
duration = episode.get("duration", 0)
if segment_timings:
# Use actual segment boundaries for accurate timestamps
lines = []
idx = 1
for seg_text, seg_start, seg_end in segment_timings:
sentences = re.split(r"(?<=[。!?\.\!])\s*", seg_text)
sentences = [s.strip() for s in sentences if s.strip()]
if not sentences:
continue
seg_dur = seg_end - seg_start
total_chars = sum(len(s) for s in sentences)
if total_chars == 0:
continue
char_offset = 0.0
for sentence in sentences:
char_frac = len(sentence) / total_chars
s_start = seg_start + char_offset * seg_dur
s_end = seg_start + (char_offset + char_frac) * seg_dur
lines.append(f"{idx}\n{_srt_time(s_start)} --> {_srt_time(s_end)}\n{sentence}")
idx += 1
char_offset += char_frac
return "\n\n".join(lines)
# Fallback: estimate by evenly distributing across duration
sentences = re.split(r"(?<=[。!?\.\!])\s*", transcript)
sentences = [s.strip() for s in sentences if s.strip()]
if not sentences or duration == 0:
return f"1\n00:00:00,000 --> {_srt_time(duration)}\n{transcript}\n"
time_per_sentence = duration / len(sentences)
lines = []
for i, sentence in enumerate(sentences):
start_sec = i * time_per_sentence
end_sec = (i + 1) * time_per_sentence
lines.append(f"{i+1}\n{_srt_time(start_sec)} --> {_srt_time(end_sec)}\n{sentence}")
return "\n\n".join(lines)
def format_txt(episode: dict, transcript: str, _segment_timings: Optional[list] = None) -> str:
"""Format transcript as plain text."""
title = episode.get("title", "未知标题")
podcast = episode.get("podcast", {})
podcast_title = podcast.get("title", "未知节目")
pub_date = episode.get("pubDate") or ""
header = f"{title}"
if podcast_title:
header += f" | {podcast_title}"
if pub_date:
header += f" | {pub_date[:10]}"
return f"{header}\n\n{transcript}"
def format_json(episode: dict, transcript: str, _segment_timings: Optional[list] = None) -> str:
"""Format transcript as JSON."""
return json.dumps({
"title": episode.get("title") or "",
"podcast": (episode.get("podcast") or {}).get("title") or "",
"date": (episode.get("pubDate") or "")[:10],
"duration": episode.get("duration") or 0,
"play_count": episode.get("playCount") or 0,
"transcript": transcript,
}, ensure_ascii=False, indent=2)
FORMATTERS = {
"markdown": format_output,
"srt": format_srt,
"txt": format_txt,
"json": format_json,
}
# --- Environment Check ---
def check_env(token: Optional[str] = None) -> bool:
"""Check all dependencies and report status. Returns True if all pass."""
checks = []
all_ok = True
# 1. ffmpeg
ffmpeg_path = shutil.which("ffmpeg")
if ffmpeg_path:
checks.append(("ffmpeg", True, ffmpeg_path))
else:
checks.append(("ffmpeg", False, "未安装"))
all_ok = False
# 2. xyz API
import urllib.request
import urllib.error
try:
req = urllib.request.Request(
f"{BASE_URL}/", method="GET",
)
with urllib.request.urlopen(req, timeout=3) as resp:
checks.append(("xyz API", True, f"{BASE_URL} (响应 HTTP {resp.status})"))
except urllib.error.HTTPError as e:
checks.append(("xyz API", True, f"{BASE_URL} (响应 HTTP {e.code})"))
except Exception:
checks.append(("xyz API", False, f"{BASE_URL} (未响应)"))
all_ok = False
# 3. Access token
actual_token = token or os.environ.get("XYZ_ACCESS_TOKEN")
if actual_token:
# Try to validate by searching
try:
api("/search", actual_token, {"keyword": "test", "type": "EPISODE"})
checks.append(("Access Token", True, "有效"))
except TokenExpiredError:
checks.append(("Access Token", False, "已过期,需重新登录"))
all_ok = False
except ApiError:
checks.append(("Access Token", False, "验证失败"))
all_ok = False
else:
checks.append(("Access Token", False, "未设置 (XYZ_ACCESS_TOKEN 环境变量或 --token)"))
all_ok = False
# 4. Refresh token
refresh = os.environ.get("XYZ_REFRESH_TOKEN") or load_config().get("refresh_token")
if refresh:
checks.append(("Refresh Token", True, "已设置"))
else:
checks.append(("Refresh Token", False, "未设置 (可选,用于自动续期)"))
# 5. ASR binary
asr_bin = _detect_asr_bin()
if Path(asr_bin).exists() or shutil.which(asr_bin):
checks.append(("qwen3-asr-rs", True, asr_bin))
else:
checks.append(("qwen3-asr-rs", False, f"未找到: {asr_bin}"))
all_ok = False
# 6. Model directory
model_dir = _detect_model_dir()
model_path = Path(model_dir)
if model_path.exists():
required_files = ["config.json", "vocab.json", "merges.txt"]
if not (model_path / "tokenizer.json").exists():
required_files.append("tokenizer_config.json")
missing = [f for f in required_files if not (model_path / f).exists()]
if not any(model_path.glob("*.safetensors")):
missing.append("*.safetensors")
if missing:
checks.append(("ASR Model", False, f"模型不完整,缺少: {', '.join(missing)}"))
all_ok = False
else:
tokenizer_ok = (model_path / "tokenizer.json").exists()
note = "" if tokenizer_ok else " (tokenizer.json 将在首次运行时自动生成)"
checks.append(("ASR Model", True, f"{model_dir}{note}"))
else:
checks.append(("ASR Model", False, f"目录不存在: {model_dir}"))
all_ok = False
# Print results
print("\n=== 环境检查 ===\n")
for name, ok, detail in checks:
icon = "✓" if ok else "✗"
print(f" [{icon}] {name}: {detail}")
print()
if all_ok:
print("所有检查通过,可以开始转录。")
else:
print("部分检查未通过,请根据上述提示修复。")
return all_ok
# --- Cleanup ---
def cleanup_audio(m4a_path: Path, wav_path: Path, seg_dir: Path) -> None:
"""Clean up temporary audio files."""
for p in [m4a_path, wav_path]:
p.unlink(missing_ok=True)
if seg_dir.exists():
for f in seg_dir.iterdir():
if f.is_file():
f.unlink()
try:
seg_dir.rmdir()
except OSError:
pass # not empty, leave it
# --- Main ---
def transcribe_episode(token: str, eid: str, model_dir: str, asr_bin: str, keep_audio: bool = False) -> tuple[dict, str, list]:
"""Transcribe a single episode. Returns (episode_meta, transcript_text, segment_timings)."""
validate_asr_environment(model_dir, asr_bin)
episode = get_episode_detail(token, eid)
title = episode.get("title", "未知")
print(f"\n单集: {title}")
media = episode.get("media", {})
audio_url = media.get("source", {}).get("url") or episode.get("enclosure", {}).get("url")
if not audio_url:
raise TranscriptionError("未找到音频链接")
size_mb = media.get("size", 0) / 1024 / 1024
print(f"音频: {size_mb:.1f} MB, {media.get('mimeType', 'unknown')}")
AUDIO_DIR.mkdir(parents=True, exist_ok=True)
safe_stem = episode_file_stem(episode, fallback=eid)
m4a_path = AUDIO_DIR / f"{safe_stem}.m4a"
wav_path = AUDIO_DIR / f"{safe_stem}.wav"
seg_dir = AUDIO_DIR / f"{safe_stem}_segments"
try:
download_audio(audio_url, m4a_path)
convert_to_wav(m4a_path, wav_path)
segments = split_audio(wav_path, seg_dir)
ensure_tokenizer(model_dir)
print("\n开始转录...")
transcript, timings = transcribe_segments(segments, model_dir, asr_bin)
return episode, transcript, timings
finally:
if not keep_audio:
cleanup_audio(m4a_path, wav_path, seg_dir)
def run_transcription(args: argparse.Namespace) -> str:
"""Core transcription logic. Returns formatted output. Raises on error."""
token = resolve_setting(args.token, "XYZ_ACCESS_TOKEN", "token") or ""
if not token:
raise ApiError("需要 access token (--login 登录, 或设置 --token / XYZ_ACCESS_TOKEN)")
model_dir = resolve_setting(args.model_dir, "QWEN3_ASR_MODEL_DIR", "model_dir") or _detect_model_dir()
asr_bin = resolve_setting(args.asr_bin, "QWEN3_ASR_BIN", "asr_bin") or _detect_asr_bin()
fmt = args.format or "markdown"
formatter = FORMATTERS.get(fmt)
if not formatter:
raise TranscriptionError(f"不支持的格式: {fmt},可选: {', '.join(FORMATTERS.keys())}")
# Batch mode: --pid + --count
if args.pid:
count = args.count or 3
print(f"批量转录播客 {args.pid} 最近 {count} 集...")
episodes_meta, _ = get_episode_list(token, args.pid, count)
if not episodes_meta:
raise TranscriptionError(f"播客 {args.pid} 没有可转录的单集")
# Checkpoint: skip already-transcribed episodes
out_dir = Path(args.output) if args.output else None
skipped = 0
if out_dir and (out_dir.is_dir() or not out_dir.suffix):
ext = output_extension(fmt)
remaining = []
for ep in episodes_meta:
safe = episode_file_stem(ep, fallback=ep.get("eid", "ep-unknown"))
ep_path = out_dir / f"{safe}.{ext}"
if ep_path.exists() and ep_path.stat().st_size > 0:
print(f" 跳过 (已存在): {ep.get('title', '?')}")
skipped += 1
else:
remaining.append(ep)
episodes_meta = remaining
if skipped:
print(f" 断点续传: 跳过 {skipped} 个已完成,剩余 {len(episodes_meta)} 个")
if not episodes_meta:
print("所有单集已完成转录。")
return ""
results = []
for i, ep in enumerate(episodes_meta):
eid = ep.get("eid")
if not eid:
print(f" 跳过 (缺少 EID): {ep.get('title', '?')}")
continue
print(f"\n{'='*40}")
print(f"批量进度: [{i+1+skipped}/{count}]")
print(f"{'='*40}")
episode, transcript, timings = transcribe_episode(
token, eid, model_dir, asr_bin, args.keep_audio,
)
output = formatter(episode, transcript, timings)
results.append((episode, output))
if out_dir:
if out_dir.is_dir() or not out_dir.suffix:
out_dir.mkdir(parents=True, exist_ok=True)
safe = episode_file_stem(episode, fallback=str(eid) or f"ep{i}")
ext = output_extension(fmt)
ep_path = out_dir / f"{safe}.{ext}"
ep_path.write_text(output, encoding="utf-8")
print(f" 已保存: {ep_path}")
return "\n\n---\n\n".join(r[1] for r in results)
# Single episode mode
if args.eid:
eid = args.eid
elif getattr(args, "url", None):
eid = extract_episode_id_from_url(args.url)
elif args.keyword:
print(f"搜索: {args.keyword}")
episodes, _ = search_episodes(token, args.keyword)
if not episodes:
raise TranscriptionError(f"未找到与 '{args.keyword}' 相关的单集")
print(f"找到 {len(episodes)} 个单集,选择第一个:")
print(f" {episodes[0].get('title', '?')}")
eid = episodes[0]["eid"]
else:
raise TranscriptionError("需要 --eid、--url、--keyword 或 --pid")
episode, transcript, timings = transcribe_episode(
token, eid, model_dir, asr_bin, args.keep_audio,
)
return formatter(episode, transcript, timings)
def main():
parser = argparse.ArgumentParser(description="小宇宙播客转录工具")
parser.add_argument("--token", help="x-jike-access-token (或设置 XYZ_ACCESS_TOKEN)")
parser.add_argument("--keyword", help="搜索关键词")
parser.add_argument("--eid", help="单集 ID (可替代关键词搜索)")
parser.add_argument("--url", help="小宇宙单集链接 (可替代 --eid)")
parser.add_argument("--pid", help="播客 ID (批量模式,配合 --count)")
parser.add_argument("--podcast-info", action="store_true", help="搜索播客并显示 PID 等信息")
parser.add_argument("--list-episodes", action="store_true", help="列出播客最近单集 (配合 --pid)")
parser.add_argument("--count", type=int, default=3, help="批量转录集数 (默认 3)")
parser.add_argument("--model-dir", help="Qwen3-ASR 模型目录 (或设置 QWEN3_ASR_MODEL_DIR)")
parser.add_argument("--asr-bin", help="qwen3-asr-rs local_transcribe 路径 (或设置 QWEN3_ASR_BIN)")
parser.add_argument("--format", choices=["markdown", "srt", "txt", "json"], default="markdown",
help="输出格式 (默认 markdown)")
parser.add_argument("--output", "-o", help="输出文件/目录路径 (默认 stdout)")
parser.add_argument("--keep-audio", action="store_true", help="保留下载的音频文件")
parser.add_argument("--check-env", action="store_true", help="检查所有依赖是否就绪")
parser.add_argument("--login", action="store_true", help="交互式登录并保存 token")
parser.add_argument("--version", action="version", version=f"%(prog)s {VERSION}")
args = parser.parse_args()
try:
if args.check_env:
token = resolve_setting(args.token, "XYZ_ACCESS_TOKEN", "token")
ok = check_env(token)
sys.exit(0 if ok else 1)
if args.login:
do_login(BASE_URL)
return
if args.podcast_info:
if not args.keyword:
parser.error("--podcast-info 需要配合 --keyword 指定搜索关键词")
token = resolve_setting(args.token, "XYZ_ACCESS_TOKEN", "token") or ""
if not token:
raise ApiError("需要 access token (--token 或 XYZ_ACCESS_TOKEN 环境变量)")
show_podcast_info(token, args.keyword)
return
if args.list_episodes:
if not args.pid:
parser.error("--list-episodes 需要配合 --pid 指定播客 ID")
token = resolve_setting(args.token, "XYZ_ACCESS_TOKEN", "token") or ""
if not token:
raise ApiError("需要 access token (--token 或 XYZ_ACCESS_TOKEN 环境变量)")
list_episodes(token, args.pid, args.count or 10)
return
if not args.eid and not args.url and not args.keyword and not args.pid:
parser.error("需要 --eid、--url、--keyword 或 --pid (或使用 --check-env / --podcast-info)")
output = run_transcription(args)
if args.output:
out_path = Path(args.output)
# Batch mode already handles per-file saving
if not (args.pid and (out_path.is_dir() or not out_path.suffix)):
Path(args.output).write_text(output, encoding="utf-8")
print(f"\n已保存到: {args.output}")
else:
print("\n" + output)
print("\n完成!")
except TranscriptionError as e:
print(f"\n错误: {e}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":
main()
#!/usr/bin/env python3
"""Unit tests for transcribe_podcast.py (no external dependencies)."""
import json
import os
import sys
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch, MagicMock
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "scripts"))
from transcribe_podcast import (
ApiError,
AudioError,
DependencyError,
FORMATTERS,
TokenExpiredError,
TranscriptionError,
_detect_asr_bin,
_detect_model_dir,
_ensure_local_proxy_bypass,
episode_file_stem,
extract_episode_id_from_url,
stitch_segments,
check_env,
format_output,
format_json,
format_srt,
format_txt,
get_all_episodes,
get_episode_list,
get_episode_detail,
get_podcast_detail,
load_config,
output_extension,
read_json_response,
resolve_setting,
run_transcription,
sanitize_filename,
save_config,
search_all_episodes,
search_all_podcasts,
search_episodes,
search_podcasts,
transcribe_segments,
validate_asr_environment,
)
MOCK_EPISODE = {
"title": "测试单集标题",
"eid": "test123",
"podcast": {"title": "测试播客"},
"pubDate": "2026-05-05T10:00:00Z",
"duration": 3661,
"playCount": 1000,
"media": {"source": {"url": "https://example.com/audio.m4a"}, "size": 10485760},
}
class TestExceptionHierarchy(unittest.TestCase):
def test_base_exception(self):
with self.assertRaises(TranscriptionError):
raise TranscriptionError("test")
def test_api_error(self):
with self.assertRaises(ApiError):
raise ApiError("test", status_code=500)
def test_token_expired_is_api_error(self):
with self.assertRaises(ApiError):
raise TokenExpiredError("expired", status_code=401)
def test_dependency_error(self):
with self.assertRaises(TranscriptionError):
raise DependencyError("missing")
def test_audio_error(self):
with self.assertRaises(TranscriptionError):
raise AudioError("failed")
class TestFormatMarkdown(unittest.TestCase):
def test_basic_format(self):
result = format_output(MOCK_EPISODE, "转录文本内容")
self.assertIn("# 测试单集标题", result)
self.assertIn("**节目**: 测试播客", result)
self.assertIn("**日期**: 2026-05-05", result)
self.assertIn("**时长**: 61分1秒", result)
self.assertIn("**播放量**: 1,000", result)
self.assertIn("## 转录文本", result)
self.assertIn("转录文本内容", result)
def test_no_play_count(self):
ep = {**MOCK_EPISODE, "playCount": 0}
result = format_output(ep, "text")
self.assertNotIn("播放量", result)
def test_missing_fields(self):
result = format_output({}, "text")
self.assertIn("未知标题", result)
self.assertIn("未知节目", result)
class TestFormatSrt(unittest.TestCase):
def test_basic_srt(self):
result = format_srt(MOCK_EPISODE, "第一句话。第二句话。第三句话。")
self.assertIn("-->", result)
self.assertIn("第一句话", result)
# Should have 3 numbered entries
self.assertTrue(result.startswith("1\n"))
self.assertIn("\n\n2\n", result)
self.assertIn("\n\n3\n", result)
def test_srt_with_segment_timings(self):
timings = [
("第一句。第二句。", 0.0, 120.0),
("第三句。", 120.0, 300.0),
]
result = format_srt({"duration": 300}, "unused", timings)
# Should have 3 entries
self.assertTrue(result.startswith("1\n"))
self.assertIn("\n\n2\n", result)
self.assertIn("\n\n3\n", result)
# First sentence starts at 0
self.assertIn("00:00:00,000", result)
# Third sentence starts at 120s
self.assertIn("00:02:00", result)
def test_empty_transcript(self):
result = format_srt({"duration": 120}, "")
self.assertIn("-->", result)
def test_zero_duration(self):
result = format_srt({"duration": 0}, "一些文字。")
self.assertIn("一些文字", result)
class TestFormatTxt(unittest.TestCase):
def test_basic_txt(self):
result = format_txt(MOCK_EPISODE, "转录内容")
self.assertEqual(result.split("\n\n")[0], "测试单集标题 | 测试播客 | 2026-05-05")
self.assertIn("转录内容", result)
def test_no_podcast(self):
result = format_txt({"title": "T"}, "text")
self.assertTrue(result.startswith("T"))
class TestFormattersRegistry(unittest.TestCase):
def test_all_formats_registered(self):
self.assertEqual(set(FORMATTERS.keys()), {"markdown", "srt", "txt", "json"})
def test_each_formatter_callable(self):
for name, fn in FORMATTERS.items():
result = fn(MOCK_EPISODE, "test text")
self.assertIsInstance(result, str, f"Formatter {name} did not return str")
def test_output_extensions(self):
self.assertEqual(output_extension("markdown"), "md")
self.assertEqual(output_extension("srt"), "srt")
self.assertEqual(output_extension("txt"), "txt")
self.assertEqual(output_extension("json"), "json")
self.assertEqual(output_extension("unknown"), "txt")
class TestFormatJson(unittest.TestCase):
def test_json_output(self):
result = format_json(MOCK_EPISODE, "转录内容")
parsed = json.loads(result)
self.assertEqual(parsed["title"], "测试单集标题")
self.assertEqual(parsed["podcast"], "测试播客")
self.assertEqual(parsed["date"], "2026-05-05")
self.assertEqual(parsed["duration"], 3661)
self.assertEqual(parsed["transcript"], "转录内容")
def test_json_missing_fields(self):
result = format_json({}, "text")
parsed = json.loads(result)
self.assertEqual(parsed["title"], "")
class TestProxyBypass(unittest.TestCase):
def test_local_xyz_base_url_adds_no_proxy_entries(self):
with patch.dict(os.environ, {"HTTP_PROXY": "http://127.0.0.1:8001"}, clear=True):
_ensure_local_proxy_bypass("http://localhost:23020")
no_proxy = os.environ.get("NO_PROXY", "")
self.assertIn("localhost", no_proxy)
self.assertIn("127.0.0.1", no_proxy)
def test_remote_base_url_keeps_existing_proxy_config(self):
with patch.dict(os.environ, {"NO_PROXY": "example.com"}, clear=True):
_ensure_local_proxy_bypass("https://api.example.com")
self.assertEqual(os.environ.get("NO_PROXY"), "example.com")
class TestPathDetection(unittest.TestCase):
def test_model_dir_env_override(self):
with patch.dict(os.environ, {"QWEN3_ASR_MODEL_DIR": "/fake/path"}):
with patch.object(Path, "exists", return_value=True):
self.assertEqual(_detect_model_dir(), "/fake/path")
def test_asr_bin_env_override(self):
with patch.dict(os.environ, {"QWEN3_ASR_BIN": "/fake/bin"}):
with patch.object(Path, "exists", return_value=True):
self.assertEqual(_detect_asr_bin(), "/fake/bin")
class TestApiFunctions(unittest.TestCase):
@patch("transcribe_podcast.api")
def test_search_episodes(self, mock_api):
mock_api.return_value = {
"data": {"data": [
{"type": "EPISODE", "eid": "1", "title": "Ep1"},
{"type": "PODCAST", "pid": "p1"},
{"type": "EPISODE", "eid": "2", "title": "Ep2"},
], "loadMoreKey": {"loadMoreKey": 20, "searchId": "123"}}
}
episodes, next_key = search_episodes("token", "test", limit=2)
self.assertEqual(len(episodes), 2)
self.assertEqual(episodes[0]["eid"], "1")
self.assertIsNotNone(next_key)
@patch("transcribe_podcast.api")
def test_get_episode_detail(self, mock_api):
mock_api.return_value = {"data": {"data": {"eid": "abc", "title": "Test"}}}
result = get_episode_detail("token", "abc")
self.assertEqual(result["title"], "Test")
@patch("transcribe_podcast.api")
def test_get_episode_list(self, mock_api):
mock_api.return_value = {"data": {"data": [
{"eid": "1"}, {"eid": "2"}, {"eid": "3"},
], "loadMoreKey": {"direction": "NEXT", "id": "abc", "pubDate": "2026-01-01"}}}
episodes, next_key = get_episode_list("token", "pid", count=2)
self.assertEqual(len(episodes), 2)
self.assertIsNotNone(next_key)
class TestCheckEnv(unittest.TestCase):
@patch("urllib.request.urlopen")
@patch("transcribe_podcast.shutil.which")
@patch("transcribe_podcast.Path.exists")
def test_check_env_all_missing(self, mock_exists, mock_which, mock_urlopen):
import urllib.error
mock_which.return_value = None
mock_exists.return_value = False
mock_urlopen.side_effect = urllib.error.URLError("refused")
result = check_env(None)
self.assertFalse(result)
@patch("transcribe_podcast.api")
@patch("urllib.request.urlopen")
@patch("transcribe_podcast.shutil.which")
@patch("transcribe_podcast._detect_asr_bin", return_value="/bin/local_transcribe")
@patch("transcribe_podcast._detect_model_dir", return_value="/model")
def test_check_env_treats_root_404_as_running(
self, _mock_model_dir, _mock_asr_bin, mock_which, mock_urlopen, mock_api
):
import urllib.error
mock_which.return_value = "/bin/ffmpeg"
mock_urlopen.side_effect = urllib.error.HTTPError(
url="http://localhost:23020/", code=404, msg="Not Found", hdrs=None, fp=None
)
mock_api.return_value = {"data": {"data": []}}
with patch("transcribe_podcast.Path.exists", return_value=True):
with patch("transcribe_podcast.Path.glob", return_value=[Path("model.safetensors")]):
result = check_env("token")
self.assertTrue(result)
class TestSanitizeFilename(unittest.TestCase):
def test_basic(self):
self.assertEqual(sanitize_filename("hello world"), "hello world")
def test_special_chars(self):
result = sanitize_filename("a/b:c|d")
self.assertNotIn("/", result)
self.assertNotIn(":", result)
self.assertNotIn("|", result)
def test_empty(self):
self.assertEqual(sanitize_filename(""), "untitled")
def test_truncation(self):
long_title = "x" * 200
result = sanitize_filename(long_title, max_len=50)
self.assertLessEqual(len(result), 50)
def test_chinese(self):
result = sanitize_filename("中文标题测试")
self.assertEqual(result, "中文标题测试")
def test_strip_whitespace(self):
self.assertEqual(sanitize_filename(" spaces "), "spaces")
class TestEpisodeFileStem(unittest.TestCase):
def test_includes_date_title_and_eid(self):
result = episode_file_stem(MOCK_EPISODE)
self.assertIn("2026-05-05", result)
self.assertIn("测试单集标题", result)
self.assertIn("test123", result)
def test_same_title_different_eids_do_not_collide(self):
ep1 = {"title": "同名标题", "eid": "eid-a", "pubDate": "2026-01-01T00:00:00Z"}
ep2 = {"title": "同名标题", "eid": "eid-b", "pubDate": "2026-01-01T00:00:00Z"}
self.assertNotEqual(episode_file_stem(ep1), episode_file_stem(ep2))
class TestEpisodeUrlParsing(unittest.TestCase):
def test_extract_from_episode_path(self):
url = "https://www.xiaoyuzhoufm.com/episode/abc123"
self.assertEqual(extract_episode_id_from_url(url), "abc123")
def test_extract_from_query(self):
url = "https://www.xiaoyuzhoufm.com/episode?eid=abc123"
self.assertEqual(extract_episode_id_from_url(url), "abc123")
def test_rejects_unsupported_domain(self):
with self.assertRaises(TranscriptionError):
extract_episode_id_from_url("https://example.com/episode/abc123")
class TestNewApiFunctions(unittest.TestCase):
@patch("transcribe_podcast.api")
def test_search_podcasts(self, mock_api):
mock_api.return_value = {
"data": {"data": [
{"type": "PODCAST", "pid": "p1", "title": "Pod1"},
{"type": "EPISODE", "eid": "e1"},
], "loadMoreKey": None}
}
podcasts, next_key = search_podcasts("token", "test")
self.assertEqual(len(podcasts), 1)
self.assertEqual(podcasts[0]["pid"], "p1")
self.assertIsNone(next_key)
@patch("transcribe_podcast.api")
def test_get_podcast_detail(self, mock_api):
mock_api.return_value = {"data": {"data": {"pid": "abc", "title": "Podcast"}}}
result = get_podcast_detail("token", "abc")
self.assertEqual(result["title"], "Podcast")
class FakeResponse:
def __init__(self, body: bytes, status: int = 200):
self._body = body
self.status = status
def read(self):
return self._body
class TestReadJsonResponse(unittest.TestCase):
def test_valid_json_object(self):
result = read_json_response(FakeResponse(b'{"ok": true}'), "测试")
self.assertEqual(result, {"ok": True})
def test_empty_body_raises_api_error(self):
with self.assertRaises(ApiError) as ctx:
read_json_response(FakeResponse(b'', status=200), "登录")
self.assertIn("响应为空", str(ctx.exception))
def test_non_json_body_raises_api_error(self):
with self.assertRaises(ApiError) as ctx:
read_json_response(FakeResponse(b'not json', status=200), "登录")
self.assertIn("非 JSON", str(ctx.exception))
class TestConfigFile(unittest.TestCase):
def setUp(self):
self.tmpdir = tempfile.mkdtemp()
self.config_path = Path(self.tmpdir) / "test-config.json"
def tearDown(self):
import shutil
shutil.rmtree(self.tmpdir, ignore_errors=True)
@patch("transcribe_podcast.CONFIG_PATH")
def test_load_nonexistent(self, mock_path):
mock_path.__str__ = lambda s: "/nonexistent/path.json"
mock_path.exists.return_value = False
config = load_config()
self.assertEqual(config, {})
@patch("transcribe_podcast.CONFIG_PATH")
def test_save_and_load(self, mock_path):
mock_path.__str__ = lambda s: str(self.config_path)
mock_path.exists.return_value = True
mock_path.write_text = lambda data, encoding: self.config_path.write_text(data, encoding=encoding)
mock_path.read_text = lambda encoding: self.config_path.read_text(encoding=encoding)
save_config({"token": "test123", "model_dir": "/tmp/models"})
config = load_config()
self.assertEqual(config["token"], "test123")
self.assertEqual(config["model_dir"], "/tmp/models")
def test_resolve_setting_cli_wins(self):
with patch.dict(os.environ, {"TEST_VAR": "env_val"}):
result = resolve_setting("cli_val", "TEST_VAR", "key")
self.assertEqual(result, "cli_val")
def test_resolve_setting_env_fallback(self):
with patch.dict(os.environ, {"TEST_VAR": "env_val"}):
result = resolve_setting(None, "TEST_VAR", "key")
self.assertEqual(result, "env_val")
@patch("transcribe_podcast.load_config")
def test_resolve_setting_config_fallback(self, mock_config):
mock_config.return_value = {"my_key": "config_val"}
result = resolve_setting(None, "NONEXISTENT_VAR", "my_key")
self.assertEqual(result, "config_val")
def test_resolve_setting_none(self):
result = resolve_setting(None, "NONEXISTENT_VAR", "missing_key")
self.assertIsNone(result)
def test_save_config_locks_down_permissions(self):
with patch("transcribe_podcast.CONFIG_PATH", self.config_path):
with patch("transcribe_podcast.os.chmod") as mock_chmod:
save_config({"token": "secret"})
mock_chmod.assert_called_once_with(self.config_path, 0o600)
class TestStitchSegments(unittest.TestCase):
def test_single_segment(self):
self.assertEqual(stitch_segments(["完整句子。"]), "完整句子。")
def test_empty(self):
self.assertEqual(stitch_segments([]), "")
def test_merge_cut_sentence(self):
result = stitch_segments([
"领跑汽车的董事长在车展",
"他们计划将海外销量的占比提升到四成。",
])
self.assertIn("车展他们计划", result)
self.assertNotIn("车展\n\n他们", result)
def test_keep_complete_sentences(self):
result = stitch_segments([
"第一段完整句子。",
"第二段完整句子。",
])
self.assertEqual(result, "第一段完整句子。\n\n第二段完整句子。")
def test_mixed_boundaries(self):
result = stitch_segments([
"完整段落。接下来被切断",
"的部分继续。新的完整句。",
"最后一段。",
])
self.assertIn("切断的部分继续", result)
self.assertIn("完整句。\n\n最后一段", result)
def test_exclamation_mark_is_boundary(self):
result = stitch_segments(["太好了!", "新的内容"])
self.assertIn("太好了!\n\n新的内容", result)
class TestTranscribeSegments(unittest.TestCase):
@patch("transcribe_podcast.get_duration_sec", return_value=10.0)
@patch("transcribe_podcast.transcribe_segment", side_effect=["", "第二句。"])
def test_empty_segment_still_advances_timing(self, _mock_transcribe, _mock_duration):
segments = [Path("seg_000.wav"), Path("seg_001.wav")]
transcript, timings = transcribe_segments(segments, "/model", "/bin")
self.assertEqual(transcript, "第二句。")
self.assertEqual(timings, [("第二句。", 10.0, 20.0)])
class TestValidateAsrEnvironment(unittest.TestCase):
def test_valid_environment(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
model = root / "model"
model.mkdir()
for name in ["config.json", "vocab.json", "merges.txt", "tokenizer_config.json"]:
(model / name).write_text("{}", encoding="utf-8")
(model / "model.safetensors").write_text("weights", encoding="utf-8")
asr_bin = root / "local_transcribe"
asr_bin.write_text("#!/bin/sh\n", encoding="utf-8")
validate_asr_environment(str(model), str(asr_bin))
def test_missing_asr_binary_fails_before_download(self):
with tempfile.TemporaryDirectory() as tmp:
model = Path(tmp) / "model"
model.mkdir()
with self.assertRaises(DependencyError):
validate_asr_environment(str(model), str(Path(tmp) / "missing"))
class TestRunTranscription(unittest.TestCase):
@patch("transcribe_podcast.transcribe_episode")
def test_url_mode_extracts_eid(self, mock_transcribe):
mock_transcribe.return_value = (MOCK_EPISODE, "转录文本", [])
args = type("Args", (), {
"token": "token",
"model_dir": "/model",
"asr_bin": "/bin",
"format": "markdown",
"pid": None,
"count": 3,
"output": None,
"keep_audio": False,
"eid": None,
"url": "https://www.xiaoyuzhoufm.com/episode/abc123",
"keyword": None,
})()
output = run_transcription(args)
self.assertIn("转录文本", output)
mock_transcribe.assert_called_once_with("token", "abc123", "/model", "/bin", False)
if __name__ == "__main__":
unittest.main()