Skip to content

AbramHsu/vibeDistill

BranchesTags

Repository files navigation

Python 3.13 License MIT Platform GPU Support

🕷️ vibeDistill · 大V风格蒸馏框架

Scrape → Transcribe → Extract → Imitate
Turn any creator's content into reusable AI style prompts.
把任何创作者的风格，变成可复用的 AI 指令。

English · 中文

📋 Table of Contents / 目录

English 中文
What is vibeDistill 是什么
Architecture 项目架构
Quick Start 快速开始
Usage Flow 使用流程
Open Core & Closed Source 开源 vs 闭源
File Structure 文件结构
Tech Stack 技术栈
License 许可证
Contact 联系方式

What is vibeDistill

vibeDistill is an open-source framework for distilling content from your favorite creators. It processes videos through a pipeline of scraping → transcription → analysis → style imitation, turning a creator's unique "vibe" into reusable AI instructions.

Core Pipeline

Stage Function Technology
🕷️ Scrape Auto-browse creator homepage, scroll-load all videos, intercept API data Playwright
📝 Transcribe Convert video speech to text Whisper (base/medium/large-v3)
🧠 Analyze Extract core ideas per video, generate comprehensive opinion profile DeepSeek API
✍️ Imitate Rewrite your text in the creator's style using extracted style prompts DeepSeek API

Supported Platforms

  • Douyin (抖音) — Playwright browser automation
  • YouTube — yt-dlp download + comment extraction

Use Cases

  • Content creators looking for inspiration and style analysis
  • Writers who want to learn from top creator's narrative techniques
  • AI researchers studying style transfer and content distillation
  • Marketers analyzing successful creator content patterns

是什么

vibeDistill（大V风格蒸馏框架）是一个开源的内容蒸馏工具。把你关注的创作者的视频内容，通过"爬取 → 转写 → 分析 → 风格化"的流程，提炼出可复用的风格指令。

核心能力

阶段 功能 技术
🕷️ 爬取 自动浏览大V主页，滚动加载全部视频，拦截 API 获取数据 Playwright
📝 转写 将视频语音转为文字 Whisper (base/medium/large-v3)
🧠 分析 提取每个视频的核心观点，生成综合观点画像 DeepSeek API
✍️ 仿写 基于风格指令，用大V的风格改写你的文案 DeepSeek API

已支持平台

  • 抖音 — Playwright 浏览器自动化爬取
  • YouTube — yt-dlp 下载 + 评论提取

适用场景

  • 内容创作者寻找灵感和风格分析
  • 写作者学习头部创作者的叙事技巧
  • AI 研究人员研究风格迁移和内容蒸馏
  • 营销人员分析成功创作者的内容模式

Architecture Overview

flowchart LR
    A[Crawler / 爬虫] -->|Video file| B[Whisper Transcription / 转写]
    A -->|Metadata| C[(SQLite)]
    B -->|Text| D[DeepSeek Extraction / 观点提取]
    D --> C
    C -->|Insights + Style| E[DeepSeek Imitation / 仿写]
    F[prompts/ Style Files / 风格指令] --> E
    E -->|Generated Content| G[User / 用户]
Loading

Three independent stages, runnable separately:

  1. Crawler — Downloads videos + fetches comments, writes to DB
  2. Analyzer — Whisper transcription → DeepSeek insight extraction
  3. Imitator — Style imitation based on prompts/ directory (no DB needed)

Design Philosophy

  • Scrape-Analyze Separation: Crawling and analysis are independent stages. Crawl first, analyze later — you can crawl many creators before deciding which to analyze.
  • Incremental Updates: Only processes new videos (platform_video_id dedup). Supports crash recovery — already-downloaded videos are skipped on re-run.
  • Dual-Mode Crawling: API pagination + auto-scroll for latest videos; bookmarklet ID extraction for archival (videos beyond API limits).

项目架构

flowchart LR
    A[爬虫] -->|视频文件| B[Whisper 转写]
    A -->|元数据| C[(SQLite)]
    B -->|文本| D[DeepSeek 观点提取]
    D --> C
    C -->|观点+风格| E[DeepSeek 仿写]
    F[prompts/ 风格指令] --> E
    E -->|仿写文案| G[用户]
Loading

三个独立阶段，可分离运行：

  1. 爬虫 — 只负责下载视频 + 抓取评论，写入 DB
  2. 分析 — 将已下载的视频过 Whisper 转写 → DeepSeek 提取观点
  3. 仿写 — 基于风格指令 + 用户输入，生成仿写文案（无需 DB）

设计理念

  • 爬虫-分析分离：爬取和分析是独立阶段。先爬后分析——你可以先爬大量大V，再决定哪些要深入分析。
  • 增量更新：只处理新视频（以 platform_video_id 去重）。支持崩溃恢复——重跑时自动跳过已下载的视频。
  • 双模式爬取：API 分页 + 自动滚动获取最新视频；书签法提取 ID 可获取更早的存档视频（超越 API 分页限制）。

Quick Start

Prerequisites

  • Python 3.13+
  • Node.js (for YouTube crawler — JS challenge bypass)
  • GPU (optional, recommended: NVIDIA 6GB+ VRAM for faster Whisper analysis)

1. Clone

git clone https://github.com/AbramHsu/vibeDistill.git
cd vibeDistill

2. Create Virtual Environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

Full dependencies include Playwright, Whisper, Streamlit, DeepSeek SDK, etc. Installation takes ~5-10 minutes.

4. Initialize Browser (for Douyin crawler)

playwright install chromium

Optionally download a dedicated Chrome for Testing to avoid conflicts with your system Chrome. See Playwright browser docs.

5. Configure API Key

# DeepSeek API Key (required — for both analysis and imitation)
export DEEPSEEK_API_KEY="sk-your-key-here"

# DeepSeek model (optional, defaults to deepseek-chat)
export DEEPSEEK_MODEL="deepseek-chat"

6. Launch

# Option A: Streamlit UI (recommended)
.venv\Scripts\streamlit run app.py --server.port 8501

# Option B: CLI crawler
python crawl_big_v.py https://www.douyin.com/user/xxxxx

快速开始

前置环境

  • Python 3.13+
  • Node.js（YouTube 爬虫需要，用于绕过 JS challenge）
  • 显卡（可选，Whisper GPU 分析推荐 NVIDIA 6GB+ 显存）

1. 克隆

git clone https://github.com/AbramHsu/vibeDistill.git
cd vibeDistill

2. 创建虚拟环境

python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

3. 安装依赖

pip install -r requirements.txt

完整依赖包含 Playwright、Whisper、Streamlit、DeepSeek SDK 等，安装时间约 5-10 分钟。

4. 初始化浏览器（抖音爬虫用）

playwright install chromium

可选：下载项目专用的 Chrome for Testing（避免与系统 Chrome 冲突），参考 Playwright 浏览器管理文档。

5. 配置 API Key

# DeepSeek API Key（必填 — 观点提取 + 仿写都需要）
export DEEPSEEK_API_KEY="sk-your-key-here"

# DeepSeek 模型（可选，默认 deepseek-chat）
export DEEPSEEK_MODEL="deepseek-chat"

6. 启动

# 方式一：Streamlit 界面（推荐）
.venv\Scripts\streamlit run app.py --server.port 8501

# 方式二：CLI 直跑爬虫
python crawl_big_v.py https://www.douyin.com/user/xxxxx

Usage Flow

🕷️ Crawl Creator Videos

Streamlit UI:

  1. Open http://localhost:8501
  2. Click "➕ Add Creator"
  3. Enter Douyin homepage URL, click "Start Crawling"
  4. The crawler auto-scrolls, downloads videos, and fetches comments

CLI:

python crawl_big_v.py <Douyin homepage URL>

Note: Douyin Web API typically returns only the latest ~36 videos. For older videos, use the bookmarklet method to extract IDs manually. Supports incremental crawling — already-downloaded videos are auto-skipped.

📝 Analyze Videos

After crawling, click "🎯 Start Analysis" on the creator detail page.

Analysis pipeline:

  1. Whisper transcribes video speech to text
  2. DeepSeek extracts core insights (100-200 chars per video)
  3. Generates a comprehensive opinion profile for the creator

Analysis is a separate stage from crawling — triggered manually by the user.

✍️ Style Imitation

Select the "✍️ Imitate" page in the sidebar:

  1. Choose a style (requires a style prompt file in prompts/)
  2. Paste your original text
  3. AI rewrites it in the chosen creator's style

Style prompt files (prompts/*.md) are the core asset of this project. The local repo includes an example.md to demonstrate the format. Full style prompts (知行派, 觉醒派, 人间清醒) are available through the online service.

使用流程

🕷️ 爬取大V视频

Streamlit 界面操作：

  1. 打开 http://localhost:8501
  2. 点击 "➕ 添加大V"
  3. 输入抖音主页链接，点 "开始抓取"
  4. 爬虫自动滚动加载、下载视频、抓取评论

CLI 方式：

python crawl_big_v.py <抖音主页URL>

抖音 Web API 通常只返回最新 ~36 个视频。更老的视频可以通过书签法手动提取 ID。 支持增量爬取 — 已下载的视频自动跳过。

📝 分析视频

爬取完成后，在大V详情页点击 "🎯 开始分析"。

分析流程：

  1. Whisper 将视频语音转写为文字
  2. DeepSeek 从转写文本提取核心观点（100-200 字）
  3. 生成该大V的综合观点画像

分析是独立阶段，由用户手动触发。

✍️ 风格仿写

在侧边栏选择 "✍️ 仿写" 页面：

  1. 选择风格（需先有风格指令文件在 prompts/ 目录）
  2. 输入你的原文
  3. AI 按该风格改写

风格指令文件（prompts/*.md）是本项目的核心资产。 本地项目包含一个 example.md 示例文件展示格式。 完整的风格指令（知行派、觉醒派、人间清醒）可通过 在线服务 使用。

Open Core & Closed Source

This project follows an Open Core + SaaS model:

Component Status Description
🕷️ Crawler Framework Open Source Douyin/YouTube crawlers — anyone can use to crawl their own creators
📝 Whisper Transcription Open Source Speech-to-text logic, configurable model & GPU params
🧠 DeepSeek Analysis Open Source Generic DeepSeek API integration for insight extraction
✍️ Imitation UI (app.py) Open Source UI code that reads prompts/ and calls the API
💾 SQLite Storage Open Source Full DB schema and CRUD operations
📚 Documentation Open Source Architecture docs, user guide, deployment reference
🎨 Style Prompts (prompts/*.md) Closed Source 🔒 Core style assets distilled from hundreds of hours of video — available only via ECS online service

Why This Design?

Style prompts are data assets, not code assets — they represent 1000+ hours of video, dozens of GPU-hours, and multiple rounds of manual tuning.

You can use this framework to crawl any creator, run your own distillation pipeline, and generate your own style prompts. But if you want production-quality, ready-to-use styles, our online service has you covered.

开源 vs 闭源

本项目采用 Open Core + SaaS 模式：

组件 状态 说明
🕷️ 爬虫框架 开源 抖音/YouTube 爬虫，任何人都可以拿去爬自己的大V
📝 Whisper 转写 开源 语音转文字逻辑，可自行配置模型和 GPU 参数
🧠 DeepSeek 观点提取 开源 调用 DeepSeek API 的通用方法
✍️ 仿写页面（app.py） 开源 读取 prompts/ 目录并调 API 的界面代码
💾 存储方案（SQLite） 开源 完整 DB 结构和 CRUD
📚 文档 开源 架构文档、使用指南、部署参考
🎨 风格指令（prompts/*.md） 闭源 🔒 基于数百个视频蒸馏的核心风格资产，仅通过 ECS 在线服务 提供

为什么这样设计？

风格指令是数据资产而非代码资产 — 它们是 1000+ 小时视频、数十小时 GPU 算力、多次人工调优的成果。

你可以用这个框架爬取任何大V，跑你自己的蒸馏流程，生成你自己的风格指令。但如果你想要开箱即用的高质量风格，可以使用我们的在线服务。

File Structure

vibeDistill/
├── spiders/
│   ├── base.py               # Crawler abstract base class / 爬虫抽象基类
│   ├── douyin.py             # Douyin crawler / 抖音爬虫
│   └── youtube.py            # YouTube crawler / YouTube 爬虫
├── core/
│   ├── transcriber.py        # Whisper transcription / 语音转写
│   └── analyser.py           # DeepSeek analysis + imitation / 观点提取 + 仿写
├── db/
│   ├── schema.sql            # Database schema / 数据库表结构
│   └── repository.py         # Database CRUD / 数据库操作
├── prompts/
│   └── example.md            # Style prompt format example / 风格指令格式示例
├── app.py                    # Streamlit UI / 界面
├── crawl_big_v.py            # Douyin crawler CLI entry / 抖音爬虫入口
├── crawl_youtube.py          # YouTube crawler CLI entry / YouTube 爬虫入口
├── export_content.py         # Content export tool / 内容导出工具
├── reset_failed_downloads.py # Reset failed download flags / 重置下载失败标记
├── docs/
│   ├── user-guide.md         # User guide / 使用指南
│   └── deploy/
│       └── database-reference.md   # DB reference for ECS deployment / 部署参考
├── start-streamlit.example.ps1     # Streamlit launcher example
├── start-claude.example.ps1        # Claude Code launcher example
├── requirements.txt          # Python dependencies
└── README.md

Runtime-only files (gitignored / 本地运行产生)

downloads/        # Video cache / 视频缓存
chrome_data/      # Browser data (cookie persistence) / 浏览器数据
chrome-win64/     # Project-specific Chrome / 项目专用 Chrome
logs/             # Runtime logs / 运行日志
data/*.db         # SQLite databases / 数据库文件

Tech Stack

Technology Purpose
Python 3.13 Main language
Playwright Douyin browser automation
yt-dlp YouTube video download
Whisper (openai-whisper) Speech-to-text
DeepSeek API Insight extraction + style imitation
Streamlit Local UI
SQLite Data storage
FastAPI ECS online API service (closed source)

License

MIT License

Contact

Author: Abram

WeChat: Abram_Star
GitHub: @AbramHsu
Online Service: stormwriter.allaiin1.com

If this project helps you, feel free to reach out — I'd love to hear your story!

联系方式

作者：Abram

微信：Abram_Star
GitHub：@AbramHsu
在线服务：stormwriter.allaiin1.com

如果这个项目对你有帮助，欢迎联系交流！

vibeDistill — Turn any creator's vibe into your tool.
把大V的风格，变成你的工具。

GitHub · @AbramHsu · Online Service

About

Scrape creator videos into AI style prompts. 爬取大V视频，蒸馏为可复用的AI风格指令。

stormwriter.allaiin1.com/

Topics

python nlp open-source youtube ai web-scraping style-transfer video-download whisper content-analysis douyin streamlit playwright voice-transcription deepseek

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 1

initial version Latest
Jul 12, 2026

Packages

 
 
 

Contributors

Languages