Skip to content

Commit 843bb8f

Browse files
saudadez21saudadez21
authored
docs(readme): improve documentation structure and fix processors config typo (#156)
* docs(notes): optimize qidian notes content * docs(readme): refine main README structure and add English version * docs(readme): fix minor typo in README * docs(guide): correct typo in processors configuration documentation
1 parent bd41d58 commit 843bb8f

9 files changed

Lines changed: 322 additions & 209 deletions

File tree

‎README.en.md‎

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# novel-downloader
2+
3+
[![PyPI](https://img.shields.io/pypi/v/novel-downloader.svg)](https://pypi.org/project/novel-downloader/)
4+
[![Python](https://img.shields.io/pypi/pyversions/novel-downloader.svg)](https://www.python.org/downloads/)
5+
[![CI](https://github.com/saudadez21/novel-downloader/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/saudadez21/novel-downloader/actions/workflows/ci.yml)
6+
[![Hits-of-Code](https://hitsofcode.com/github/saudadez21/novel-downloader?branch=main&label=Hits-of-Code)](https://hitsofcode.com/github/saudadez21/novel-downloader/view?branch=main&label=Hits-of-Code)
7+
8+
[中文](./README.md) | [English](./README.en.md)
9+
10+
Asynchronous, modular, and extensible toolkit for downloading and processing online novels.
11+
12+
Supports resumable crawling, multi-format exporting, text processing pipeline, CLI, and optional Web UI.
13+
14+
**Documentation**: [Project Documentation](https://saudadez21.github.io/novel-downloader/)
15+
16+
**Requirements**: Python 3.11+ (development tested on Python 3.13)
17+
18+
---
19+
20+
## Features
21+
22+
* Asynchronous and high-performance crawling
23+
* Resumable downloads (checkpoint recovery)
24+
* Pluggable HTTP backends: `aiohttp`, `httpx`, `curl_cffi`
25+
* Export to TXT, EPUB, and HTML
26+
* Text processing pipeline: ad removal, zh conversion, translation, etc.
27+
* Optional support for image chapters and obfuscated content
28+
* Plugin system for site parsers, exporters, and processing pipelines
29+
* CLI and optional Web GUI
30+
31+
See the full feature list in the documentation: [Full Feature Overview](https://saudadez21.github.io/novel-downloader/)
32+
33+
---
34+
35+
## Installation
36+
37+
Install the latest stable release:
38+
39+
```bash
40+
pip install -U novel-downloader
41+
```
42+
43+
Install with Web UI support:
44+
45+
```bash
46+
pip install novel-downloader[web-ui]
47+
```
48+
49+
For all optional features (Web UI, OCR, image-to-text, extra backends, exporters, etc.), refer to the [Full Installation Guide](https://saudadez21.github.io/novel-downloader/guide/installation/).
50+
51+
---
52+
53+
## Quick Start (CLI)
54+
55+
```bash
56+
# Set preferred interface language
57+
novel-cli config set-lang en_US
58+
59+
# Download a novel
60+
novel-cli download https://www.example.com/book/123/
61+
62+
# Using site + book ID
63+
novel-cli download --site n23qb 12282
64+
```
65+
66+
More examples: [CLI Examples](https://saudadez21.github.io/novel-downloader/guide/cli-examples/)
67+
68+
---
69+
70+
## Programmatic API
71+
72+
```python
73+
import asyncio
74+
from novel_downloader.plugins import registrar
75+
from novel_downloader.schemas import BookConfig, ClientConfig
76+
77+
async def main() -> None:
78+
site = "n23qb"
79+
book = BookConfig(book_id="12282")
80+
81+
cfg = ClientConfig(request_interval=0.5)
82+
client = registrar.get_client(site, cfg)
83+
84+
async with client:
85+
await client.download_book(book)
86+
87+
client.export_book(book, formats=["txt", "epub"])
88+
89+
if __name__ == "__main__":
90+
asyncio.run(main())
91+
```
92+
93+
More examples: [API Examples](https://saudadez21.github.io/novel-downloader/reference/api-examples/)
94+
95+
---
96+
97+
## Development
98+
99+
```bash
100+
git clone https://github.com/saudadez21/novel-downloader.git
101+
cd novel-downloader
102+
103+
pip install .
104+
# Optional:
105+
# pip install .[all]
106+
# pip install -e .[dev,all]
107+
```
108+
109+
Translations (optional):
110+
111+
```bash
112+
pip install babel
113+
pybabel compile -d src/novel_downloader/locales
114+
```
115+
116+
PRs and issues are welcome.
117+
118+
---
119+
120+
## Notes
121+
122+
* Site structures may change. If parsing issues occur, please open an issue or submit a patch.
123+
* Login support depends on site policies. Cookies or manual account setup may be required.
124+
* Configure request intervals responsibly to avoid rate limiting or IP blocking.
125+
126+
---
127+
128+
## Disclaimer
129+
130+
This project is for learning and research purposes only.
131+
132+
Do not use it for commercial or illegal activities.
133+
134+
Users are responsible for complying with target sites' `robots.txt` and local regulations.
135+
The author assumes no liability for misuse.

‎README.md‎

Lines changed: 33 additions & 152 deletions
Original file line numberDiff line numberDiff line change
@@ -5,50 +5,30 @@
55
[![CI](https://github.com/saudadez21/novel-downloader/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/saudadez21/novel-downloader/actions/workflows/ci.yml)
66
[![Hits-of-Code](https://hitsofcode.com/github/saudadez21/novel-downloader?branch=main&label=Hits-of-Code)](https://hitsofcode.com/github/saudadez21/novel-downloader/view?branch=main&label=Hits-of-Code)
77

8+
[中文](./README.md) | [English](./README.en.md)
9+
810
异步、可扩展的小说下载与处理工具包。
911

1012
支持断点续爬、多格式导出、文本处理流水线, 并提供 CLI 与可选 Web 界面。
1113

12-
> 运行要求: **Python 3.11+** (开发环境: Python 3.13)
13-
14-
## 功能特性
15-
16-
### 下载能力
17-
18-
* **可恢复下载**: 自动识别已完成的章节, 跳过重复抓取
19-
* **可插拔式 HTTP 后端**: 支持 `aiohttp` (默认)、`httpx`、`curl_cffi`
14+
**文档**: [项目文档](https://saudadez21.github.io/novel-downloader/)
2015

21-
### 多格式导出
16+
**运行要求**: Python 3.11+ (开发环境: Python 3.13)
2217

23-
* **TXT**
24-
* **EPUB**
25-
* **HTML**
26-
27-
### 内容清洗与增强
28-
29-
* 广告与活动过滤
30-
* 章节标题过滤
31-
* 正文章节过滤
32-
* 文本处理流水线 (processors)
33-
* 正则清理
34-
* 繁简转换
35-
* 机器翻译
36-
* 图片章节 / 混淆章节处理 (`image-utils` 可选)
37-
* 原图下载
38-
* 去水印
39-
* 图像预处理
40-
* 图片章节转文字 (需要 `enable_ocr`)
41-
* 字体混淆还原 (需要 `enable_ocr`)
42-
43-
### 扩展性
18+
---
4419

45-
* **插件系统**: 可扩展站点解析、文本处理器、导出器等能力
46-
* **可插拔式下载后端**: 适配不同 HTTP 客户端
20+
## 功能特性
4721

48-
### 使用方式
22+
- **异步与高性能下载**
23+
- **可恢复下载 (断点续爬)**
24+
- **可插拔 HTTP 后端:** `aiohttp` / `httpx` / `curl_cffi`
25+
- **多格式导出:** TXT / EPUB / HTML
26+
- **文本处理流水线:** 去广告、繁简转换、自动翻译等
27+
- **图片章节与混淆章节支持 (可选)**
28+
- **插件系统:** 可扩展站点解析器、导出器、Pipeline 等
29+
- **CLI 与可选 Web GUI**
4930

50-
* **命令行 (CLI)**
51-
* **Web 图形界面 (GUI)**
31+
完整功能列表见: [功能总览](https://saudadez21.github.io/novel-downloader/)
5232

5333
---
5434

@@ -60,95 +40,29 @@
6040
pip install -U novel-downloader
6141
```
6242

63-
如需启用字体解密 / 图片转文字 (`enable_ocr`), 请参见: [安装](docs/guide/installation.md)
64-
65-
---
66-
67-
## 快速开始
68-
69-
### 0. 设置语言 (可选)
70-
71-
```bash
72-
# 设置为中文
73-
novel-cli config set-lang zh_CN
74-
75-
# 设置为英文
76-
novel-cli config set-lang en_US
77-
```
78-
79-
### 1. 初始化配置文件
43+
如需启用 Web GUI:
8044

8145
```bash
82-
# 生成默认配置 ./settings.toml
83-
novel-cli config init
46+
pip install novel-downloader[web-ui]
8447
```
8548

86-
生成 `settings.toml` 后可编辑 `request_interval`、`book_ids` 等参数。
87-
88-
详见: [settings.toml 配置说明](docs/guide/settings-reference.md)
89-
90-
### 2. 命令行 (CLI)
49+
如需启用其它可选功能 (Web UI、图片转文字、额外后端等), 请参见: [安装指南](https://saudadez21.github.io/novel-downloader/guide/installation/)
9150

92-
![cli_download](docs/assets/images/cli_download.gif)
51+
---
9352

94-
常用示例:
53+
## 快速开始 (CLI)
9554

9655
```bash
97-
# 使用书籍页面 URL 自动解析并下载
98-
novel-cli download https://www.hetushu.com/book/5763/index.html
99-
100-
# 使用配置文件中的 book_ids 启动下载
101-
novel-cli download --site qidian
56+
# 下载一本小说
57+
novel-cli download https://www.example.com/book/123/
10258

103-
# 指定站点 + 书籍 ID 启动下载
59+
# 使用站点 + 书籍 ID
10460
novel-cli download --site n23qb 12282
10561
```
10662

107-
更多参数:
63+
更多示例见: [CLI 使用示例](https://saudadez21.github.io/novel-downloader/guide/cli-examples/)
10864

109-
```bash
110-
novel-cli --help
111-
novel-cli download --help
112-
```
113-
114-
* 支持站点见: [支持站点列表](docs/supported-sites/index.md)
115-
* 更多示例见: [CLI 使用示例](docs/guide/cli-examples.md)
116-
* 运行中可使用 `CTRL+C` 取消任务
117-
118-
### 3. 图形界面 (Web GUI)
119-
120-
Web GUI 依赖额外组件 (如 NiceGUI), 默认不会随主程序一起安装。
121-
122-
如需使用 Web 图形界面，请先安装对应的可选依赖。
123-
124-
#### 3.1. 安装 Web GUI 依赖
125-
126-
```bash
127-
pip install novel-downloader[web-ui]
128-
```
129-
130-
> 若只需使用 CLI，可忽略此步骤。
131-
132-
#### 3.2 启动 Web GUI
133-
134-
```bash
135-
novel-web
136-
```
137-
138-
如需提供局域网/外网访问 (请自行留意安全与网络环境):
139-
140-
```bash
141-
novel-web --listen public
142-
```
143-
144-
在运行过程中, 可使用 `CTRL+C` 停止服务。
145-
146-
#### 3.3 更多资料
147-
148-
* 支持站点见: [支持站点列表](docs/supported-sites/index.md)
149-
* 更多示例见: [WEB 使用示例](docs/guide/web-examples.md)
150-
151-
### 4. 编程接口 (Programmatic API)
65+
## 编程接口 (Programmatic API)
15266

15367
```python
15468
import asyncio
@@ -167,49 +81,20 @@ async def main() -> None:
16781

16882
# 在异步上下文中执行下载
16983
async with client:
170-
await client.download(book)
84+
await client.download_book(book)
17185

17286
# 下载完成后执行导出操作
173-
client.export(book, formats=["txt", "epub"])
87+
client.export_book(book, formats=["txt", "epub"])
17488

17589
if __name__ == "__main__":
17690
asyncio.run(main())
17791
```
17892

179-
---
180-
181-
## 文本处理 (`processors`)
182-
183-
导出前可执行多阶段流水线处理, 包括:
184-
185-
* 正则清理 (自定义去广告/去水印)
186-
* 繁简转换 (基于 [opencc-python](https://github.com/yichen0831/opencc-python))
187-
* 自动翻译 (支持 `google` / `edge` / `youdao` 等翻译器)
188-
* 文本纠错 (基于 [pycorrector](https://github.com/shibing624/pycorrector))
189-
190-
处理顺序可配置, 并可生成中间产物用于导出
191-
192-
> 详细配置示例见: [processors 配置](docs/guide/processors-reference.md)
193-
194-
---
195-
196-
## 插件系统
197-
198-
通过插件可扩展站点解析、文本处理器、导出器等能力。
199-
200-
在 `settings.toml` 启用插件并实现对应接口后, 即可自动加入下载流程。
201-
202-
示例: 新增站点解析器 (如 "刺猬猫" -> `ciweimao`), 实现目录页与章节页的抓取及解析方后即可直接下载:
203-
204-
```bash
205-
novel-cli download --site ciweimao 123456
206-
```
207-
208-
> 详见: [插件系统文档](docs/reference/plugins.md)
93+
更多示例见: [API 示例](https://saudadez21.github.io/novel-downloader/reference/api-examples/)
20994

21095
---
21196

212-
## 从源码安装 (开发版)
97+
## 贡献与开发
21398

21499
```bash
215100
git clone https://github.com/saudadez21/novel-downloader.git
@@ -222,15 +107,10 @@ cd novel-downloader
222107
pip install .
223108
# 或安装带可选功能:
224109
# pip install .[all]
110+
# pip install -e .[dev,all]
225111
```
226112

227-
---
228-
229-
## 常见问题 / 排错
230-
231-
* **网站结构变更导致解析失败**: 请更新至最新版或按站点文档自定义适配。
232-
* **需要登录的站点**: 参考 [复制 Cookies](docs/guide/copy-cookies.md)。
233-
* **导出文件位置**: 见 [文件保存](docs/guide/file-saving.md)。
113+
欢迎提交 Issue / PR。
234114

235115
---
236116

@@ -244,5 +124,6 @@ pip install .
244124

245125
## 项目说明
246126

247-
* 本项目仅供学习和研究使用, **不得**用于任何商业或违法用途; 请遵守目标网站的 `robots.txt` 及相关法律法规
127+
* 本项目仅供学习和研究使用, **不得**用于任何商业或违法用途
128+
* 请遵守目标网站的 `robots.txt` 及相关法律法规
248129
* 使用本项目产生的任何法律责任由使用者自行承担, 作者不承担相关责任

‎docs/guide/processors-reference.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@
2828
假设配置中写入:
2929

3030
```toml
31-
[[plugins.processors]]
31+
[[general.processors]]
3232
name = "cleaner"
3333
remove_invisible = true
3434
title_removes = "title-remove.json"

0 commit comments

Comments
 (0)