novel-reader

Novel Reader - Intelligent Long-Text Novel Reader

Use Python to read long-text novels safely and reliably, solving the problem of the limited context window of LLMs. Perfectly supports Chinese novels in UTF-8 encoding. Automatically filters irrelevant content and extracts detailed character, item, and scene information in real time.

Core Features

• Multi-format support: If the input file is in PDF, DOC, or DOCX format, first use the doc-to-txt skill to convert it to TXT plain text format, then read the converted TXT file
• Read by character position: Use Python to read precisely by character position, with no garbled text
• Segmented reading: Read at most 3000 characters at a time, with a default value of 3000 characters, to avoid exceeding the tool output limit.Skipping reading is strictly prohibited, and reading must be continuous segment by segment
• Flexible positioning: Can start reading from any position
• Encoding safety: Natively supports UTF-8 encoding and correctly handles Chinese characters
• Intelligent content filtering: Automatically identifies and skips content unrelated to the main text in the novel through the large model (release notes, author thanks, interviews, advertisements, etc.)
• Real-time asset extraction: Each time a segment of content is read, immediately identify and record detailed information about the characters, items, and scenes in the novel
• Outline recording: Record a summary of each segment of content to form a complete outline
• Progress tracking: Record progress information such as the current reading position and the number of characters read
• Context management: Make full use of the Agent's context compact mechanism, automatically compressing when the context is almost used up

Important Requirements

Skipping reading is strictly prohibited

• Reading must be continuous segment by segment: After reading 3000 characters each time, the next read must continue from the current end position (that is, start = previous start + 3000)
• Skipping reading is prohibited: It is absolutely not allowed to skip intermediate content and jump directly to a later position
• For loop reading is absolutely prohibited: It is strictly prohibited to use for loops to read files in batches (such as for i in $(seq ...)), because the Agent tool compresses the output, and for loops will cause the content to be truncated. When you see (some characters truncated), it means the content has been truncated, and you can only manually read it again segment by segment.
• Prohibited to call the python script multiple times in one command: Each tool can output only 3000 characters, and content beyond that will be truncated. One tool call can invoke the python command at most once!
• Improve efficiency: Because super-long text needs to be read, you can continuously call the reading tool script (but you cannot use for loops), without thinking or speaking in between.
• Update cycle: Every time 10 novel segments are read (3000 characters each time, 30000 characters in total), update the outline, progress, and asset files once, then continue to the next cycle.
• Pre-setting tasks is prohibited: It is prohibited to use TODO lists and Task list-related tools!

How to Use

Core Commands

python3 read_novel.py <小说文件路径> [--start <起始位置>]

Parameter Description

Parameter	Description	Example
<小说文件路径>	Path to the novel file (required), supports TXT, PDF, DOC, DOCX formats. If it is in PDF/DOC/DOCX format, first use the doc-to-txt skill to convert it to TXT.	test-files/novel.txt or test-files/novel.pdf
--info	Get novel information (total character count, total line count, non-empty line count)	--info
--start	Start position (character index, starting from 0, default: 0, optional)	--start 10000

Examples

Get novel information:

python3 read_novel.py ./novel.txt --info

Read the first 3000 characters of the novel from the beginning (using default parameters):

python3 read_novel.py ./novel.txt --start 0

Read 3000 characters of the novel starting from the 3000th character (specify only start):

python3 read_novel.py ./novel.txt --start 3000

View help:

python3 read_novel.py --help

Workflow

The complete workflow is divided into three stages: file format processing → initialization → loop reading → completion summary

Pre-step: File format processing

• Check the input file format
• If it is in PDF, DOC, or DOCX format, first use the doc-to-txt skill to convert it to TXT plain text format
• After the conversion is complete, use the converted TXT file for subsequent operations

Stage 1: Initialization (only once)

Step 1: Get novel information

python3 read_novel.py novel.txt --info

Example output:

总字符数: 5512508
总行数: 158108
非空行数: 79053

Step 2: Check reading progress

• Check whether 读取进度.txt file exists
• If it exists: read the current position and continue from that position (resume reading from the breakpoint)
• If it does not exist: start a new reading from position 0

Stage 2: Loop reading (core stage)

Each loop reads 10 segments (30000 characters in total), divided into two sub-stages:

Sub-stage A: Batch reading (executed continuously, with no analysis in between)

Call the read command 10 times continuously, reading 3000 characters each time:

• 1st time:--start <当前位置>
• 2nd time:--start <当前位置+3000>
• ...
• 10th time:--start <当前位置+27000>

Example (the first loop starting from 0):

读取片段 1：--start 0
读取片段 2：--start 3000
...
读取片段 10：--start 27000

Example (the third loop starting from 60000):

读取片段 21：--start 60000
读取片段 22：--start 63000
...
读取片段 30：--start 87000

⚠️ Key ruleThe 10 reads must be executed continuously, with no analysis, no summarization, and no file updates in between

Sub-stage B: Analysis and update (executed after reading 10 segments)

1. Analyze the content of these 10 segments
2. Update 大纲.txt: add chapter summaries for these 10 segments
3. Update 读取进度.txt: record the current position (such as 30000)
4. Create or update asset files: extract newly appearing characters, items, and scenes

Sub-stage C: Determine whether to continue

• Calculate the reading progress (characters read/total characters)
• If the user's requirement has not been met (such as "read 5%"): return to sub-stage A and continue with the next batch of 10 segments
• If the user's requirement has been met: enter stage 3

Stage 3: Completion summary

• Output the final reading progress
• Summarize the statistics of the extracted assets (number of characters, items, and scenes)

Execution order example

Assume the user requests reading the first 5% of the novel, and the novel file is novel.pdf：

Pre-step: File format processing

• Check the file format: .pdf
• Use the doc-to-txt skill to convert novel.pdf to novel.txt
• Subsequent steps use the novel.txt file

Step 1: Initialization

1. Get novel information → total word count 5512508
2. Check 读取进度.txt → does not exist, start from position 0

Step 2: Loop reading

Loop 1 (segments 1-10, 0-30000 characters):

• Batch reading: consecutively execute --start 0 to --start 27000
• Analysis update: update outline, progress (30000), assets
• Check progress: 30000/5512508 = 0.54%, has not reached 5%, continue

Loop 2 (segments 11-20, 30000-60000 characters):

• Batch reading: consecutively execute --start 30000 to --start 57000
• Analysis update: update outline, progress (60000), assets
• Check progress: 60000/5512508 = 1.09%, has not reached 5%, continue

Loop 3 (segments 21-30, 60000-90000 characters):

• Batch reading: consecutively execute --start 60000 to --start 87000
• Analysis update: update outline, progress (90000), assets
• Check progress: 90000/5512508 = 1.63%, has not reached 5%, continue

... continue looping ...

Loop 10 (segments 91-100, 270000-300000 characters):

• Batch reading: consecutively execute --start 270000 to --start 297000
• Analysis update: update outline, progress (300000), assets
• Check progress: 300000/5512508 = 5.44%, has reached 5%, stop

Step 3: Completion summary

• Output final progress: 5.44%
• Count assets: X characters, Y items, Z scenes

Key rules

Rule	Description
Strictly prohibited to analyze while reading	Analyzing and updating after reading each segment is the wrong approach
Must read in batches	Analyze and update uniformly only after all 10 segments have been read
Supports resuming from a breakpoint	Resume after interruption by reading progress.txt
Progress calculation	Read 30000 characters per loop (10 × 3000)

Asset extraction

Intelligent content filtering

After reading the novel content by executing a Python script, let the large model understand the content, automatically identify and ignore content unrelated to the main text of the novel, and retain only the main text chapter content of the novel for analysis and asset extraction. Do not use any rules or regular expressions for filtering; rely entirely on the large model's comprehension ability.

Real-time asset extraction

Each time a passage of content is read (no more than 3000 characters), immediately identify and extract newly appearing assets (characters, items, scenes) from it, and update the corresponding files in real time. Do not wait until everything has been read before updating. The large model will make full use of the context compact mechanism, automatically compressing when the context is about to run out.

Extract elements

From each passage of content read, identify and extract the following three types of assets, collecting detailed information as much as possible:

1. Characters - People appearing in the novel, including:

   • Basic information: name, age, gender, identity, role positioning (protagonist/supporting role/villain/second male lead/second female lead, etc.)
   • Appearance features: facial appearance, height, build, clothing
   • Personality traits: catchphrases, habitual actions, behavior patterns
   • Background relationships: family, friends, enemies, mentorship
   • Abilities and cultivation: power level, special abilities, weapons and equipment
   • Plot appearances: first appearance, important events

2. Items - Objects, magical treasures, weapons, etc. that appear in the novel, including:

   • Basic information: name, type, source
   • Appearance features: shape, color, material, size
   • Functional characteristics: special abilities, usage methods, effects
   • Historical background: origin, previous owners, important events
   • Related characters: owners, users, contenders

3. Scenes - Locations, environments, places, etc. that appear in the novel, including:

   • Basic information: name, type, geographical location
   • Environmental features: terrain, climate, architectural style, atmosphere
   • Functions and uses: residence, cultivation, trade, battle
   • Related forces: affiliated forces, managers, regular residents
   • Important events: key plot points that occurred

Directory structure

Create the following structure in the directory where the novel file is located:

小说名或项目名/
├── 大纲.txt
├── 读取进度.txt
├── 角色/
│   ├── <角色名1>.txt
│   ├── <角色名2>.txt
│   └── ...
├── 道具/
│   ├── <道具名1>.txt
│   ├── <道具名2>.txt
│   └── ...
└── 场景/
    ├── <场景名1>.txt
    ├── <场景名2>.txt
    └── ...

File content format

Outline.txt

1. <第一段内容的摘要>

2. <第二段内容的摘要>

...

Description：

• Add summaries by logical sections of the novel's content, not by fixed word count
• The large model naturally decides when to add new summary entries based on the content
• Each summary is marked with a sequence number and arranged in reading order
• Only record summaries of the novel's main text content, skipping irrelevant content
• Skip content in the novel unrelated to the main text, such as release statements, author thanks, interviews, advertisements, etc.

Reading progress.txt

当前位置: <字符索引>
已读字数: <数字>
总字数: <数字>
进度: <百分比>%

Character files (characters/<character name>.txt)

Each character file contains a detailed description of that character, with the reference format as follows:

<角色名称>

【基本信息】
姓名：
年龄：
性别：
身份：
角色定位：

【外貌特征】
容貌：
身高：
体型：
着装：

【性格特点】
性格：
口头禅：
习惯性动作：
行为模式：

【背景关系】
家庭：
朋友：
敌人：
师承：

【能力修为】
实力等级：
特殊能力：
武器装备：

【出场情节】
首次出场：

Item files (items/<item name>.txt)

Each item file contains a detailed description of that item, with the reference format as follows:

<道具名称>

【基本信息】
名称：
类型：
来源：

【外观特征】
形状：
颜色：
材质：
大小：

【功能特性】
特殊能力：
使用方法：
效果：

【历史背景】
来历：
重要事件：

【相关角色】
拥有者：
使用者：
争夺者：

Scene files (scenes/<scene name>.txt)

Each scene file contains a detailed description of that scene, with the reference format as follows:

<场景名称>

【基本信息】
名称：
类型：
地理位置：

【环境特征】
地形：
气候：
建筑风格：
氛围：

【功能用途】
居住：
修炼：
交易：
战斗：

【相关势力】
所属势力：
管理者：
常驻人物：

【重要事件】
发生过的关键情节：

Extraction rules

1. Deduplication: If an asset file already exists, only append new information; do not recreate it
2. Accuracy: Ensure the extracted asset names are accurate and avoid typos
3. Completeness: Record all relevant information about the asset as much as possible, using the detailed format above
4. Natural language: Use natural language descriptions so the large model understands the extraction requirements
5. Content filtering: Before extracting assets, filter out irrelevant content first and extract only from the main text
6. Real-time updates: Each time a segment of content is read (no more than 3000 characters), immediately identify and extract newly appearing assets; do not wait until everything is read before updating
7. Do not use rules/regex: Obtain content entirely through executing Python scripts, let the large model understand and analyze on its own, and do not use any rules or regular expressions for matching
8. Importance screening: Only record important assets; unimportant characters, scenes, and items may be ignored and skipped. Judgment criteria include:
   • Characters: Main characters who drive the plot, key supporting characters, important villains, etc.; passersby, extras, and minor characters who appear only once can be ignored
   • Props: Magical treasures, weapons, and important items that play a key role in the plot; ordinary items and one-time props can be ignored
   • Scenes: The main locations and important places where the story takes place; temporary scenes and places mentioned in passing can be ignored

Tips

1. Get the total character count of the file：

python3 -c "
with open('test-files/novel.txt', 'r', encoding='utf-8', errors='replace') as f:
    print(len(f.read()))
"

1. Read in segments: Read at most 3000 characters at a time; the default value is 3000 characters to avoid exceeding the tool output limit

2. Context management: Large models usually have a 128k context window. The Agent will automatically compress when the context is almost used up. Please use it reasonably

Why use Python?

• ✅ Encoding safety: Native support for UTF-8, no garbled text
• ✅ Read by character: Not by byte; correctly handles Chinese characters (each Chinese character counts as 1)
• ✅ Cross-platform: The same on Windows/Mac/Linux
• ✅ Preinstalled on systems: Almost all modern systems have Python 3 preinstalled
• ✅ Simple syntax: One pattern handles all scenarios