tushare-plugin-builder

Purpose

Convert Tushare documentation URLs into production-grade plugins in this repository, including data extraction, ClickHouse table schemas, query services, and testable curl examples.

When to use

The user provides a Tushare documentation URL or page content and requests that a plugin be generated.
The user needs to generate a plugin + ingestion + service + Agent/MCP invocation based on a certain Tushare interface.
Validate an existing plugin: The user requests a check of whether a certain plugin conforms to the specification.

Workflow

1) Collect the necessary inputs

Ensure the user has provided a Tushare documentation URL. If it cannot be accessed, request a screenshot or copied documentation content.
If the plugin name is not specified, ask the user. Use snake_case naming, consistent with existing tushare_* plugins.

2) Extract the API specification from the documentation

Parse the interface name, input parameters, output fields, and usage notes (rate limits, pagination, data volume limits).
Record field naming differences (such as pct_change vs pct_chg) and the required conversions.

3) Plan the plugin structure (refer to the `references/plugin_conventions.md`）

directory:src/stock_datasource/plugins/<plugin_name>/
Files:__init__.py、plugin.py、extractor.py、service.py、schema.json、config.json、<plugin_name>.md
Use existing tushare_* plugins as templates.

4) Implement the extractor

Use the tushare SDK; API calls must be wrapped with proxy_context() .
Implement rate limiting, timeouts, and retries (tenacity).
Support based on API characteristics trade_date or start_date/end_date。

5) Implement the plugin

Implement extract_data、validate_data、transform_data、load_data。
Before insertion, add the version and _ingested_at columns.
Convert numeric types,trade_date convert to Date。

Plugin categories and roles

The following methods must be implemented to specify the plugin's category and role:

from stock_datasource.core.base_plugin import PluginCategory, PluginRole

def get_category(self) -> PluginCategory:
    """插件分类 - 按市场划分"""
    return PluginCategory.CN_STOCK  # 或 HK_STOCK, INDEX, ETF_FUND, SYSTEM

def get_role(self) -> PluginRole:
    """插件角色"""
    return PluginRole.PRIMARY  # 或 BASIC, DERIVED, AUXILIARY

Category descriptions：

CN_STOCK: A-share related data
HK_STOCK: Hong Kong stock related data
INDEX: Index related data
ETF_FUND: ETF/fund related data
SYSTEM: System data (such as trading calendars)

Role descriptions：

PRIMARY: Primary data (such as daily quotes)
BASIC: Basic data (such as stock_basic)
DERIVED: Derived data (such as adjusted factors)
AUXILIARY: Auxiliary data (such as index weights)

Dependency configuration

Implement the dependency method in plugin.py (not configured in config.json）：

def get_dependencies(self) -> List[str]:
    """必须依赖 - 这些插件的数据必须存在才能运行当前插件。
    
    例如：tushare_daily 依赖 tushare_stock_basic 提供股票代码列表。
    """
    return ["tushare_stock_basic"]

def get_optional_dependencies(self) -> List[str]:
    """可选依赖 - 同步主插件时默认会同步这些依赖，用户可选择禁用。
    
    例如：tushare_daily 可选同步 tushare_adj_factor 复权因子。
    """
    return ["tushare_adj_factor"]

Dependency rules：

Required dependency: before running the current plugin, it will check whether there is data in the dependency plugin table
Optional dependency: in the frontend display, associated plugins that can be checked will be shown, checked by default

6) Implement the service query

Provide at least one date range query and one latest data query.
Parameterized queries must be used(string concatenation is prohibited).
Return a JSON-serializable structure.

7) Define schema/config

schema.json: Use ReplacingMergeTree，partition_by as toYYYYMM(trade_date)，order_by as the primary key.
config.json: Contains the complete plugin configuration.

Complete structure of config.json

{
  "enabled": true,
  "rate_limit": 120,
  "timeout": 30,
  "retry_attempts": 3,
  "description": "插件描述",
  "schedule": {
    "frequency": "daily",
    "time": "18:00",
    "day_of_week": "monday"
  },
  "parameters": {
    "max_empty_days": 5,
    "validate_prices": true
  },
  "parameters_schema": {
    "trade_date": {
      "type": "string",
      "format": "date",
      "required": true,
      "description": "Trade date in YYYYMMDD format"
    }
  }
}

Field descriptions：

Field	Required	Description
`enabled`	Yes	Whether to enable the plugin
`rate_limit`	Yes	API call rate limit (times/minute)
`timeout`	Yes	Request timeout (seconds)
`retry_attempts`	Yes	Retry count
`description`	Yes	Plugin description
`schedule`	No	Scheduling configuration
`schedule.frequency`	No	Scheduling frequency:`daily` or `weekly`, default `daily`
`schedule.time`	No	Execution time, format `HH:MM`, default `18:00`
`schedule.day_of_week`	No	Only `weekly` valid when, e.g., `monday`
`parameters`	No	Plugin-specific parameters
`parameters_schema`	Yes	Parameter schema, used for validation and frontend display

Note: Dependency configuration (dependencies、optional_dependencies）is not defined in config.json, but is implemented through the plugin.py in get_dependencies() and get_optional_dependencies() methods.

8) Create ClickHouse table

Based on schema.json generate CREATE TABLE SQL。

Use script：

# 仅生成 SQL
python .codebuddy/skills/tushare-plugin-builder/scripts/generate_create_table_sql.py \
  src/stock_datasource/plugins/<plugin_name>/schema.json

# 生成并执行
python .codebuddy/skills/tushare-plugin-builder/scripts/generate_create_table_sql.py \
  src/stock_datasource/plugins/<plugin_name>/schema.json --execute

9) Verify database connection

Before execution, first verify whether the ClickHouse connection is normal:

# 测试连接
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_clickhouse_connection.py

# 列出所有表
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_clickhouse_connection.py --list

# 验证指定表是否存在
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_clickhouse_connection.py --table <table_name>

10) Run data pull test

Execute the plugin to pull and store sample data (such as for a certain trading day).
Proxy configuration: before running, ensure the proxy settings in runtime_config.json are correct.

Use script：

# 运行插件并验证
python .codebuddy/skills/tushare-plugin-builder/scripts/run_plugin_test.py \
  <plugin_name> --date 20250110 --verify

or use the module approach:

python -m stock_datasource.plugins.<plugin_name>.plugin --date 20250110

11) Verify ClickHouse data

Query ClickHouse to confirm the data has been stored:

# 使用脚本验证
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_clickhouse_connection.py \
  --table <table_name> --date 20250110

or query manually:

SELECT count(*), min(trade_date), max(trade_date)
FROM <table_name>
WHERE trade_date = '2025-01-10';

12) Test HTTP service endpoints

If not running, start the HTTP service:python -m stock_datasource.services.http_server

Use script：

# 列出所有 API 路由
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_service_http.py --list

# 测试指定服务方法
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_service_http.py \
  --service <service_name> \
  --method get_by_date_range \
  --params '{"ts_code":"XXX","start_date":"20250101","end_date":"20250110"}'

# 仅生成 curl 命令
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_service_http.py \
  --service <service_name> --method get_latest --params '{}' --curl

13) Test MCP tool availability

If not running, start the MCP service:python -m stock_datasource.services.mcp_server

Use script：

# 列出所有 MCP 工具
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_mcp_tool.py --list

# 搜索相关工具
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_mcp_tool.py --pattern <service_name>

# 验证指定工具
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_mcp_tool.py --tool <tool_name>

14) Test Agent integration

Send a natural language query to the Orchestrator; it should call the new plugin's service.
Example question: "Query the market data of the XXX index for the past week"
Verification:
- The Agent correctly identifies the intent and selects the tool.
- The tool call returns data from ClickHouse.
- The response is formatted and returned to the user.

15) Verify integration points

The Service is automatically discovered; no additional manual registration is required.
MCP/HTTP endpoints work according to the generated routes.

Verification script description

This skill provides the following verification scripts (located in the scripts/ directory):

Script	Purpose
`validate_plugin.py`	Comprehensively verify whether the plugin complies with the specification
`verify_clickhouse_connection.py`	Test database connection, list tables, verify data
`generate_create_table_sql.py`	Generate table creation SQL based on schema.json
`run_plugin_test.py`	Run the plugin ETL and verify data ingestion
`verify_service_http.py`	Test HTTP service endpoints
`verify_mcp_tool.py`	Verify MCP tool registration status

Plugin verification process

When the user requests verification of an existing plugin (whether self-written or generated), perform the following steps:

1) Run comprehensive verification script

# 验证单个插件
python .codebuddy/skills/tushare-plugin-builder/scripts/validate_plugin.py <plugin_name>

# 显示所有检查详情
python .codebuddy/skills/tushare-plugin-builder/scripts/validate_plugin.py <plugin_name> --verbose

# 列出所有可用插件
python .codebuddy/skills/tushare-plugin-builder/scripts/validate_plugin.py --list

# 验证所有插件
python .codebuddy/skills/tushare-plugin-builder/scripts/validate_plugin.py --all

2) Verification check items

validate_plugin.py The following will be checked automatically:

File structure check:

__init__.py exists and exports the Plugin/Service class
plugin.py exists and inherits from BasePlugin
extractor.py exists and uses proxy_context
service.py exists and inherits from BaseService
schema.json exists and contains the required fields
config.json exists and contains the required configuration

Code standards check:

plugin.py implements the extract/validate/transform/load methods
plugin.py adds the version and _ingested_at system columns
plugin.py implements get_category() to return the correct PluginCategory
plugin.py implements get_role() to return the correct PluginRole
plugin.py implements get_dependencies() (returns an empty list or dependent plugin names)
plugin.py implements get_optional_dependencies() (if there are optional dependencies)
extractor.py uses retry logic (tenacity)
service.py uses parameterized SQL (no SQL injection risk)
service.py contains query methods (get_/query_/fetch_*）

Schema standards check:

contains table_name, columns, engine, order_by
uses the ReplacingMergeTree engine
contains the version and _ingested_at system columns
sets the partitioning strategy (partition_by)

Runtime check:

the module can be imported normally
the database table has been created
the table contains data (if ETL has been run)

3) Handle validation results

Take action based on the validation results:

✅ Passed: The plugin meets the standards and can proceed to the next step of testing.

⚠️ Warning: The plugin can run but has potential issues; it is recommended to fix them:

Missing retry logic → add the tenacity decorator
No parameterized queries detected → check whether the SQL uses %(param)s
Table is empty → run the data pull test

❌ Failed: The plugin has serious issues and must be fixed:

SQL injection risk → switch to parameterized queries
Missing required files → add the missing files
proxy_context not used → API calls will fail
Missing system columns → add them in schema.json and plugin.py

4) End-to-end validation

After passing the basic checks, execute the complete test process:

# 1. 验证数据库连接
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_clickhouse_connection.py

# 2. 建表（如果表不存在）
python .codebuddy/skills/tushare-plugin-builder/scripts/generate_create_table_sql.py \
  src/stock_datasource/plugins/<plugin_name>/schema.json --execute

# 3. 运行 ETL 并验证入库
python .codebuddy/skills/tushare-plugin-builder/scripts/run_plugin_test.py \
  <plugin_name> --date 20250110 --verify

# 4. 测试 HTTP 服务
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_service_http.py \
  --service <service_name> --method get_latest --params '{}'

# 5. 测试 MCP 工具
python .codebuddy/skills/tushare-plugin-builder/scripts/verify_mcp_tool.py \
  --pattern <service_name>

Output checklist

The plugin directory has been created and contains all required files.
Extractor uses proxy_context() and tushare SDK。
Plugin implements get_category() and get_role() methods.
Plugin implements get_dependencies() and get_optional_dependencies() methods.
Service uses parameterized SQL.
Schema matches the Tushare output fields.
Config contains the rate limit, schedule, and parameter schema.
The ClickHouse table has been created and the data has been loaded.
Verify via SQL query that there is data in ClickHouse.
The HTTP service endpoint has been tested with curl.
The MCP tool has been registered and can be called.
The Agent can call this service via natural language queries.

About this skill

Purpose

When to use

Workflow

1) Collect the necessary inputs

2) Extract the API specification from the documentation

3) Plan the plugin structure (refer to the references/plugin_conventions.md）

4) Implement the extractor

5) Implement the plugin

Plugin categories and roles

Dependency configuration

6) Implement the service query

7) Define schema/config

Complete structure of config.json

8) Create ClickHouse table

9) Verify database connection

10) Run data pull test

11) Verify ClickHouse data

12) Test HTTP service endpoints

13) Test MCP tool availability

14) Test Agent integration

15) Verify integration points

Verification script description

Plugin verification process

1) Run comprehensive verification script

2) Verification check items

3) Handle validation results

4) End-to-end validation

Output checklist

3) Plan the plugin structure (refer to the `references/plugin_conventions.md`）