byted-data-search

Data Query Tool

Prerequisites

Environment variables are required (the script reads them automatically; if they cannot be read, remind the user to set them):

VOLCENGINE_ACCESS_KEY(or VOLC_ACCESS_KEY）
VOLCENGINE_SECRET_KEY(or VOLC_SECRET_KEY）

Workflow (execute strictly in order)

Step 1: Query available data sources (must be executed first)

Before constructing any query,this step must be called firstto learn which data sources exist and their field definitions. The purpose of this step is: to confirm which data source contains the data the user needs, and what fields and filtering rules that data source has. Skipping this step and directly guessing field names will almost certainly lead to errors.

# 列出所有可用数据源摘要（含 datasource_id、名称、描述、维度/过滤字段数量）
python3 scripts/describe_datasource.py --datasource-id all

Obtain the complete field definitions of a data source (dimensions, field types, available filter operators)

python3 scripts/describe_datasource.py --datasource-id <data source ID>


返回内容包含：
- `datasource_id`：数据源唯一标识
- `datasource_name`：数据源中文名称
- `description`：数据源说明
- `dimensions`：所有字段列表，每个字段包含 field（字段名）、label（显示名）、type（类型）、description（描述）、filterable（是否可作为过滤条件字段）
- `notes`：使用备注

**关键：根据返回的字段信息（尤其是 field 名称和 type 类型），确定需要用到的字段和过滤操作符，再进入第二步。**

### 字段类型与操作符对照表

每种字段类型只支持特定操作符。用错操作符会直接报错，所以在构造 filters 之前请务必对照此表。

| 字段类型 | 支持的操作符 | 说明 |
|---------|------------|------|
| `keyword` | `eq`, `in`, `not_in` | 精确匹配类字段（如编码、状态、类型） |
| `text` | `like`, `keyword` | 文本类字段（如名称、地址、描述），支持模糊搜索 |
| `date` / `datetime` | `between`, `eq` | 日期类字段，范围查询用 between |
| `long` / `integer` / `float` / `double` | `range`, `eq` | 数值类字段，范围查询用 range |

> 注意：`long` 类型字段如企业标签（`is_longtou_flag` 等）虽然是数值类型，但用于布尔判断时用 `eq` 即可，如 `is_longtou_flag:eq:1`。

### 字段取值不确定时：先探查再过滤

构造过滤条件时，经常会遇到"知道要按某个字段过滤，但不确定该字段的实际取值是什么"的情况。比如用户想按企业状态筛选，但不知道取值是"存续"、"在业"还是"正常"；或者想按产业分类过滤，但不确定分类名称的准确写法。

**正确做法：先做一次不带该过滤条件（或只带其他确定条件）的查询，从返回数据中观察目标字段的实际取值，再用准确的值构造过滤条件。**

具体步骤：
1. 先用宽松条件查询几条数据，观察目标字段返回了哪些值
2. 如果需要看该字段有哪些不同取值，可以用 `--group-by` + `--aggregation` 做分组统计
3. 确认取值后，再加上精确的过滤条件做正式查询

示例——想按"企业状态"过滤但不确定取值：
```bash
# 第 1 步：先查几条数据，观察 reg_status 字段的实际值
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'company_name:like:科技'

# 第 2 步：或者直接做分组统计，看 reg_status 有哪些取值及各有多少条
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'company_name:like:科技' \
  --group-by 'reg_status' \
  --aggregation 'company_id:count'

# 第 3 步：确认取值后，加上精确过滤条件
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'company_name:like:科技;reg_status:eq:存续'

This strategy applies to all keyword type enum fields (such as reg_status、category、region_level、taxpayer_type、company_org_type , etc.), because these fields use eq exact matching, and a single wrong character will result in zero results.

Query field enum values

When you are not sure what optional values a field has (especially keyword type enum fields), you can use a dedicated enum value query script to obtain them in one step, instead of manually combining --group-by and --aggregation parameters. This script returns up to 200 distinct values, sorted by occurrence frequency from high to low.

This is especially useful in the following scenarios:

Before constructing eq or in filter conditions, you need to know the exact values of the field (such as whether reg_status is "active" or "in operation")
You want to quickly learn what options a categorical field (such as category、region_level、company_org_type) has
You need to view the value distribution of a field within a specific condition scope (such as within a certain industry chain)

# 基本用法：查看某个字段有哪些取值
python3 scripts/get_field_enums.py \
  --datasource-id <数据源ID> \
  --field <字段名>

# 带过滤条件：只看满足条件的数据中该字段有哪些取值
python3 scripts/get_field_enums.py \
  --datasource-id <数据源ID> \
  --field <字段名> \
  --filters '<过滤条件>'

Parameter description:

Parameter	Required	Description
`--datasource-id`	Yes	Data source ID
`--field`	Yes	The field name whose enum values are to be queried
`--filters`	No	Pre-filter conditions, format same as query_datasource
`--limit`	No	Maximum number of enum values to return (default 20, upper limit 50)

Output example:

数据源: enterprise_basic_wide
字段:   reg_status
共找到 8 个不同取值（最多显示 200 个）:

   1. 存续  (5832174 条)
   2. 注销  (3021487 条)
   3. 在业  (1245633 条)
   4. 吊销  (412056 条)
   ...

[JSON] ["存续", "注销", "在业", "吊销", ...]

The last line's [JSON] line is a machine-readable format, convenient for programmatically extracting the enum value list.

Common examples:

# 查看企业状态有哪些取值
python3 scripts/get_field_enums.py \
  --datasource-id enterprise_basic_wide --field reg_status

# 查看所属行业分类有哪些
python3 scripts/get_field_enums.py \
  --datasource-id enterprise_basic_wide --field category

# 查看产业链区域指标中 region_level 的取值
python3 scripts/get_field_enums.py \
  --datasource-id industry_chain_node_region_metric --field region_level

# 在"新能源汽车"产业链范围内，查看企业省份分布
python3 scripts/get_field_enums.py \
  --datasource-id industry_chain_company_info --field base_name \
  --filters 'chain_name:like:新能源汽车'

# 查看纳税人类型有哪些
python3 scripts/get_field_enums.py \
  --datasource-id enterprise_basic_wide --field taxpayer_type

Tip: After obtaining the enum values, you can use eq or in for exact filtering in the formal query. For example, after confirming the value is "active", you can use reg_status:eq:存续 to filter.

Step 2: Query data

Construct the query command based on the field information obtained in Step 1:

python3 scripts/query_datasource.py \
  --datasource-id <数据源ID> \
  --filters '<过滤条件>' \
  --page 1

Complete parameter description:

Parameter	Required	Description
`--datasource-id`	Yes	Data source ID, obtained from step 1
`--filters`	No	Filter conditions, format shown below, separate multiple conditions with `;` separated
`--aggregation`	No	Aggregation operation:`count`(total count),`field:count`(field count),`field:distinct`(distinct count),`field:sum/avg/max/min`
`--group-by`	No	Grouping fields, comma-separated, must be used with `--aggregation` used
`--sort-field`	No	Sort field name; if left blank, default sorting is used
`--sort-order`	No	`asc` or `desc`(default desc)
`--page`	No	Page number, starting from 1 (default 1)

Filter condition format

Format:字段名:操作符:值, separate multiple conditions with ; separated.

Operator	Meaning	Example	Applicable field types
`eq`	Exact match	`reg_status:eq:存续`	keyword, date, numeric
`like`	Fuzzy match (phrase match)	`company_name:like:字节跳动`	text
`in`	Multi-value match (comma-separated)	`reg_status:in:存续,在业`	keyword
`not_in`	Exclude match (comma-separated)	`reg_status:not_in:注销,吊销`	keyword
`between`	Date range (start,end)	`estiblish_time:between:2020-01-01,2025-12-31`	date, datetime
`range`	Numeric range (min,max; for half-open intervals use `,100` or `50,`）	`company_total_count:range:100,`	long, integer, float, double
`keyword`	Full-text search	`keyword:keyword:新能源补贴`	text

Common errors：

text type fields (such as company_name) cannot use eq, must use like or keyword
keyword type fields (such as reg_status) cannot use like, must use eq / in / not_in
If the query reports "field does not support operator", go back to step 1 and check the field type

Multi-condition combination example:

company_name:like:科技;region_province_name:like:广东;estiblish_time:between:2020-01-01,2025-12-31

Current known data source quick reference

The following are common data sources; for the complete list, call describe_datasource.py --datasource-id all to get:

Data source ID	Name	Typical use	Key notes
`enterprise_basic_wide`	Enterprise basic information	Business registration information, legal representative, registered capital, business scope, shareholders and financing, etc.	`company_name` is text type → use `like`；max_page_size=5
`industry_chain_company_info`	Industry chain enterprise information	Enterprises under the industry chain, filtered by tags (leading enterprise/high-tech/specialized and sophisticated/unicorn, etc.)	`chain_name`/`node_name` is text → use `like`; tag fields (is_longtou_flag, etc.) are long → use `eq:1`
`industry_chain_node_region_metric`	Industry chain node regional indicators	Regional distribution statistics of the industry chain, number of enterprises, financial indicators, intellectual property, etc.	Supports aggregate statistics, max_page_size=50
`stock_company_brief`	Listed company profile	A-share listed company fundamentals, F10, IPO, dividends, executives, etc.	Mustuse `code:eq:证券编码` query (such as `code:eq:000001`）；max_page_size=5

Common query examples

Enterprise business information query

# 模糊搜索企业
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'company_name:like:字节跳动'

# 按省份+行业筛选企业
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'region_province_name:eq:广东;category:eq:信息传输、软件和信息技术服务业'

# 查询特定日期后成立的企业
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'company_name:like:科技;estiblish_time:between:2023-01-01,2025-12-31'

Industry chain enterprise query

# 查某产业链下的龙头企业
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_company_info \
  --filters 'chain_name:like:新能源汽车;is_longtou_flag:eq:1'

# 查某产业链下的专精特新企业（按省份筛选）
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_company_info \
  --filters 'chain_name:like:半导体;is_ssdi_flag:eq:1;base_name:eq:广东'

Industry chain regional indicator query

# 查某产业链在各省份的企业数量
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_node_region_metric \
  --filters 'chain_name:like:新能源;region_level:eq:省' \
  --sort-field company_total_count \
  --sort-order desc

# 聚合统计某产业链的总企业数
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_node_region_metric \
  --filters 'chain_name:like:人工智能' \
  --aggregation 'company_total_count:sum'

Listed company query

# 按证券编码查询上市公司简况（code 必传）
python3 scripts/query_datasource.py \
  --datasource-id stock_company_brief \
  --filters 'code:eq:000001'

Explore field values before filtering

# 不确定 category（所属行业）有哪些取值？用枚举值查询一步搞定
python3 scripts/get_field_enums.py \
  --datasource-id enterprise_basic_wide --field category

# 看到取值后，用精确值过滤
python3 scripts/query_datasource.py \
  --datasource-id enterprise_basic_wide \
  --filters 'category:eq:信息传输、软件和信息技术服务业'

Aggregate statistics query

# 统计符合条件的总记录数
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_company_info \
  --filters 'chain_name:like:新能源汽车' \
  --aggregation 'count'

# 按省份分组统计企业数量
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_company_info \
  --filters 'chain_name:like:新能源汽车' \
  --group-by 'base_name' \
  --aggregation 'company_id:count'

# 按省份分组统计并去重
python3 scripts/query_datasource.py \
  --datasource-id industry_chain_company_info \
  --filters 'chain_name:like:新能源汽车' \
  --group-by 'base_name' \
  --aggregation 'company_id:distinct'

Call limits

Rate limit: at most 10 calls per minute
Daily limit: up to 200 calls per day
For more calls, purchase Volcengine-High Quality Dataset

Error Handling

Error Type	Cause	Solution
Authentication Failed	Environment variable not set or credential invalid	Check `VOLCENGINE_ACCESS_KEY` / `VOLCENGINE_SECRET_KEY`, refer to User Guide to obtain AK/SK
Data source does not exist	datasource_id incorrect	Call `describe_datasource.py --datasource-id all` to verify
Field does not support operator	Operator does not match field type	Check the field type and refer to the "Field Type and Operator Reference Table"
Field unavailable	Field name misspelled or not filterable	Call `describe_datasource.py --datasource-id <ID>` to verify
No results returned	Filter conditions too strict or field value incorrect	First use `get_field_enums.py` to probe actual field values, or relax the conditions (such as `eq` to `like`）
Rate limit	Call count exceeded	For minute rate limit wait 1 minute; for daily rate limit wait until the next day

Environment variable configuration:

export VOLCENGINE_ACCESS_KEY="your-access-key"
export VOLCENGINE_SECRET_KEY="your-secret-key"