Purpose
Collect men's fashion market data (real-world axis) from three sources and load it into a single standard schema. Each source provides different fields — the design philosophy of this skill is not to hide those differences but to record them as nullable.
Collection ethics (mandatory compliance)
- From public pages, aggregate figures(price, discount rate, review count, rating, ranking) are collected only.
- Data that constitutes copyrighted works or personal information, such as review bodies and author names, is not collected.
- Limit to 1–2 pages per keyword and up to 50 items per source. Do not make repeated bulk requests.
- This assumes collection for educational purposes, and collection results are not to be redistributed commercially.
Procedure
- Confirm collection targets: Confirm with the user the keyword (e.g., men's shirts) and sources (default: Musinsa+Naver+Coupang).
- Collection:
references/site-field-map.mdFollowing the per-source URL patterns and field extraction rules in Bright Data tools (scrape_as_markdownfamily) to fetch listing pages and extract fields.- 11st and 29CM are fully JS-rendered and cannot be collected this way (measured 2026-07-26). Do not attempt.
- Ad products are
is_ad=truemarked as and included (not excluded — material for the cleaning stage).
- CSV generation: Create
collected_products.csvin the exact order of the standard schema (below). Leave fields missing from a source as empty values (e.g., Coupang's rating). - Validation:
python3 scripts/validate_csv.py collected_products.csvto pass column, type, and range checks. Fix failed rows or report exclusions with reasons. - Confirmation gate: Show the user the per-source collection counts, 5 sample rows, and validation results, and obtain approval for loading. Do not write to the database before approval.
- Loading:
references/schema.sql의collected_productstable, and after loading, query the count per source and report against the CSV.
Standard CSV schema (column order fixed)
| Column | Type | Description |
|---|---|---|
| source | text | musinsa · naver · coupang |
| keyword | text | Collection keyword |
| rank | int | Sequence number within the listing (on-screen order including ads) |
| brand | text | Brand name (may be empty for Naver and Coupang) |
| name | text | Product name |
| price | int | Sale price (KRW) |
| discount_rate | int | Discount rate % (empty if none) |
| rating | numeric | Rating (empty for Coupang — not in listing) |
| review_count | int | Review count |
| purchase_count | int | Purchase count (Naver only) |
| is_ad | bool | Whether it is an ad product |
| product_url | text | Product URL (query parameters removed) |
| collected_at | timestamptz | Collection time (ISO 8601) |
Rules
- Do not arbitrarily add or rename schema columns (this diverges from the lecture screen). If extension is needed, ask the user first.
- If collection fails (blocking, structural change), honestly report the failed sources, and confirm whether to proceed with only the successful sources.
- Keep the original markdown response only as a temporary file and do not put it into the repository or DB.