google-trends-ath-detector

<essential_principles> Google Trends ATH Detector Core Principles

1. Simulate real human browser behavior to scrape Google Trends

This skill uses Selenium to simulate a real human browser:

• Remove navigator.webdriver automation markers
• Randomly rotate User-Agent (Chrome/Firefox/Safari)
• Random delay between requests (0.5-2 seconds)
• First visit the homepage to establish a session, then scrape data

2. Signal Typing

Search trend spikes are divided into three types:

Type	Characteristics	Interpretation
Seasonal spike	Repeats in fixed months each year	Institutional cycle (insurance enrollment season, tax season)
Event-driven shock	Short-term spike, high z-score	News/policy/breaking events
Regime shift	Trend line shifts upward, sustained high level	Structural increase in attention

3. Analysis Formulas

ATH 判定：latest_value >= max(history) * 0.98
異常判定：zscore >= threshold (default: 2.5)
訊號分型：based on (is_ath, is_anomaly, trend_direction)

4. Descriptive Analysis First

This skill providesobjective mathematical analysis results：

• Output quantitative indicators such as signal type and anomaly score
• Extract related queries as reference for driving factors
• Users interpret it themselves based on professional knowledge </essential_principles>

1. Detect - Quickly detect whether an ATH has been reached or an anomaly has occurred
2. Analyze - In-depth analysis of signal types and driving factors
3. Compare - Compare trend resonance across multiple topics

Wait for a response before continuing.

After reading the workflow, please follow its steps completely.

<reference_index> Reference Documents (references/)

Document	Content
input-schema.md	Complete input parameter definitions and default values
data-sources.md	Google Trends data sources and Selenium scraping guide
signal-types.md	Signal typing definitions and determination logic
seasonality-guide.md	Seasonal decomposition methods and interpretation
</reference_index>

<workflows_index>

Workflow	Purpose
detect.md	Quickly detect ATH and anomaly scores
analyze.md	In-depth analysis, signal typing, driving terms
compare.md	Multi-topic trend resonance analysis
</workflows_index>

<templates_index>

Template	Purpose
output-schema.yaml	Standard output JSON schema
</templates_index>

<scripts_index>

Script	Purpose
trend_fetcher.py	Core scraping and analysis logic (Selenium version)
</scripts_index>

<examples_index> Example Output (examples/)

Document	Content
health_insurance_ath.json	Health Insurance ATH detection example
seasonal_vs_anomaly.json	Seasonal vs anomaly determination example
multi_topic_comparison.json	Multi-topic comparison example
</examples_index>

<quick_start> Quick Start: Install Dependencies

pip install selenium webdriver-manager beautifulsoup4 lxml loguru

Python API：

from scripts.trend_fetcher import fetch_trends, analyze_ath

# 抓取數據（使用 Selenium 模擬瀏覽器）
data = fetch_trends(
    topic="Health Insurance",
    geo="US",
    timeframe="2004-01-01 2025-12-31"
)

# ATH 分析
result = analyze_ath(data, threshold=2.5)

print(f"Is ATH: {result['analysis']['is_all_time_high']}")
print(f"Signal Type: {result['analysis']['signal_type']}")
print(f"Z-Score: {result['analysis']['zscore']}")

CLI Quick Start:

# 基本分析
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --geo US \
  --output ./output/health_insurance.json

# 比較多個主題
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --compare "Unemployment,Inflation" \
  --geo US \
  --output ./output/comparison.json

# 跳過 related queries（更快、更少請求）
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --no-related \
  --output ./output/health_insurance.json

# Debug 模式（顯示瀏覽器、保存 HTML）
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --debug \
  --no-headless

# 登入模式（預設等待 120 秒供 2FA 驗證）
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --output ./output/health_insurance.json

# 跳過登入等待（不需要登入時）
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --login-wait 0 \
  --output ./output/health_insurance.json

# 從已下載的 CSV 檔案分析（跳過瀏覽器抓取）
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --csv ./downloads/multiTimeline.csv \
  --output ./output/health_insurance.json

# 自動從 Downloads 目錄找最新 CSV
python scripts/trend_fetcher.py \
  --topic "Health Insurance" \
  --csv auto \
  --output ./output/health_insurance.json

CLI Parameter Description:

Parameter	Description	Default Value
--topic	Search topic (required)	-
--geo	Region code	US
--timeframe	Time range	2004-01-01 2025-12-31
--threshold	Anomaly z-score threshold	2.5
--compare	Comparison topics (comma-separated)	-
--no-related	Skip related queries	false
--no-headless	Show browser window	false
--login	Force enable login mode	false
--login-wait	Login wait seconds (0=interactive Enter)	120
--csv	CSV file path or 'auto' to find automatically	-
--debug	Enable debug mode	false
--output	Output JSON file path	-
</quick_start>

<success_criteria> When the Skill executes successfully:

• Selenium successfully starts and simulates a browser
• Correctly scrape Google Trends time series
• Determine ATH status and anomaly score
• Identify signal types (seasonal/event/regime)
• Extract related queries driving terms (if enabled)
• Output structured JSON results </success_criteria>

<anti_detection_strategy> Anti-detection strategy summary

This skill implements the following anti-detection measures (based on design-human-like-crawler.md):

Strategy	Effect	Priority
Remove navigator.webdriver	core, to prevent JS detection	🔴 Required
Random User-Agent	Avoid fixed UA being identified	🔴 Required
Random delay before requests	Simulate human behavior	🔴 Required
Disable automation extensions	Remove Chrome traces	🟡 Recommended
Visit homepage before API	Establish a normal session	🟡 Recommended

Chrome options configuration:

chrome_options.add_argument('--disable-blink-features=AutomationControlled')
chrome_options.add_experimental_option('excludeSwitches', ['enable-automation'])
chrome_options.add_experimental_option('useAutomationExtension', False)

</anti_detection_strategy>