agentic-data-science-competition

Agentic Data Science Competition

「The agent doesn't just submit — it learns from failures, adapts strategies, and iterates autonomously.」

Core Philosophy

This skill distills practical patterns from real competition experience:

Agents as teammates — Not just tools, but collaborators that can research, debug, and iterate
Spec-driven development — Document before coding, delegate with clear constraints
Fail fast, learn faster — Early scores are misleading; systematic debugging wins
Automation where it matters — Cronjobs for monitoring, delegation for complex work

Quick Reference

When to Use This Skill

Trigger	Action
Starting a new competition	→ Read "Project Setup" section
Submission returns 400 error	→ Check "Troubleshooting" section
Score dropped unexpectedly	→ Read "Score Stabilization" section
Need to replicate top notebook	→ Use "Replication Workflow"
Kernel push fails	→ Check "Kernel Workflow" section
GPU required but unavailable	→ Use "Delegation Strategy"

Key Commands

# Submit to competition
kaggle competitions submit <name> -f <file> -m "<message>"

# Check submission status
kaggle competitions submissions <name>

# Pull top notebook WITH metadata
kaggle kernels pull <owner>/<kernel> -p ./path/ -m

# Push kernel to Kaggle
kaggle kernels push -p ./path/

# Monitor kernel status
kaggle kernels status <username>/<kernel-name>

Score Stabilization Pattern

Critical insight: Kaggle scores take time to stabilize after submission.

|| Time | Score Behavior | What To Do || |------|----------------|------------|| | Start | Baseline | Submit early to start evaluation || | +2 hours | Peak (inflated) | Don't trust! Often artificially high || | +4 hours | Stabilized | True score — make decisions now ||

Lesson: Never celebrate early highs. Wait 4+ hours before judging performance.

Submission Troubleshooting

400 Bad Request Error

Check submission format (with/without header, quotes)
Verify IDs match test set exactly
Try .zip format — some competitions require zipping the CSV
Check if competition requires model submission vs answer submission

Zip Submission Format (Critical!)

Common mistake: Zipping everything in the folder (including __notebook__.ipynb)

Correct way:

import zipfile
with zipfile.ZipFile('submission.zip', 'w', zipfile.ZIP_DEFLATED) as zf:
    zf.write('submission.csv', 'submission.csv')  # Only the CSV!

Competition Types

Type	What You Submit	Examples
Answer Submission	CSV with predictions	Most competitions
Model Submission	Trained model weights (LoRA, checkpoints)	Some LLM competitions

Detection: Look at top notebooks — do they train models or just generate predictions?

Kernel Workflow

Run Mode vs Commit Mode

Mode	Test Set	Use Case
Run	Hidden	Development, debugging
Commit ("Save & Run All")	Mounted	Production, final submission

Why this matters: kaggle kernels push runs in Run mode. Test set is NOT mounted. Use sample_submission.csv for placeholder.

Data Path Pattern

/kaggle/input/competitions/<competition-name>/   ← Correct!
NOT /kaggle/input/<competition-name>/            ← Wrong!

Kernel Metadata Best Practices

{
  "id": "username/kernel-name",
  "is_private": true,          // ← Always true by default!
  "enable_internet": false,    // ← Check competition rules
  "competition_sources": ["competition-name"],
  "dataset_sources": ["dataset-with-dependencies"]
}

Replicating Top Notebooks

Workflow

# 1. Pull with metadata (-m flag is critical!)
kaggle kernels pull <owner>/<kernel> -p ./solution/ -m

# 2. Edit kernel-metadata.json
#    - Change "id" to your username/new-name
#    - KEEP all dataset_sources, model_sources, kernel_sources

# 3. Push
kaggle kernels push -p ./solution/

# 4. Monitor
kaggle kernels status <your-username>/<new-kernel-name>

Critical Points

Always use -m flag — gets kernel-metadata.json with dependencies
Preserve ALL dependencies — dataset_sources, model_sources, kernel_sources
Only change id and title — everything else should match original
Check enable_internet — if original has false, keep it false

When to Delegate

Delegate to OpenCode/Claude Code when:

Adapting solutions to different contexts
Combining multiple techniques
Integrating with existing codebase
Complex transformation or refactoring

Don't delegate when:

Simple file downloads
Metadata-only changes
Direct forks without modifications

Spec-Driven Development

Process

Document SPEC.md first before coding:
- Competition overview (task, data, evaluation)
- Top approaches from notebooks
- Strategy and implementation plan
- Technical constraints
- Success metrics and timeline
Delegate to AI coding agents with the spec:
- Provide SPEC.md as context
- Specify constraints (time limit, GPU availability)
- Set clear success criteria
Iterate based on results

SPEC.md Template

# SPEC.md - Competition Name

## 1. Competition Overview
- Task, deadline, prize, current top score

## 2. Task Definition
- Input/Output format, data summary

## 3. Top Approaches (from Kaggle Notebooks)
- Approach A: [votes/score], strategy
- Approach B: [votes/score], strategy

## 4. Strategy & Implementation Plan
- Phase 1: Quick baseline
- Phase 2: High-ROI improvements
- Phase 3: Fine-tuning

## 5. Technical Constraints
- Submission format, time limits, GPU requirements

## 6. Success Metrics
| Milestone | Target | Timeline |
|-----------|--------|----------|
| Baseline | 0.xx | Day 1 |
| Improved | 0.xx | Day 2 |

Delegation Strategy

Local vs Kaggle

Task	Where	Why
Data analysis	Local	Fast iteration, no GPU needed
Prompt engineering	Local	Quick testing
Model training	Kaggle	Free GPU
Large-scale inference	Kaggle	GPU + internet

When to Use Which Agent

Agent	Best For
OpenCode	Code development, feature implementation
Claude Code	PR review, complex debugging
Subagents (delegate_task)	Parallel research, isolated workstreams

Common Pitfalls

1. Debug Fallback to Training Data

Wrong:

test_files = sorted(TEST_DIR.glob("*.ogg"))
if not test_files:
    test_files = sorted((COMP_DIR / "train_soundscapes").glob("*.ogg"))[:10]

Correct:

if not test_files:
    submission = pd.read_csv("sample_submission.csv")
    submission.iloc[:, 1:] = 0.0
    submission.to_csv("submission.csv", index=False)

2. Game AI Competitions Require Comprehensive Features

Pattern observed: Top-performing game agents often require comprehensive feature implementations.

Takeaway: Don't oversimplify game AI agents. Study top solutions to understand the full feature set needed to compete.

3. Double File Extension

Check if filename column already includes extension:

# Check this:
print(train_df['filename'].head())

# If it shows "1161364/iNat1216197.ogg", don't add .ogg again!

4. Silent Exception Catching

Bad (hides errors):

try:
    audio, sr = load_audio(file)
except Exception:
    continue  # No idea why it failed!

Better:

try:
    audio, sr = load_audio(file)
except Exception as e:
    print(f"Failed on {file}: {type(e).__name__}: {e}")
    continue

API Usage Patterns

Rate Limit Pattern

Some AI APIs hit limits during peak hours (e.g., 10:00-17:00 Beijing time)
Solution: Switch to backup model/provider during limit window
Recovery: Limits clear in evening hours

Plan around this: Check time before delegating to AI coding agents.

Verification Checklist

After submitting:

Check submission shows in kaggle competitions submissions
Monitor status: PENDING → RUNNING → COMPLETE
Wait 4+ hours before evaluating final score
Compare against baseline and target

Advanced Patterns

See references/research/04-advanced-patterns.md for:

State Machine Execution Loop — Fault-tolerant autonomous agent pattern
Config-Driven Cronjob Design — Swappable experiments without recreating jobs
Supervisor Validation Duties — Fraud detection and provenance verification
Rank Averaging for AUC — +0.001–0.005 improvement technique
ONNX Model Validation — Forbidden ops and error debugging
Dependency Vendoring — Offline package installation

See references/research/05-environment-setup-patterns.md for:

Kaggle Environments Source Install — Install latest envs from GitHub when PyPI is outdated

See references/research/06-competition-types-submission-workflow.md for:

Competition Type Taxonomy — Prediction/Adapter/ONNX and Pure Code/Mixed/Standard
The -k -v Flag Requirement — Prevent orphan submissions with notebook linkage

Related Skills

Skill	Purpose
`agentic-competition-workflow`	Git-first project management, validation pipelines, full competition lifecycle
`kaggle-auto-submit`	End-to-end automation with cronjob monitoring
`autonomous-iteration`	ANALYSIS → BUILD → EXPERIMENT → REVIEW loops
`opencode`	Delegate coding to OpenCode CLI
`claude-code`	Delegate coding to Claude Code CLI