atai-rare-event-detection-agent-data-prep

RED Data Prep — a labelled recording → role files

The RED agent learns one prototype per class from a handful of labelled incidents and normal operation, and flags windows that sit closer to an event's prototype than to normal. With so few examples, a few prep decisions carry the whole result, and the ways to get them wrong rarely error: a channel that goes dark because the pump failed becomes a perfect but meaningless signature; an incident shorter than half a window is all but undetectable at that window; two incidents close together share their normal context between validation and test; channels chosen on held-out incidents score optimistically. This skill makes each decision visible and checks every rule before you upload.

one recording (any rate, jitter, gaps, missing readings)
  + a per-row label column     prepare.py → check_events.py → build_roles.py → preflight_roles.py → RED agent skill, Path 2

Everything here is local and needs no API key. The contract is references/role-files.md. It is the layout of the RED pump quickstart, generalised; the platform rules in it were measured on the dev deployment on 2026-10-13.

When to Apply

• You have a long recording of an asset or process with a label per row, and the label marks a few incidents of a named event (a breakdown, a trip, a slip) against normal operation
• You want a RED agent that recognises the same event when it recurs, fitted on the first incidents and tested on later ones
• A RED optimization or eval scored oddly (an event class at 0, every window invalid, a suspicious false-alarm rate) and you want to check the role files

Do not use this skill when:

• You only want to run the maintained RED Quick Start bundle on one CSV — that's atai-rare-event-detection-agent, Path 1
• You have no labelled incidents, only normal operation — that's anomaly discovery: atai-anomaly-discovery-agent-data-prep
• Your labels are operating states that cover the whole recording (fill / wash / spin) — use atai-operational-state-monitoring-agent-data-prep

Your input

One recording (CSV or Parquet): a timestamp column, the sensor channels (numeric), and a label column.

,timestamp,sensor_00,sensor_01,...,sensor_51,machine_status
0,2018-04-01 00:00:00,2.465394,47.09201,...,201.3889,NORMAL

• Timestamps: epoch seconds, epoch milliseconds or ISO 8601. Jitter is snapped to the grid; a gap (more than 1.5 steps) starts a new segment and is never filled.
• Labels: any values; you map them to classes with --classes, background first: normal=NORMAL pump_breakdown=BROKEN,RECOVERING. Several event classes are fine (trip=TRIP slip=SLIP). An unmapped value stops the script.
• Missing readings: leave them. Step 2 decides per channel, and step 3 interpolates the short runs it keeps.

Roles are assigned per incident, by its number from step 1. Use the first incidents for training, the next for validation (choosing settings), later ones for the test (scored once), and deliver test incidents again, or new operation after them. Time order matters: a deployment only ever has the past to learn from, and every choice here is made on data from before the first held-out incident.

The steps (on the shipped sample)

references/sample_data/ holds the first four breakdowns of the Kaggle pump recording, raw: 82,492 rows a minute apart, 15 of its 52 channels (including ones that go dark during breakdowns, a long gap, an empty one), ISO timestamps, machine_status. Everything below runs on it in about 3 s:

cd skills/atai-rare-event-detection-agent-data-prep/references
pip install -r requirements.txt

1. prepare.py — an exact grid, classes, numbered incidents

python prepare.py --input sample_data/pump_sample.parquet --label-col machine_status \
    --classes normal=NORMAL pump_breakdown=BROKEN,RECOVERING --out prepared

Sorts by time, drops duplicate timestamps, snaps jitter to the grid (the median interval), numbers segments at gaps, maps the labels and numbers each incident (a run of one event class inside one segment). Values are untouched. Expect:

82,492 rows, every 60 s, 15 numeric channels (1 other columns dropped: Unnamed: 0)
grid: 0 timestamps snapped, 0 rows off the grid dropped, 1 segment(s)
classes: normal 76,517, pump_breakdown 5,975
4 incidents:
  #1   pump_breakdown     rows    17,155-18,100        945 rows  from 2018-04-12 21:55
  #2   pump_breakdown     rows    24,510-27,621      3,111 rows  from 2018-04-18 00:30
  #3   pump_breakdown     rows    69,318-70,631      1,313 rows  from 2018-05-19 03:18
  #4   pump_breakdown     rows    77,790-78,396        606 rows  from 2018-05-25 00:30

Why an exact grid: the red blueprint checks that a window's timestamps are monotonic and evenly spaced (each interval within 5% of the window's mean); a window failing either comes back INVALID_STATE, not classified.

2. check_events.py — can these incidents and channels make a RED agent?

python check_events.py --prepared prepared --training 1,2 --validation 3 --test 4

On data from before the first held-out incident only:

check	FAIL / WARN when
incidents	a role has none (FAIL); an event class has no training incident, so no prototype (FAIL), or is never scored (WARN)
windows, at --windows 16,32,64,128,256	a training class yields no window (FAIL); a scored incident cannot fill half a window (WARN): the classifier compares whole windows, so it would barely move one
channels	the triage step 3 applies: dead; dropout (missing on over 1% of rows; WARN when the missing rows sit inside events: interpolated, the sensor going dark would become the signature); long gap (a missing run over 16 rows, more than the platform repairs). Zero-saturation (exactly 0 in most event rows, almost never otherwise) is kept with a WARN: a real reading of a stopped machine, which makes the task "has it stopped?"
separation	a selected channel separates the held-out incidents less than half as well as the training ones (WARN)

Expect 0 FAIL, 7 WARN: sensor_00 dropped because its dropout tracks the breakdowns; sensor_01, 06 and 15 dropped; four zero-saturating channels; and

separation (mean |z| against build-time normal)
  sensor_04      training 10.02   held-out  6.43  <- selected
  sensor_05      training  8.82   held-out  8.59  <- selected
  ...
WARN  sensor_02: separation does not transfer (4.73 on training incidents, 2.06 held out)
WARN  sensor_03: separation does not transfer (2.87 on training incidents, 0.84 held out)

The warnings are for reading, not fixing: they say what the agent will key on (a stopped pump reads zero current) and which channels are weaker evidence than they look.

3. build_roles.py — the role files

python build_roles.py --prepared prepared --out roles --training 1,2 --validation 3 --test 4 --delivery 4

Triage and selection as reported in step 2 (the --top 10 channels by training separation, or exactly --channels ...); short missing runs interpolated; every channel z-scored with the build-time normal rows; then:

• training: normal.csv (the first --normal-rows 8192 of a normal run before any held-out incident) and one file per training incident, each one class, no label column;
• validation, test: one file per incident with --context 4096 normal rows either side, cut at the midpoint where incidents are closer, with a label column;
• delivery: the --delivery test incidents with --delivery-context 1024 rows either side, no label column, labels in labels/; or --tail-chunk N for the rows after the last scored file, in chunks (new operation: false alarms per day).

Expect:

channels (10): sensor_02 sensor_03 sensor_04 sensor_05 sensor_10 sensor_11 sensor_12 sensor_38 sensor_41 sensor_42
z-scored with normal rows before row 65,222
training    3 files    12,248 rows    4,056 event rows
validation  1 files     8,988 rows    1,313 event rows
test        1 files     8,282 rows      606 event rows
delivery    1 files     2,654 rows      606 event rows

The same 10 channels and row ranges as the pump quickstart's. Its z-scores differ: it took its statistics from the first 30,000 rows, this from every normal row before the first held-out incident.

4. preflight_roles.py — check every rule

python preflight_roles.py --roles roles --windows 16,32,64

Per file: the manifest's channels in order, an exact grid with no jump, finite values, row counts. Across files: no row in two of training, validation and test; delivered incidents inside their test files, tail files apart from everything; labels only in validation and test, with the background and the incident's class; each training file one class, and every class trained; delivery labels row for row; every scored incident able to fill half of each window you will search. Expect: 16 PASS, then all checks pass.

Then run the RED agent skill's Path 2 on roles/.

The rules, and what breaks

Each rule, and why the platform needs it, is in role-files.md. The ones that fail silently, as the tests here exercise:

broken	what happens on the platform
a normal row in an incident's training file	the event's prototype moves toward normal: fewer detections, no error
a channel that goes dark during events, interpolated	the agent learns "sensor offline", which tells you nothing about the machine
validation and test sharing normal context	the test's false-alarm rate is partly a validation number
a time jump inside a file	every window across it is INVALID_STATE
an incident shorter than half the window	at that window it barely moves the embedding: missed, with nothing flagged

Pitfalls

• The agent recognises the event's state, not its cause. On the pump, BROKEN is one row per incident and RECOVERING the rest; folding them into one class detects "the pump is down", not the failure. Choose the label mapping for what an operator needs to know.
• Choose the window from an operational requirement: the shortest event the deployment must catch, from the people who run the asset. Deriving it from the test incidents' lengths leaks them into model selection.
• Keep the z-score statistics (manifest.json → zscore): any input the deployed agent sees must be scaled the same way.
• More normal rows barely change a prototype (it is a mean) but cost encoder time in every trial; --normal-rows is a cost knob more than an accuracy one.

Local Setup

cd skills/atai-rare-event-detection-agent-data-prep/references
pip install -r requirements.txt        # numpy, pandas, pyarrow

Tests:

pip install pytest
pytest skills/atai-rare-event-detection-agent-data-prep/tests/ -v

File Layout

skills/atai-rare-event-detection-agent-data-prep/
├── SKILL.md
├── references/
│   ├── prepare.py            ← 1. exact grid, classes, numbered incidents
│   ├── check_events.py       ← 2. incidents, windows, channels, separation
│   ├── build_roles.py        ← 3. the role files (contract red-role-files/v1)
│   ├── preflight_roles.py    ← 4. every rule
│   ├── red_common.py         ← shared: reading, incidents, slices, channel rules
│   ├── role-files.md         ← the contract and why each rule exists
│   ├── requirements.txt
│   └── sample_data/          ← the pump's first four breakdowns, raw, with attribution
└── tests/test_red_prep.py    ← synthetic end to end, and role files broken on purpose

Data attribution

The sample is derived from the Kaggle dataset pump_sensor_data by nphantawee (https://www.kaggle.com/datasets/nphantawee/pump-sensor-data); credit for the underlying data belongs to the operations team who shared it. No licence is declared on the dataset. Full attribution and what was changed: references/sample_data/README.md.