puda-opentrons-vision-validation

Opentrons Vision Validation

Goal

Prevent Opentrons OT-2 runs from starting with the wrong physical deck setup. Before any BEARS PUDA workflow executes a physical Opentrons run, capture a fresh deck image, verify the slots used by the protocol, identify visible labware/items, and stop for user confirmation when anything is uncertain or mismatched.

This is a physical safety gate. PUDA protocol validation checks command structure; it does not prove the robot deck contains the expected labware.

When to Use

Use this skill before running any Opentrons protocol that depends on deck setup, including:

liquid transfer, mixing, aspiration, dispense, or plate workflows
protocols with tipracks, source plates, destination plates, reservoirs, vials, or modules
colour-mixing or viscosity workflows that generate/run OT-2 protocols
any user request that names deck slots or labware placement

Do not use it as a substitute for Opentrons protocol validation. Use both: validate protocol structure first, then visually validate physical setup before robot execution.

Critical Rule

Before any physical Opentrons run, perform a vision validation gate:

Extract the expected deck map from the user request and protocol.
Capture a fresh image of the OT-2 deck.
Inspect every slot used by the protocol.
Identify occupied slots and visible labware/items.
Compare expected vs observed deck setup.
Ask the user to confirm uncertain labware or any mismatch.
Do not execute robot motion until the visual check passes or the user explicitly approves proceeding despite the risk.

If exact labware identity is unclear from the image, ask the user. Do not guess exact Opentrons labware definitions from appearance alone.

OT-2 Deck Slot Layout

Use the standard OT-2 deck layout:

Row	Slots
Back	10, 11, 12
Middle	7, 8, 9
Front/middle	4, 5, 6
Front	1, 2, 3

Vertical relationships:

Slot 10 is above slot 7.
Slot 11 is above slot 8.
Slot 12 is above slot 9.

If a slot is outside the camera view or obscured, mark it not visible or obscured. Do not infer occupancy from neighbouring slots.

Vision Validation Workflow

1. Extract the Expected Deck Map

List every slot used by the planned Opentrons run:

Slot	Expected labware/item	Role
8	300 µL tiprack	tips
3	mixing plate	destination

Completion criterion: every slot the protocol will use has an expected item, and any required-empty slots are explicitly listed.

2. Capture a Fresh Deck Image

Prefer the opentrons machine capture_image command when available:

puda machine commands opentrons
# create/run a one-command protocol:
# name: capture_image
# machine_id: opentrons
# params.filename: opentrons-vision-<UTC timestamp>.jpg

If PUDA capture fails with No camera configured. Pass camera_index when constructing Opentrons., use the configured BEARS RTSP fallback without restarting edge services:

mkdir -p /home/opentron/temp_opentrons/captures /tmp/puda-latest-images
ffmpeg -y -rtsp_transport tcp -i rtsp://100.102.45.58:8554/cam0 \
  -frames:v 1 -q:v 2 /home/opentron/temp_opentrons/captures/<filename>.jpg
cp /home/opentron/temp_opentrons/captures/<filename>.jpg /tmp/puda-latest-images/<unique-copy>.jpg
stat -c '%n %s bytes %y' /tmp/puda-latest-images/<unique-copy>.jpg
sha256sum /tmp/puda-latest-images/<unique-copy>.jpg

Completion criterion: a new image file exists, has non-zero size, and its path/hash are recorded.

3. Inspect the Image Conservatively

For each expected slot, report:

whether the slot is visible
whether the slot is occupied
visible labware/item description
confidence: high, medium, or low

Also report any unexpected occupied or obstructed slot that could affect the run.

Use conservative language:

appears to be a 300 µL tiprack is acceptable when visually likely but not certain.
black multi-well rack/plate; please confirm exact labware is better than guessing.
not visible or obscured is better than assuming.

Completion criterion: there is a slot-by-slot observation table for all expected slots plus any unexpected occupied/obstructed slots.

4. Compare Expected vs Observed

Use this status table:

Slot	Expected	Observed	Status
8	300 µL tiprack	appears to be 300 µL tiprack	OK / needs confirmation
3	mixing plate	glass vial	MISMATCH

Status values:

OK — expected slot is occupied by the expected labware/item with high confidence.
needs confirmation — slot is occupied but exact labware identity is uncertain.
MISMATCH — observed item differs from expected, expected slot is empty, or unexpected item blocks a required empty slot.
not visible — camera cannot verify.

Completion criterion: every protocol slot has one status.

5. Gate Execution

If every required slot is OK, continue with the normal PUDA Opentrons run workflow.
If any slot is needs confirmation, ask the user to confirm before running.
If any slot is MISMATCH or not visible, do not run. Ask the user whether to correct/reposition the deck or explicitly approve proceeding despite the risk.
If the user corrects labware identity or slot occupancy, update the current run's deck map and re-check the protocol against it.
If the correction changes the required setup, patch or regenerate the protocol and re-validate before running.

Completion criterion: the user's approval/correction is recorded before any physical robot execution.

Example

User asks for a liquid transfer using Opentrons with:

300 µL tiprack on slot 8
mixing plate on slot 3

Expected deck map:

Slot	Expected item	Role
8	300 µL tiprack	tips
3	mixing plate	source/destination

Required pre-run check:

Capture a fresh deck image.
Inspect slot 8 and slot 3.
Report the comparison:

Slot	Expected	Observed	Status
8	300 µL tiprack	appears to be 300 µL tiprack	OK
3	mixing plate	glass vial	MISMATCH

Stop and ask the user to confirm/correct slot 3 before executing the transfer.

Common Pitfalls

Guessing labware from appearance. Ask the user when exact labware identity is uncertain.
Ignoring user corrections. If the user says a slot is empty or names labware, immediately incorporate that correction into the current deck map.
Forgetting back-row slots. Slot 10 is above 7, slot 11 above 8, and slot 12 above 9.
Mistaking artifacts for occupancy. Transparent lids, reflections, cables, and neighbouring labware can look like slot occupation.
Running after a mismatch. Stop until the user corrects the deck or explicitly approves proceeding.
Treating puda protocol validate as physical validation. It is not; vision validation is separate.

Verification Checklist

Expected deck map extracted from the user request/protocol.
Fresh deck image captured and verified with path/size/hash.
Every expected slot inspected.
Unexpected occupied/obstructed slots reported.
Expected-vs-observed table produced.
User asked to confirm uncertain or mismatched labware.
Robot execution only proceeds after visual validation passes or user explicitly approves.