Real-time Fish Movement Prediction

Summary

I built a full pipeline that goes from camera frames to a live forecasted path. A YOLO detector finds fish and a Kalman tracker with Hungarian assignment maintains IDs through crossings and partial occlusion. An LSTM sequence model predicts 10 seconds ahead using a 20-second history window. The server keeps a rolling history per track so model inputs match training, and I exposed debug endpoints to verify class mapping, feature windows, and to detect collapse-to-center issues. The UI records clips and provides human-versus-model path comparisons so the system is measurable and easy to demo.

What I built

• Training dataset: converted multi-hour video to per-frame CSVs at 15 FPS, normalized coordinates, derived velocities, and labeled tracks.
• Detector and tracking: trained YOLOv8; implemented Kalman filtering and Hungarian assignment with active, grace, and inactive pools to keep IDs stable.
• LSTM forecaster: sequence-to-sequence model with 20 s lookback and 10 s prediction; inputs assembled as relative position, velocity, one-hot class, and origin; avoided center-collapse by including velocity features and regularization.
• Evaluation and baseline: compared against a constant-velocity extrapolation; reported ADE, FDE, and Hit@0.05 alongside mAP and IDF1 for detection and ID.
• Jetson application: Flask server with MJPEG stream, rolling history buffers, consistent class list loading via Keras 3, and CV fallback if the predictor fails.
• UI and playback: select a fish, draw a path, and replay 10 s clips; overlay shows selection box following the actual path, with legend and running score.
• Diagnostics: endpoints for class_list, feature-window inspect, collapse probe, and model_path so I can sanity-check inputs and predictions in production.

Results

• Stable multi-target IDs during crossings; fewer ID switches after adding grace/inactive pools.
• Trajectory forecasts that exceed a constant-velocity baseline on ADE/FDE and Hit@0.05 across test clips.
• Demo-ready UI that makes performance visible to non-technical users and encourages fast iteration.

Notes

• Standardized capture to 15 FPS to avoid 30 FPS desync and camera thread issues; isolated camera runs to prevent hung threads.
• Next step: embed the Jetson MJPEG viewer directly into the project page and gate CORS to production domain.