Anomaly detection on time series data for autonomous robot operation

Abstract

Autonomous robots deployed in safety-critical applications require real-time monitoring systems to detect both environmental and mechanical anomalies. While anomalies in visual sensor data (changing lighting, unexpected obstacles) are often obvious, mechanical failures – such as motor malfunctions or wheel slippage – may not be directly observable in images, making detection challenging. Additionally, real-world operational data is severely imbalanced: normal behavior is well-represented, but anomalous events are rare. This imbalance renders traditional supervised learning approaches ineffective. To address these challenges, this paper presents a staged multimodal autoencoder architecture – a neural network system that processes both visual information (RGB camera images) and motion sensor data simultaneously. Unlike conventional multimodal systems that train all components jointly and suffer from modality competition, proposed architecture employs a three-stage training curriculum that trains visual and motion encoders independently before joint optimization, preventing gradient imbalance and ensuring robust representations. The system performs anomaly detection through reconstruction error analysis: lower errors indicate normal operation patterns, while deviations signal potential anomalies. The method requires only normal operational data for training – no labeled anomaly samples are necessary. Experimental validation demonstrates that the architecture detects visual anomalies (color distortions, unexpected objects) and motion anomalies (sudden stops, jerks, velocity changes) in real-time. The proposed method is needed for safety-critical applications like autonomous robot navigation and warehouse automation, where detecting mechanical and environmental anomalies is essential for operational safety.