Real-time crackmeter-based warning systems for rainfall-induced small-scale slope failures suffer from prohibitively high false alarm rates. Across diverse geological settings, existing approaches relying on fixed thresholds or isolated rainfall–displacement relationships are highly sensitive to noise, lacking the robustness to reliably distinguish genuine deformation signals from spurious measurements. To address this operational bottleneck, this study reframes the early warning paradigm from a traditional fixed-threshold alerting process reliant on manual discrimination to an automated alert truth verification problem, proposing a real-time binary classification framework to isolate true early warnings from sensor-triggered false alarms. Utilizing a multi-source dataset of crackmeter displacement and rainfall measurements from diverse monitored slopes in Fujian Province, China, all true early warning instances were rigorously calibrated via field investigations. A patch-based dual-branch temporally-aware Transformer model was developed to explicitly address asynchronous multi-rate data fusion, strict temporal causality between rainfall and displacement, and stringent real-time decision constraints. By simultaneously capturing long-term rainfall-displacement interactions and high-resolution displacement dynamics, the model outperforms competitive baselines, achieving a precision of 90.91 %, a recall of 93.53 %, and an F2-score of 92.99 % in identifying true early warnings. Interpretability analysis reveals the model’s decisions are primarily driven by localized displacement trends and relative rainfall intensity, aligning with expert judgment. The proposed framework significantly curtails false alarms without compromising reliability, acting as a robust decision-support layer to enhance automated slope hazard monitoring. Future work will fuse additional sensor types to suppress false alarms via cross-validation of multi-physical responses.