The accurate classification of infrasound signals is a cornerstone of global geophysical monitoring, essential for both natural hazard early warning systems (e.g., volcanic eruptions, debris flows, and earthquakes) and the verification of the Comprehensive Nuclear-Test-Ban Treaty. However, the development of reliable automated systems is hindered by the inherent scarcity of representative event data, particularly for rare extreme events, as well as the presence of complex, non-stationary background interferences. To maximize the diagnostic value of limited geophysical datasets, this paper proposes Infra-Net, a novel parallel decision-making network driven by multi-view feature learning and a confidence-based fusion mechanism. We introduce a logarithmic wavelet scattering transform to produce robust, mathematically grounded feature representations. Unlike conventional methods that process scattering matrices holistically, our approach treats individual columns as independent feature vectors, providing multiple localized perspectives of the same acoustic event. Architecturally, Infra-Net utilizes a dual-branch structure to simultaneously capture multi-scale spatial features and discriminative temporal patterns. These parallel evaluations are synthesized through a custom confidence-based fusion module, which employs weighted averaging and an inner-product mechanism to ensure a stable and comprehensive final classification. Tested on both public infrasound datasets and real-world CTBTO-measured data, Infra-Net achieved accuracies of 100 % and 82.07 %, respectively. These results demonstrate that Infra-Net offers a highly robust soft computing solution for enhancing Earth system monitoring and the reliable identification of natural hazards amidst anthropogenic noise.