High-resolution rainfall fields are essential for hydrometeorological applications such as flood forecasting and urban drainage modelling. In practice, however, observations are often limited to sparse and irregular rain gauge networks, making it difficult to reconstruct spatial–temporal rainfall structures and maintain temporal continuity. Conventional interpolation-based approaches struggle under these conditions, particularly when observations are highly sparse. This study proposes a data-driven framework, termed P2I-GAN, to reconstruct rainfall fields directly from irregular point measurements. Inspired by the concept of video inpainting in computer vision, the method learns spatial–temporal rainfall structures from radar observations and applies this knowledge to infer rainfall fields from sparse gauge data. This allows spatial organisation of rainfall to be recovered in a temporally consistent manner, even when observations are limited. Evaluation results show that the proposed approach produces realistic rainfall structures while maintaining strong performance in standard statistical metrics, outperforming conventional interpolation methods and remaining competitive with existing learning-based approaches. The framework provides a practical pathway for reconstructing high-resolution rainfall fields from sparse observation networks.