Diffusion models provide a promising way to model the distribution of implicit structural models, potentially improving generalization across surveys. However, existing diffusion Transformer pipelines scale poorly to high-dimensional geophysical data because noise- or velocity-prediction objectives are often unstable at large patch sizes, forcing the use of small patches that lead to long token sequences and high computational cost. To reduce computation, most approaches rely on Variational autoencoders (VAEs) and latent diffusion, but robust pretrained VAEs are scarce in geophysics, and enforcing geological priors in latent space is difficult. To address these scalability bottlenecks and the difficulty of enforcing geological priors in latent space, we propose <em><strong>L</strong>atent-Compression-<strong>F</strong>ree Generative <strong>D</strong>iffusion (LFD) with Geological Priors and Geophysical Regularization</em> for implicit structural modeling. Built on flow matching, LFD generates implicit structural models directly in the data space, enabling efficient large-patch Vision Transformer (ViT) inference and allowing fault/horizon constraints and geophysical regularization to be applied explicitly during generation. To strengthen structural conditioning, we design a structure-enhanced Transformer that injects horizon and fault embeddings at multiple layers. We further introduce two prior-guided losses: a horizon loss to match the generated models to the input horizons, and a fault-aware bending-energy term that regularizes smoothness while ignoring stencils across faults. By enforcing these priors directly in the data space, the model is effectively constrained to generate geologically reasonable structures. Experiments on both synthetic data and real surveys validate the effectiveness of LFD for prior-guided implicit structural modeling. Benefiting from large-patch inference, LFD generates a 512x512 model in 1.56 s on an NVIDIA H20 GPU. With relative positional encoding, LFD can be extended to higher resolutions via simple adaptation without retraining. Overall, LFD offers new insights into deploying diffusion models for high-dimensional geophysical data, enabling efficient generation with interpretable, prior-guided constraints.