3D Gaussian Splatting (3DGS) has enabled efficient 3D scene reconstruction from everyday images with real-time, high-fidelity rendering, greatly advancing VR/AR applications. Fisheye cameras, with their wider field of view (FOV), promise high-quality reconstructions from fewer inputs and have recently attracted much attention. However, since 3DGS relies on rasterization, most subsequent works involving fisheye camera inputs first undistort images before training, which introduces two problems: 1) Black borders at image edges cause information loss and negate the fisheye's large FOV advantage; 2) Undistortion's stretch-and-interpolate resampling spreads each pixel's value over a larger area, diluting detail density and causing 3DGS to overfit these low-frequency zones, producing blur and floating artifacts.
In this work, we integrate a fisheye camera model into the original 3DGS framework, enabling native fisheye image input for training without preprocessing. Despite correct modeling, we observed that reconstructed scenes still exhibit floaters at image edges: Distortion increases toward the periphery, and 3DGS's original per-iteration random-selecting-view optimization ignores the cross-view correlations of a Gaussian, leading to extreme shapes (e.g., oversized or elongated) that degrade reconstruction quality. To address this, we introduce a feature-overlap-driven cross-view joint optimization strategy that establishes consistent geometric and photometric constraints across views, a technique equally applicable to existing pinhole-camera-based pipelines. Our DirectFisheye-GS matches or surpasses state-of-the-art performance on public datasets.
(a) Common fisheye camera projection models.
(b) Illustration of the 3DGS single-view training paradigm and our proposed cross-view joint optimization strategy.
We propose DirectFisheye-GS, a novel framework that enables native fisheye inputs in 3DGS while enhancing geometric and photometric consistency through cross-view joint optimization. We integrate the Kannala-Brandt fisheye projection model into the 3DGS pipeline, eliminating the need for undistortion preprocessing and preserving the efficiency of rasterization-based rendering. This not only ensures full compatibility with existing 3DGS viewers and commercial tools, but also retains the original spatial relationships among neighboring rays-- an essential condition for multi-view consistency constraints. We then introduce a cross-view joint optimization strategy that adaptively groups training views based on feature overlap and viewpoint divergence. By simultaneously optimizing Gaussians across correlated views, our method enforces geometric consistency and mitigates shape irregularities, significantly improving reconstruction quality without sacrificing efficiency.
Ablation study content coming soon.
VR demo content coming soon.
@misc{yang2026directfisheyegsenablingnativefisheye,
title={DirectFisheye-GS: Enabling Native Fisheye Input in Gaussian Splatting with Cross-View Joint Optimization},
author={Zhengxian Yang and Fei Xie and Xutao Xue and Rui Zhang and Taicheng Huang and Yang Liu and Mengqi Ji and Tao Yu},
year={2026},
eprint={2604.00648},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2604.00648},
}[1] Kannala, Juho and Brandt, Sami S. A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8):1335-1340, 2006.
[2] Liao, Zimu, Chen, Siyan, Fu, Rong, Wang, Yi, Su, Zhongling, Luo, Hao, Ma, Li, Xu, Linning, Dai, Bo, Li, Hengjie, and others. Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras. arXiv preprint arXiv:2409.04751, 2024.
[3] Wu, Qi, Esturo, Janick Martinez, Mirzaei, Ashkan, Moenne-Loccoz, Nicolas, and Gojcic, Zan. 3DGUT: Enabling Distorted Cameras and Secondary Rays in Gaussian Splatting. arXiv preprint arXiv:2412.12507, 2024.
[4] Deng, Youming, Xian, Wenqi, Yang, Guandao, Guibas, Leonidas, Wetzstein, Gordon, Marschner, Steve, and Debevec, Paul. Self-Calibrating Gaussian Splatting for Large Field of View Reconstruction. arXiv preprint arXiv:2502.09563, 2025.