Spherical Pixel-Aligned Gaussians (SPAG)

1. Introduction

1.1 Motivation

3D Gaussian Splatting (3DGS) [Kerbl et al., 2023] has revolutionized real-time novel view synthesis by representing scenes as collections of anisotropic 3D Gaussians. However, the optimization-based nature of 3DGS presents several challenges:

1. No pixel correspondence: Optimized Gaussians have no direct mapping to source image pixels
2. Editing difficulty: Modifying the 3D scene requires re-optimization or complex manipulation
3. Multi-view requirement: Traditional 3DGS needs multiple calibrated images
4. Optimization time: Minutes to hours of training per scene

For 360° content creation—VR tours, real estate visualization, immersive media—these limitations are particularly problematic. Users often want to quickly generate explorable 3D from a single panoramic capture, edit the scene by painting/modifying the panorama, and see changes reflected in 3D instantly.

1.2 My Contribution

I introduce SPAG (Spherical Pixel-Aligned Gaussians), which:

1. Maintains 1:1 pixel-to-Gaussian correspondence: Every pixel (u,v) maps to exactly one Gaussian
2. Enables single-shot generation: No optimization, instant 3D from monocular depth
3. Supports direct editing: Paint on panorama → modify 3D scene
4. Handles spherical distortion: Novel pole-region reconstruction for equirectangular artifacts

2. Related Work

2.1 3D Gaussian Splatting

Kerbl et al. [2023] introduced 3DGS as a point-based alternative to NeRF, achieving real-time rendering through rasterization of 3D Gaussians. Extensions include Mip-Splatting for anti-aliasing, 4D Gaussian Splatting for dynamic scenes, and GaussianEditor for text-guided editing.

2.2 Gap in Literature

No existing work combines: Gaussian splatting representation, single 360° image input, direct pixel editability, and real-time generation.

3. Method

3.1 Spherical Coordinate System

For an equirectangular panorama of dimensions W×H, I define normalized pixel coordinates:

The unit direction vector for each pixel is:

3.2 Pixel-Aligned Gaussian Positioning

Given depth map D(u,v) from monocular estimation (e.g., DA-2), the 3D position of each Gaussian is:

Key Property: This creates a bijective mapping between pixels and Gaussians.

3.3 Gaussian Parameters

Total parameters: ~59 (3DGS) vs ~8-14 (SPAG) — 75-85% reduction

3.4 Latitude-Aware Scaling

Equirectangular projection causes area distortion. To maintain uniform visual density:

4. Editability Framework

4.1 The Editing Paradigm

Traditional 3DGS editing requires identifying Gaussians to modify (spatial selection), applying transformation, and potentially re-optimizing for consistency.

SPAG editing is direct: modify pixel(s) in panorama → update corresponding Gaussian(s) → done.

4.2 Edit Operations

4.3 Complexity Comparison

Traditional 3DGS Edit:
┌─────────────────────────────────────────────────────────┐
│ User Edit → Spatial Query O(log n) → Select Gaussians   │
│     → Modify Parameters → Re-render → Compute Loss      │
│     → Backpropagate → Update ALL Gaussians → Repeat     │
│                     (1000s of iterations)               │
└─────────────────────────────────────────────────────────┘
                    Total: O(n) + O(iterations × n)

SPAG Edit:
┌─────────────────────────────────────────────────────────┐
│ User Edit at (u,v) → index = v*W + u → Update G[index]  │
│                         DONE                            │
└─────────────────────────────────────────────────────────┘
                    Total: O(1)

5. Implementation

5.1 Pipeline Overview

Input: Equirectangular panorama I (W×H×3)
       ↓
[Monocular Depth Estimation] → Depth map D (W×H)
       ↓
[Spherical Projection] → 3D positions P (W×H×3)
       ↓
[Pole Reconstruction] → Fixed floor/ceiling
       ↓
[Gaussian Assembly] → SPAG scene {μ, s, q, c, α}
       ↓
Output: Renderable 3D Gaussian Splat (.ply)

5.2 Complexity Analysis

6. Experimental Results

6.1 Reconstruction Quality

Note: SPAG trades some quality for instant generation and editability

6.2 Editing Speed Comparison

7. Limitations and Future Work

7.1 Current Limitations

1. View quality degradation: Novel views far from capture point show artifacts
2. No view-dependent effects: Specular reflections not captured (no SH)
3. Depth estimation errors: Relies on monocular depth quality
4. Single viewpoint: Designed for single-capture scenarios

7.2 Future Directions

1. Multi-capture fusion: Extend SPAG to multiple panoramas with alignment
2. Learned refinement: Light optimization pass for quality improvement
3. Semantic editing: Combine with segmentation for object-level edits
4. Real-time depth: On-device depth estimation for live capture

8. Conclusion

I presented Spherical Pixel-Aligned Gaussians (SPAG), a novel representation bridging 2D panoramic imagery and 3D Gaussian scenes. By maintaining direct pixel correspondence, SPAG enables an intuitive editing paradigm where users modify the familiar 2D panorama and see instant 3D updates.

While trading some novel-view quality for speed and editability, SPAG opens new possibilities for interactive 360° content creation, VR tour generation, and accessible 3D editing.

References