VisArch

Immersive computing is moving beyond basic overlays into complex, real-world deployments. In this talk, I will highlight several cutting-edge XR applications developed in our lab, including solutions for healthcare and human-robot interaction, that rely on tight integration with edge computing and advanced sensing pipelines. As these systems become more capable, ensuring their safety and reliability becomes a critical bottleneck.

When deployed in unconstrained environments, mixed reality experiences can fail in subtle but consequential ways. Virtual objects may drift or occlude critical real-world elements; spatial audio cues may contradict visual content; and MR-based guidance may suggest actions that are impossible or unsafe. These failures are highly context-dependent: content that is useful in one environment can be unsafe in another. They therefore cannot be detected by classical signal processing or conventional ML approaches alone.

Recent large multimodal models (LMMs) provide a new mechanism for monitoring immersive systems. By jointly reasoning over video streams captured by MR devices, rendered virtual content, spatial audio, and scene context, these models have the potential to identify misaligned overlays, unsafe occlusions, inconsistent audio-visual cues, and policy violations in MR applications. In this talk, I will present an LMM-based MR evaluation pipeline that uses LMMs together with auxiliary ML models to analyze continuous multimodal streams and trigger corrective actions when problematic virtual content is detected. A key challenge in LMM-based XR evaluation is reducing inference latency to support near real-time operation. I present a collection of system design choices and lay out a research agenda with the potential to bring down the cost and latency of these systems.

Spatial intelligence requires machines to continuously construct, maintain, and query representations of the 3D world under stringent latency, energy, memory, and form-factor constraints. Yet neural 3D techniques have largely emerged as offline, GPU-intensive, and representation-specific pipelines. In this talk, I will present our efforts to transform neural 3D into a deployable computing substrate. Our early work made neural scene reconstruction and rendering practical on resource-constrained devices. We then moved the architectural boundary from isolated kernels to the complete reconstruction-rendering lifecycle, including integrated chip and multi-chip systems. More recently, to keep pace with rapidly evolving representations, we developed workload-aware optimizations, GPU and graphics-pipeline extensions, and reconfigurable architectures that remain efficient across neural radiance fields, Gaussian splatting, and hybrid neural renderers. Together, these works reveal three principles for architecting emerging visual systems: optimize the work that hardware actually executes; co-design the full lifecycle of persistent 3D state; and specialize what is stable while reconfiguring what continues to change. I will conclude with opportunities in persistent spatial memory, continuous world updates, geometry foundation models, and shared 3D state for rendering, reasoning, and embodied action.

Recent advances in AI algorithms, from LLM/VLM reasoning to neural rendering, have demonstrated impressive capabilities, but they also impose substantial computational and memory overheads on modern systems, limiting their deployment in real-world settings. Advancing the efficiency frontier of these AI workloads requires cross-stack co-design across algorithms, systems, and hardware, so that large-scale deployment becomes practical, not just possible, for a broad range of applications. This talk presents a cross-stack optimization pathway for efficient multimodal AI, showing how algorithm-system-hardware co-design can improve system performance while preserving algorithmic quality.

With the rapid development of 3D Gaussian Splatting (3DGS) in virtual reality, spatial computing, and related fields, the demand for dedicated hardware support has been steadily increasing. As an emerging rendering technique, Gaussian splatting not only significantly improves rendering quality, but also demonstrates broad application potential in complex scenes. However, practical deployment of 3DGS still faces a series of challenges, including strong data dependencies, high computational intensity, and incompatibility with existing GPU architectures. This talk focuses on the rendering pipeline of 3DGS and the current bottlenecks in GPU architectures. It provides an in-depth analysis of the limitations across the algorithm, dataflow, and architecture layers, and proposes corresponding optimization strategies to promote 3DGS toward a more general-purpose and efficient rendering technology.