Papers on AR-Mediated Human-Robot Collaboration Coming Out in RO-MAN and MobileHCI

Two AR-mediated human-robot collaboration papers we have been working on for quite some time are coming out later this summer. 

fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation is about human-robot collaborations in large, natural environments. Using a Meta Quest 3 and a Unitree Go2 robot dog, we demonstrate a system that combines a semantically aware world-in-miniature AR representation of the environment with vision-language-model-driven content adaptation. Compared with a baseline approach, the proposed method enables faster task completion and reduces user workload across several measures.

ARTOO-DARTU: Studying AR-HRC With AR Obstruction Mitigation During a Warehouse Task is about mitigation of AR content obstruction of important elements of the real world. In this work, for which we build an AR-mediated HRC system with a Microsoft HoloLens 2 and a TurtleBot 4, we demonstrate a scenario where AR assistance is appreciated only when AR-induced obstruction of the real world is detected and mitigated. We have been developing methods for detecting AR-induced obstruction of the real world in a parallel line of work (see VIDDAR, TVCG 2025) — it is very nice to also demonstrate that obstruction detection and mitigation can have a positive impact on user experience, in the important case of AR-mediated HRC.

Both studies are led by Christian Fronk, who is very happy to get a chance to present the outcomes of his work in Wales, UK and Kitakyushu, Japan, all in a course of two last weeks of August. Way to go Christian!

Posted in Augmented reality, Publications and Demonstrations, Robotics, Safety | Leave a comment

New Funding: 4-year NSF CISE FutureCore Award on Automated AR Guidance Monitoring

We are grateful to the NSF for supporting our work on automated AR guidance monitoring, in pervasive AR and human-robot collaboration (HRC) applications. This award will support our joint work with Miroslav Pajic’s group at Duke, which is very exciting to me personally as this synergetic collaboration will help ensure impact on multiple research communities.

AR guidance can be hard to read, appear in the wrong place, provide incorrect instructions, hide important real-world information, or encourage actions that are unsafe, intrusive, or poorly suited to the surrounding environment. A user may not always notice these problems before acting on the guidance. This project will develop new ways to automatically check whether augmented reality guidance is clear, correct, safe, and appropriate in the physical setting where it appears.

Posted in Augmented reality, Edge computing, Funding and Awards, Highlights, Mobile computing, Research, Robotics, Safety, Security | Leave a comment

RIP Magic Leap Headsets

It is official: Magic Leap is no longer making its own headsets. Our thoughts are with everyone affected by the layoffs. May you land on your feet; any team will be lucky to have you.

Magic Leap has a very special place in the XR research ecosystem: this headset became a true workhorse for XR research labs using eye tracking to study attention and spatial interaction. Its compact form factor made it uniquely practical for experiments involving mobile cognition and substantial head-and-body movement, such as our lab’s work with AR-assisted CPRContinue reading

Posted in Augmented reality, Perspectives, Technology | Leave a comment

Keynote and Technology Demonstrations at IEEE VisArch Co-located with ISCA

Duke I3T Lab at IEEE VisArch Workshop co-located with IEEE ISCA

It was our pleasure to participate in IEEE VisArch Workshop co-located with IEEE ISCA, which came to Raleigh, NC in June 2026.

Professor Gorlatova delivered a keynote speech titled “Intelligent and Secure XR: Building Adaptive Spatial Systems That Know When They Are Wrong;” PhD students Yanming Xiu and Christian Fronk brought their live technology demonstrations previously showcased as part of DARPA Intrinsic Cognitive Security Program PI meetings.

Posted in Augmented reality, Edge computing, Events, Publications and Demonstrations, Research, Students and Mentorship, Talks, Technology | Leave a comment

Recent I3T Lab Graduate Sarah Eom Highlighted by the Duke University Center for Computational and Digital Health Innovation

Sarah Eom, PhD, was highlighted by the Duke University Center for Computational and Digital Health Innovation for her work at the intersection of next-generation immersive systems and healthcare. Congratulations Dr. Eom!

Posted in Highlights, Research, Students and Mentorship | Leave a comment

Is This the Future of our Day to Day Lives with Smart Glasses and AI?

If you do not believe that the future is smartglasses + AI, the VisionClaw paper might change your mind.

The authors created a system that “running on Meta RayBan smart glasses, continuously perceives real-world context and enables in-situ, speech-driven action delegation via Gemini Live and OpenClaw AI agents.” Paper’s description of a (small; N=4) deployment study is fascinating in its demonstration of how agentic interaction changes everyday behavior.

I’ve always believed in augmented reality – obviously! – but this is just a whole other level of opportunities for AR and AI.

Full text of the paper on ArXiv 

Posted in Perspectives | Leave a comment

I3T Lab Tour a Part of Regional NAE Meeting

It was our sincere pleasure to offer a Duke I3T Lab Tour as part of the National Academy of Engineers Regional Meeting held at Duke University in May 2026.

 

May 2026 NAE Regional Meeting attendees exploring live AR and VR demonstrations as part of the Duke University Pratt School of Engineering lab tour

Just like the Frontiers of Engineering (FOE) NAE Symposium I attended back in 2021, the Regional Meeting was a real treat: it outlined both the promise and the challenges in the exciting and important space of commercial aviation, from multiple technical and non-technical perspectives. Like the FOE Symposium, it reminded me how exciting it is to be an engineer: to have not only an in-depth understanding of the problems and the underlying processes, but also know how to solve them. Thankful for the organizers for the opportunity to participate in this event and to contribute to it. [ Short video with several snippets from the lab tour ]

 

Posted in Augmented reality, Demonstrations, Duke University, Events, Highlights, Lab tours, Mobile computing, Students and Mentorship | Leave a comment

I am a Bass Chair!

I am one of only nine faculty, across all of Duke, to be named the 2026 Bass Chair the honor that recognizes faculty whose teaching and research exemplify Duke’s commitment to undergraduate education.

Continue reading

Posted in Achievement, Career and Leadership, Funding and Awards, Highlights | Leave a comment

One More Half-marathon: Garmin Durham 1/2

My third half marathon in the last two years – May 2026 Garmin Durham 1/2 marathon – done!

Lovely race; running through the Durham Bulls stadium is a very special experience, and so is finishing on Duke’s football stadium.

Problem with finishing on this stadium: then you have to climb all of the stadium’s stairs, on shaky, wooden legs. But the unique experience is worth it. It reminded me of some of the truly unique races I have done before, like a race that goes through the Lincoln Tunnel under the Hudson River and a race on the airstrip of the Teterboro Airport. Huge thanks to the organizers for bringing these one-of-a-kind events to life.

Posted in Outside of the Lab, Races, Running, Sports | Leave a comment

Center for Human-Computer Interaction at Virginia Tech (CHCI@VT) Headliner Seminar

Possible at Last: Using Large Multimodal Models to Evaluate and Secure Mixed-Reality Experiences – this was the title of the talk I gave as part of the VT Center for Human-Computer Interaction Seminar. Sincere thanks to the organizers for the opportunity to share my perspective: we are finally, finally at the cusp of being able to provide real-time oversight of AR experiences.

Continue reading

Posted in Perspectives, Talks | Leave a comment