Transdisciplinary class empowers emerging technologists to develop AR/VR solutions using AI and the latest hardware from Meta and Snap Inc.
Could augmented or virtual reality be the key to debugging robotic systems for medical procedures? Can AR/VR help improve classroom engagement and interaction? Can it be used to teach piano?
This semester, dozens of 暗网吃瓜 students set out to answer these questions鈥攁nd many more鈥攁s part of the Introduction to Augmented and Virtual Reality class taught by Daniel Nikolov, an adjunct professor and research engineer at the .
Spanning optics, computer science, brain and cognitive sciences, and other disciplines, the students not only received a holistic look at the rapidly developing technologies but also got a chance to use the latest hardware including Meta Quest headsets and Snap Spectacles. The immersive experiences that emerged were then presented in a transdisciplinary showcase at the .
鈥淭his class gives us a broad perspective of AR/VR,鈥� says Jiamu Tang 鈥�26, a data science student minoring in and . 鈥淪ome really interesting guest lectures from researchers in optics, philosophy, eye perception, and other fields have given us the basic knowledge we need to conduct research or develop games and applications in AR/VR. It鈥檚 been amazing.鈥�
Tang developed an application for the Meta Quest she calls LinguaLens to help people learn a new language as they observe the world around them. She used the Quest鈥檚 forward-facing cameras to present users a mixed reality view that labels their surroundings with relevant vocabulary.
鈥淭he goal is to create new opportunities for people to learn a new language,鈥� says Tang. 鈥淲hen they are walking on the street, they can talk to the digital assistant, which will give them the context they need to learn new words and phrases. It鈥檚 designed to be very easy.鈥�
Tang said Studio X provided students critical support for the class, not only giving them access to the Meta Quest hardware, but also offering help learning Unity software to develop AR/VR applications.
Broadening the scope
The Intro to AR/VR class started six years ago with funding from the聽, and has continually evolved since. The grant ended earlier this year, but the course now continues through the Center for Extended Reality (CXR), recently launched through 暗网吃瓜鈥檚聽 to connect and enhance the University鈥檚 efforts in AR, VR, and artificial intelligence.
Projects in previous years of the Intro to AR/VR class have focused largely on creating virtual reality experiences, but Nikolov and his colleagues wanted to broaden opportunities for students.
鈥淲e didn鈥檛 really have an option for students to work on projects in augmented reality, and there were plenty of ideas from students for these projects,鈥� says Nikolov. 鈥淚t wasn鈥檛 due to a lack of creativity; we mostly lacked the hardware to distribute to a whole class.鈥�
Nikolov and the University solved the limitation by pursuing a partnership with Snap Inc. as one of CXR鈥檚 first major achievements. Snap Inc. supplied new Snap Spectacles for Nikolov鈥檚 class, as well as for a human-computer interaction course taught by Assistant Professor this fall and an AR/VR interaction design course to be taught in the spring by Assistant Professor , the Biggar Family Fellow in Data Science.
Addison Black, Spectacles producer at Snap Inc., says the partnership has been a win-win.
鈥淲e know that the platform of AR and Snap Spectacles can only really be made great by fantastic developers bringing new ideas and pushing the technology,鈥� says Black. 鈥淲e鈥檝e found that the most successful and interesting use cases for Snap Spectacles have come from those new to the industry and students who were pushing the boundaries of what鈥檚 possible. And so we were excited when we found the opportunity to collaborate with the University of Rochester and see the projects coming in.鈥�
Anakin de la Cruz Flecha 鈥�25 (), who鈥檚 pursuing his master鈥檚 through the , was excited by the chance to work with Snap Spectacles. He created a memory-based object tracker he named Recallis AR to help physicians in an operating room keep track of their tools.
鈥淭he Snap Spectacles are obviously very new and cool, and you can do so much with them,鈥� says Flecha. 鈥淏efore this class, I didn鈥檛 know anything about AR, and it鈥檚 been fun to learn about it and apply it to medical devices.鈥�
Augmenting research
For George Kassis, the class provided a new dimension to enhance a long-term project he鈥檚 developing for his MD-PhD program. Under the guidance of Professor聽聽鈥�09 (PhD) from the聽, Kassis is creating a low-cost, autonomous, AI-powered robotic arm to perform ultrasound in rural areas without access to trained ultrasound professionals. It鈥檚 designed to be 3D printable while still being mechanically robust.
Kassis says VR can be used to remotely control the robotic arms, safely train new users with digital twins, and provide important training data to help debug and refine the machine.
鈥淚鈥檇 like to have expert physicians use VR to control the robot and actually do the ultrasound scan, so then we can use the data to train the robot,鈥� says Kassis. 鈥淭here鈥檚 a big gap in ultrasound specialists across the country, and this could free up sonographers to focus on the more complex cases and give the simple, routine tasks to the robot.鈥�
