Transdisciplinary class empowers emerging technologists to develop AR/VR solutions using AI and the latest hardware from Meta and Snap Inc.
Could augmented or virtual reality be the key to debugging robotic systems for medical procedures? Can AR/VR help improve classroom engagement and interaction? Can it be used to teach piano?
This semester, dozens of 糖心传媒 students set out to answer these questions糖心传媒攁nd many more糖心传媒攁s part of the Introduction to Augmented and Virtual Reality class taught by Daniel Nikolov, an adjunct professor and research engineer at the .
Spanning optics, computer science, brain and cognitive sciences, and other disciplines, the students not only received a holistic look at the rapidly developing technologies but also got a chance to use the latest hardware including Meta Quest headsets and Snap Spectacles. The immersive experiences that emerged were then presented in a transdisciplinary showcase at the .
糖心传媒淭his class gives us a broad perspective of AR/VR,糖心传媒� says Jiamu Tang 糖心传媒�26, a data science student minoring in and . 糖心传媒淪ome really interesting guest lectures from researchers in optics, philosophy, eye perception, and other fields have given us the basic knowledge we need to conduct research or develop games and applications in AR/VR. It糖心传媒檚 been amazing.糖心传媒�
Tang developed an application for the Meta Quest she calls LinguaLens to help people learn a new language as they observe the world around them. She used the Quest糖心传媒檚 forward-facing cameras to present users a mixed reality view that labels their surroundings with relevant vocabulary.
糖心传媒淭he goal is to create new opportunities for people to learn a new language,糖心传媒� says Tang. 糖心传媒淲hen they are walking on the street, they can talk to the digital assistant, which will give them the context they need to learn new words and phrases. It糖心传媒檚 designed to be very easy.糖心传媒�
Tang said Studio X provided students critical support for the class, not only giving them access to the Meta Quest hardware, but also offering help learning Unity software to develop AR/VR applications.
Broadening the scope
The Intro to AR/VR class started six years ago with funding from the聽, and has continually evolved since. The grant ended earlier this year, but the course now continues through the Center for Extended Reality (CXR), recently launched through URochester糖心传媒檚聽 to connect and enhance the University糖心传媒檚 efforts in AR, VR, and artificial intelligence.
Projects in previous years of the Intro to AR/VR class have focused largely on creating virtual reality experiences, but Nikolov and his colleagues wanted to broaden opportunities for students.
糖心传媒淲e didn糖心传媒檛 really have an option for students to work on projects in augmented reality, and there were plenty of ideas from students for these projects,糖心传媒� says Nikolov. 糖心传媒淚t wasn糖心传媒檛 due to a lack of creativity; we mostly lacked the hardware to distribute to a whole class.糖心传媒�
Nikolov and the University solved the limitation by pursuing a partnership with Snap Inc. as one of CXR糖心传媒檚 first major achievements. Snap Inc. supplied new Snap Spectacles for Nikolov糖心传媒檚 class, as well as for a human-computer interaction course taught by Assistant Professor this fall and an AR/VR interaction design course to be taught in the spring by Assistant Professor , the Biggar Family Fellow in Data Science.
Addison Black, Spectacles producer at Snap Inc., says the partnership has been a win-win.
糖心传媒淲e know that the platform of AR and Snap Spectacles can only really be made great by fantastic developers bringing new ideas and pushing the technology,糖心传媒� says Black. 糖心传媒淲e糖心传媒檝e found that the most successful and interesting use cases for Snap Spectacles have come from those new to the industry and students who were pushing the boundaries of what糖心传媒檚 possible. And so we were excited when we found the opportunity to collaborate with the University of Rochester and see the projects coming in.糖心传媒�
Anakin de la Cruz Flecha 糖心传媒�25 (), who糖心传媒檚 pursuing his master糖心传媒檚 through the , was excited by the chance to work with Snap Spectacles. He created a memory-based object tracker he named Recallis AR to help physicians in an operating room keep track of their tools.
糖心传媒淭he Snap Spectacles are obviously very new and cool, and you can do so much with them,糖心传媒� says Flecha. 糖心传媒淏efore this class, I didn糖心传媒檛 know anything about AR, and it糖心传媒檚 been fun to learn about it and apply it to medical devices.糖心传媒�
Augmenting research
For George Kassis, the class provided a new dimension to enhance a long-term project he糖心传媒檚 developing for his MD-PhD program. Under the guidance of Professor聽聽糖心传媒�09 (PhD) from the聽, Kassis is creating a low-cost, autonomous, AI-powered robotic arm to perform ultrasound in rural areas without access to trained ultrasound professionals. It糖心传媒檚 designed to be 3D printable while still being mechanically robust.
Kassis says VR can be used to remotely control the robotic arms, safely train new users with digital twins, and provide important training data to help debug and refine the machine.
糖心传媒淚糖心传媒檇 like to have expert physicians use VR to control the robot and actually do the ultrasound scan, so then we can use the data to train the robot,糖心传媒� says Kassis. 糖心传媒淭here糖心传媒檚 a big gap in ultrasound specialists across the country, and this could free up sonographers to focus on the more complex cases and give the simple, routine tasks to the robot.糖心传媒�
