LAVA Lab was featured as the “KAISTIAN LAB” in the 2024 Autumn & Winter issue of KAIST’s official newsletter, kaistian, along with an interview with Professor Sung-Hee Lee([PDF]). Below, you will find the full Korean text followed by its English translation.
The Future of the Metaverse Pioneered by the Graduate School of Culture Technology’s LAVA Lab
Metaverse technology, once plagued by overblown expectations and the ensuing disappointment, has continued to advance. KAIST has established the Graduate School of Metaverse to open new frontiers in metaverse research. We spoke with Professor Sung-Hee Lee, Chair of the KAIST Graduate School of Culture Technology, who studies the “avatar”—the core of the metaverse—to learn about recent research directions.
Over the past few years, as the COVID-19 pandemic made remote activities part of everyday life, the metaverse attracted intense attention. Much like the 2018 Steven Spielberg film Ready Player One, it sparked hopes that, by wearing VR equipment, one could enter an entirely different world and enjoy a new life there. Spurred by these possibilities, countless researchers and companies jumped into the metaverse market, and Facebook even changed its name to “Meta.” However, limited technology and content at the time failed to meet people’s soaring expectations. As COVID-19 transitioned into an endemic phase and daily life returned to normal, public interest in the metaverse started to wane.
Still, the metaverse remains a strong candidate for the next-generation platform following the mobile device era, attracting steady research and investment. Rapid advances in core technologies—such as displays, computer graphics/vision, artificial intelligence (AI), and human-computer interaction (HCI)—have led to an optimistic outlook on the metaverse’s future. According to Professor Lee, “The gap between expectations and reality was large because the range of possible activities within metaverse platforms was still limited, preventing them from offering significant value. Researchers today think that the metaverse, rather than being a purely virtual world detached from reality, can create far more value when integrated into real life. That’s why current research focuses on augmented reality–based metaverse technology that merges actual physical spaces with virtual reality.”
Research on Avatar Modeling and Motion Generation
Within this metaverse, one of the most crucial elements is the “avatar”—the entity that acts on one’s behalf in the metaverse. To ensure the avatar can represent the user’s identity, customization and motion are key focal points of avatar research.
At Professor Lee’s “LAVA Lab” (Lab for Avatar and Virtual Agents), the team studies avatar modeling and motion generation. “We’re exploring ways to easily create realistic avatars,” says Lee. “In particular, we are concentrating on clothing modeling.” The goal is to capture or 3D-scan a user’s real clothing and convert it into a flexible, modifiable digital garment. In this context, “modifiable” means that the clothing should naturally respond to changes in the avatar’s posture and movement—even down to details like wrinkles. To achieve this, Professor Lee’s team is using deep-learning models to study every pattern and fabric property, enabling the AI to predict the sewing patterns and behavior of garments.
Motion generation for avatars is a specialty of Professor Lee, who earned a Ph.D. in computer animation. “Whether we’re dealing with humanoid robots, game characters, or metaverse avatars, the underlying research principles are closely related,” he explains. “We’re working on intelligent motion-generation technology that allows an avatar to interpret user intentions and respond with suitable movements.” By training AI on vast amounts of human motion data, the team aims to enable automatic generation of motions tailored to various environments and situations.
Selected for the Ministry of Science and ICT’s “Metaverse Lab” and the STEAM Program
Building on this foundation, Professor Lee’s research team is also conducting studies on modeling real users’ spaces and merging multiple spaces into a single environment. This work is part of the Ministry of Science and ICT (MSIT) “Metaverse Lab” support program, for which they were selected in May. The Metaverse Lab program assists in developing metaverse-specific services and content while also fostering interdisciplinary experts by linking startup and commercialization processes. Along with Korea University, Sogang University, and Hongik University, Professor Lee’s lab at KAIST was chosen to participate.
Professor Lee explains, “3D scanning technology has progressed significantly in recent years, making it easy to create a 3D digital twin by scanning a physical space. We’re exploring a concept where people who are physically apart each scan their individual spaces into 3D digital twins, then merge them into a single environment to achieve telepresence.” By creating these merged spaces, the team is developing systems that enable people to meet and engage in close interaction and communication via avatar motion-generation technology. Professor Lee adds, “We plan to release our work on a platform like Steam at a later stage so we can gather user feedback after we’ve developed the technology.”
In July, the team was also selected for the STEAM program—a fusion research support initiative by the MSIT. Their project is titled “AR-AI-Cognition-Architecture Convergence Research for Transcending Time and Space.” According to Professor Lee, “This concept expands on what we’re doing with the Metaverse Lab program. We aim to develop technologies that enable interaction across not just physical distances but also time. It’s a multidisciplinary project conducted in collaboration with Professors Woo Un-Taek, Cha Seung-Hyun, and Lee Jung-Mi at KAIST, and Professor Yeo Jin-Young at Yonsei University.”
The first step in this research is creating an AI agent capable of interacting across both time and space. For instance, a student in Korea would be able to experience a lecture by a professor at MIT in the United States with the same level of presence and immersion. To accomplish this, the AI learns the content, motion, voice, and video data of the professor as they lecture. Based on that information, the AI agent creates course materials and simulates the professor’s movements when it delivers the lecture. Professor Lee notes, “We’re integrating large language models (LLMs), motion generation, augmented reality, architecture, and cognitive science to design spaces that can reflect users’ emotions. We’re also studying how the fusion of virtual and real-world elements affects people cognitively.”
Ultimately, Professor Lee hopes these projects will enable “everyday telepresence.” He says, “I began this research after seeing how many people—like so-called ‘goose fathers’—live apart from their families. My hope is that, through the metaverse, people who are physically separated can share a single space without feeling lonely and can truly experience an emotional connection and sense of togetherness.”
The Imminent Future of the Metaverse
According to the global market research firm Research and Markets, the worldwide metaverse market is projected to grow at an average annual rate of 43.9%, reaching USD 936.6 billion by 2030 (approximately KRW 1,300 trillion). With not only Apple and Meta but also Samsung Electronics recently joining the development of metaverse-related devices, metaverse research is picking up speed. Professor Lee forecasts, “After smartphones, there are two possible futures: implanting chips in the body, like Neuralink, or using wearable devices. Given the current landscape, wearable devices will likely advance first, so the metaverse environment will ultimately become integrated into our daily lives.” In other words, the metaverse is not merely a fleeting trend—it is poised to become an essential infrastructure for the society of the future.
Korea, alongside the United States and China, maintains a globally competitive edge in metaverse research. In particular, KAIST boasts one of the world’s premier teams of metaverse researchers and has played a central role in fostering a metaverse research ecosystem since establishing its Graduate School of Metaverse in 2022. “The Graduate School of Metaverse has brought together a diverse group of faculty, creating an environment where synergy can flourish,” Professor Lee notes. “If we continue to pursue long-term, interdisciplinary research, I believe we can achieve even greater results.”
Still, Professor Lee emphasizes that Korea will need sustained student interest and ongoing government support to stay at the forefront of metaverse research. “Students at other universities seem to know about and be quite interested in the Graduate School of Culture Technology and the Graduate School of Metaverse, but KAIST students seem less aware,” he observes. “We’ll be working hard to raise the profile of these programs within our department, but we also hope that students will take an interest in interdisciplinary graduate studies, where they can collaborate with people from various backgrounds and further develop their own areas of expertise.” Finally, he concludes, “The greatest advantage of an augmented reality metaverse is that it’s interactive media. In a metaverse environment, a wide range of interactive content—spanning cultural exchange, education, and commerce—will become possible.”