Augmented reality to provide new skills for manufacturing workforce education

WEST LAFAYETTE, Ind. — Workforce education in manufacturing has traditionally been very low-tech: in-person teaching sessions, one-on-one apprenticeship, written manuals, and perhaps a video. But to keep pace with an increasing skills gap, some manufacturers are now turning to augmented and virtual reality to teach their workers new skills. A Purdue team has entered into a $5 million cooperative agreement with the National Science Foundation (NSF) to create an augmented and virtual reality experience prototype called Skill-XR.

“The skills gap in hands-on trades and emerging technology is real,” said Karthik Ramani, the Donald W. Feddersen Distinguished Professor in Mechanical Engineering and principal investigator for the project. “Today, one of the best ways to transfer hands-on skills is through the traditional one-on-one apprenticeship model. However, it is costly and not scalable. You lose both the trainer and trainee. Also, older workers are retiring, and it’s difficult to recruit younger workers to replace them. And when they are hired, it is challenging to transfer those skills and knowledge to the next generation. The skills themselves are also changing, complicating the landscape. Skill-XR is all about boosting that skill transfer in a timely manner and enabling it to happen anywhere at any scale.”

The “X” in “Skill-XR” stands for a range of technologies that includes augmented reality (AR), virtual reality (VR), and extended reality (XR).  For example, a newly hired factory worker may wear augmented reality glasses while being trained on a piece of equipment and see graphics overlaid on the machine about how to operate the controls.  The instant feedback upon performing the task correctly ensures that workers are trained quickly, effectively, and safely.

“In this generation, YouTube videos have become stand-in educators,” Ramani said.  “People turn to YouTube to figure out how to tie a tie, change a tire or cook a meal.  We want to push that concept into the next dimension so that it’s no longer just a 2D video on a screen, but it’s an augmented reality experience that actually responds to you and gives you feedback.”

Anyone can produce a YouTube video with today’s technology, but the same is not true of augmented reality. Most AR experiences involve complex programming and other specialized skills that companies don’t have and cannot afford to purchase. Skill-XR will level that playing field, enabling anyone to generate an augmented reality experience, whether it’s in the factory, the classroom or their own home. It’s also designed to be platform-agnostic, able to be implemented in whatever new devices or mixed-reality technologies develop in the marketplace.

Scalability is also a big aspect to the project. 

“It’s one thing to have a person with skills or knowledge, who shares that knowledge with a handful of others in a room,” Ramani said. “Technology will allow that same person in the same room to teach those same hands-on skills to thousands or even millions. As we’ve seen from the COVID-19 pandemic, remote learning is now a vital piece of our everyday lives, and Skill-XR will enable remote hands-on skilling to happen anywhere.”

While high levels of technology are involved, Ramani has built it upon the most foundational of human conditions: the desire to learn. 

“As babies, we learn by doing,” he said.  “We all have eyes, ears, and hands. We pick objects up and move things around. We copy what we see others doing. There’s no technology involved in these processes; we just naturally do it.  With Skill-XR, we’re enhancing that process so that people can learn by doing and not just watching. That’s why learning scientists are such a big part of our team.”

The Skill-XR team includes co-principal investigators Alex Quinn, assistant professor of electrical and computer engineering at Purdue; Thomas Reddick, associate professor of cognitive psychology at Purdue; Niklas Elmqvist, professor of computer science at the University of Maryland; and Kylie Peppler, associate professor of informatics at University of California, Irvine.

“As skilled older manufacturing workers retire, their hands-on skills must be passed on to a new generation,” said Mark Lundstrom, Purdue’s acting dean of engineering. “Professor Ramani’s team is addressing this critical challenge by making it possible for the first time for skilled workers to easily create augmented reality experiences that will allow younger workers to learn by doing rather than by watching. The impact on the nation’s manufacturing workforce should be substantial.”

Brian Edelman, president of the Purdue Research Foundation, said, “All the companies we work with in the Discovery Park District and the Aerospace District are keenly focused on their workforce, and the ability to get their workforce fully productive as quickly and economically as possible. I believe the Skill-XR technology is a game changer, both during the current pandemic and post-pandemic, as we all are learning how capable we are of working remotely.”

The building blocks for Skill-XR began in 2018, when Ramani’s team received a $2.5 million NSF grant to explore the “Future of Work at the Human-Technology Frontier.”  One of the ideas that emerged was Skill-LeARn, a concept for using augmented reality (AR) to educate workers.  This received Phase I funding from the NSF’s Convergence Accelerator, a new NSF capability designed to address national-scale societal challenges through use-inspired convergence research. Ramani's team conducted several research projects under this umbrella, such as developing an AR “ghost” that imitates the actions of humans, and integrating AR content into student classrooms. The Purdue Research Foundation’s Office of Technology Commercialization applied for several patents for technologies emerging from the project. Now, Ramani’s team has received Phase II NSF Convergence Accelerator funding of $5 million over two years, to implement the project (now renamed Skill-XR) in the real world.

For a true convergence approach, Ramani has partnered with several Indiana manufacturing-focused companies, such as Wabash National, Kirby Risk, and Gaylor Electric, along with other large and small companies across the country.  The partners have been testing prototypes and determining how they can be best used on a real-world factory floor.  He’s also partnered with the Indiana Economic Development Corporation and Ivy Tech, the community college system of Indiana, whose educational programs are often focused on specific job fields. 

“Indiana is the perfect test bed for this platform,” Ramani said. “The economy of rural areas depends heavily on manufacturing, so when manufacturing succeeds, so does Indiana.”

David Roberts, chief innovation officer for the Indiana Economic Development Corporation, said, “Every day, Indiana manufacturers across the state are delivering critical solutions and propelling innovative products that help power our world. As disruptive trends change the face of today’s manufacturing industry, we’re focused on supporting strategic initiatives like the Skill-XR technology to develop a diverse and skilled workforce in Indiana, equipping Hoosiers with the industry-focused skills and training needed for the future.”

Ramani sees potential for Skill-XR beyond the factory floor. 

“The project also involves a variety of educational environments, such as the Strong National Museum of Play in Rochester, New York, and Indiana's own Rural Innovation Network,” Ramani said.  “Young children can learn in a hands-on manner, even while they play games. College students can learn engineering concepts. Even doctors and nurses can practice surgical techniques. Our overarching goal is to extend the capabilities of all people, using technology.”

About Purdue University

Purdue University is a top public research institution developing practical solutions to today’s toughest challenges. Ranked the No. 5 Most Innovative University in the United States by U.S. News & World Report, Purdue delivers world-changing research and out-of-this-world discovery. Committed to hands-on and online, real-world learning, Purdue offers a transformative education to all. Committed to affordability and accessibility, Purdue has frozen tuition and most fees at 2012-13 levels, enabling more students than ever to graduate debt-free. See how Purdue never stops in the persistent pursuit of the next giant leap at https://purdue.edu/.

Writer: Jared Pike, jaredpike@purdue.edu, 765-496-0374

Source: Karthik Ramani, ramani@purdue.edu, 765-494-5725 

Journalists visiting campus: Journalists should follow Protect Purdue protocols and the following guidelines:

• Campus is open, but the number of people in spaces may be limited. We will be as accommodating as possible, but you may be asked to step out or report from another location.
• To enable access, particularly to campus buildings, we recommend you contact the Purdue News Service media contact listed on the release to let them know the nature of the visit and where you will be visiting. A News Service representative can facilitate safe access and may escort you on campus.
• Correctly wear face masks inside any campus building, and correctly wear face masks outdoors when social distancing of at least six feet is not possible.