Photo credit Dr. Lorans Alabood and Nada Albayati

ETHEREAL research group develops augmented reality programs for non-speaking autistic individuals

By Nazeefa Ahmed, January 19 2025—

The University of Calgary’s ETHEREAL research group is developing extended reality (XR) solutions for non speaking autistic individuals through their user-centric software products. With nine research articles in the past three years and various programs in development, the team aims to fill a research and technological gap for an underserved community. 

“What I noticed was that there was a vacuum in building assistive technology which can help them communicate better,” said Dr. Diwakar Krishnamurthy, ETHEREAL researcher and professor in the Department of Electrical and Software Engineering. “If you can somehow help them communicate better, their quality of life is going to increase exponentially.” 

An estimated 25-30 per cent of children with Autism Spectrum Disorder (ASD) are minimally verbal — communicating with fewer than 30 words — or completely non-verbal. Research on these communities is limited, prompting the formation of ETHEREAL in 2021.  

“We wanted to build practical, evidence-based assistive technology that can help these folks while being centred around a collaboration model,” said Krishnamurthy, adding that misconceptions about the community create a lack of user-centric design.

Photo credit Dr. Lorans Alabood and Nada Albayati

Collaboration, to Krishnamurthy, means working with the individuals, their families and their educators for input before making changes. Over the past three years, ETHEREAL has conducted 80 hours of testing through user sessions, in addition to time brainstorming and consulting with clients. Because the technology is mobile, the team travelled to the client’s homes, clinics and therapy centres to get more feedback and data on their systems.

“There’s a lot of work on using AR [and] VR for autistic folks but mostly it’s for speaking autistic people who are easier to work with,” said Krishnamurthy. “But if you’re doing research with a non-speaking population, it’s kind of difficult to get feedback, and there’s no precedent.”

 Personalization and user-centric design

Given the limited research in this space, ETHEREAL researchers did not have a blueprint to follow and relied on consultation, observation, and community outreach to understand how to build their product. Parents, educators, and autistic individuals shared common challenges, including difficulty sitting still, trouble focusing, and the inability to use a standard keyboard for communication.

Krishnamurthy and his team explored AR headsets as a possible solution. Most models have head tracking, which is when the virtual display moves with the user, so that an autistic user no longer has to sit still while completing a lesson. Additionally, an AI agent could be programmed to state certain cues to help with focus. This would provide users with more independence in their learning.

“One thing became clear was that augmented reality, and headsets like the HoloLens, which was just showing up in the market right then, could actually be very helpful,” said Krishnamurthy, adding that customizability to the user was an important feature for non-speaking autistic users.

Photo credit Becca McInnes

ETHEREAL researchers developed the HoloBoard, a virtual keyboard to be used with Microsoft’s AR device HoloLens 2. The HoloBoard lessors aimed to reduce visual clutter and support educator-led lessons specific for the age and development of the user. The transparent visor of the HoloLens also allows the autistic user to interact with the world while typing or communicating in a virtual space catered to their needs.

“A 31-year-old non-speaking man might present very severely. They might be shaking their head, flapping their hands [and] making vocalizations. So to an outsider, it may seem that they’re not there, and there’s a tendency to infantilize it.”

The personalization allows for the program to adapt to the user’s needs upon consulting with parents and educators instead of lowering the expectations based on perceived incompetence. This also gives the user greater independence and agency over their learning.

“We have seen that some people, not all, even with half an hour of training, can do more independent communication and more independent work.”

Independence may be possible outside of the classroom as well. The team recently built a prototype for the HoloLens, which allows users to order coffee from a coffee shop, translating the visually cluttered menu into simple choices such as iced or hot. Then, using a large language model (LLM) similar to ChatGPT, it constructs a response for the barista.

“We are actually building this device to be not just used as a system to acquire typing skills, but actually use the system to communicate,” said Krishnamurthy. “Once they become good at their typing skills, it itself can evolve into a communication device.”

Users often struggle with the lack of haptic feedback in a virtual space, which refers to tactile sensations or physical feedback that a user feels in response to interactions, typically through touch. To help with the adjustment, the team recorded sounds of clicking to act as cues for the next letter.

The lack of verbal feedback can make it difficult for parents and educators to see progress. AR lessons, however, collect data such as eye-tracking, hand movements, and can be put into an algorithm to measure success.

“When you kind of understand that the end result looks very simple, but there is a lot of back and forth involved in getting to that simplicity,” said Krishnamurthy, adding that the technology aims to be the bridge between educators, students, and parents.

One user expressed a desire to be more independent from their parents during therapy, as the HoloBoard system would allow the student to communicate without having a third party be the mediator.

Challenges, scalability and the future

Photo credit Dr. Lorans Alabood and Nada Albayati

The Microsoft HoloLens is priced at $5,000, whereas other AR options, such as Meta’s Quest 3, cost around $700. Krishnamurthy explains that, with more iterations and advancements over time, the price of the HoloLens is expected to decrease, following a trend similar to Moore’s Law. 

Another concern is the size of the headset, as it is bulkier than some other models.

“It’s not very discreet. So the hope is that the technology will follow the same trend [where] these devices would become less intrusive and as compact a form factor as a prescription glass,” said Krishnamurthy. “So if you get to that level, then from a comfort perspective, it becomes more scalable [and] a lot more people would buy it.”

Another option is to custom-build for the client, with the basic pieces needed for functionality. 

“A little bit of eye tracking and ability to point at virtual keyboards itself would give you a lot of the functionality,” Krishnamurthy.  “I think the biggest challenge even now which we face with is that there is no one size fits all.”

Student involvement

In addition to their research on assistive technologies, ETHEREAL actively involves students at all levels in their work, allowing them to contribute to real-world solutions. The ETHEREAL research group has hosted students researchers from all levels of education. Previous students have built custom lessons using Natural Language Processing (NLP) models, smart virtual keyboards that adapt to the user and sensory interventions to increase comfort. 

Photo credit Dr. Lorans Alabood and Nada Albayati

Therapy for non speaking autistic individuals can be expensive, limited and inaccessible in parts of Canada. A masters student in the ETHEREAL lab worked to increase accessibility by building virtual avatars for educators to use. 

“We welcome students to contact us and we look for enthusiastic students from all levels,” said Krishnamurthy. “We can look at some scholarship programs and other things to support their work.”

More information about Krishnamurthy and his research interests can be found on the University of Calgary website


Hiring | Staff | Advertising | Contact | PDF version | Archive | Volunteer | SU

The Gauntlet