Designing a VR + LLM communication coach @ DePaul

During my Master's in Human-Computer Interaction at DePaul, I became much more interested in a question that had followed me for years: what makes an intelligent system actually feel useful, supportive, and intuitive to a real user?

One of the projects that pushed that question furthest was a VR platform that used LLMs to help students improve communication skills through real-time interaction and feedback.

This was one of the first projects that made me think seriously about AI not just as a model problem, but as an experience problem.

TL;DR

Led a team of 2 students on a VR + LLM communication platform at DePaul
Worked with PUSH Studio and LeAnne Wagner on the project direction
Designed the system around real-time practice for students, especially non-native English speakers
Built an end-to-end interaction loop: speech → text → LLM → audio + avatar expression mapping
Delivered feedback in under 1.2 seconds
Improved participant outcomes by ~25%, based on observed communication performance, student self-reports, and professor feedback
Opened up follow-on interest for staff communication training and possible clinical applications

Context

By the time I got to DePaul, I had already spent years building products and shipping systems at scale.

What I had less of was a formal way to think about why some systems feel intuitive and others feel frustrating, even when the underlying technology works.

That shift started early in the HCI program. One phrase from Dr Peter Hastings stuck with me: I am not the user. The user is not me.

It sounds simple, but it changed how I looked at every product I had built before and everything I've built since.

Around the same time, one problem kept surfacing in our research conversations: after COVID, many students were struggling to communicate confidently in academic settings. That challenge was even sharper for non-native English speakers, who often knew what they wanted to say but had difficulty expressing it clearly in the moment.

The project started with student-professor communication in mind, but the core problem turned out to be broader: how to help people practice communication in situations that felt socially demanding, emotionally uncomfortable, or high stakes.

My role

I led a team of 2 students, combining user research, prototyping, and engineering.

Working with PUSH Studio and LeAnne Wagner, we spoke with students, mapped where communication was breaking down, and built the system around practice in context rather than static instruction. The goal was not to give students generic advice. It was to create a space where they could rehearse difficult conversations, get feedback quickly, and improve over time.

My role covered:

shaping the research direction
leading the prototyping effort
defining the interaction flow
helping build the real-time technical pipeline
translating user feedback into system changes

Why VR

The core question was not just whether AI could generate useful responses. It was whether the interaction could feel natural enough to support learning.

VR mattered because communication is embodied. Timing, presence, eye contact, social pressure, and emotional safety all shape how people speak and respond. A flat interface would have captured only part of the problem.

We believed a VR setting could create a more immersive and lower-stakes environment for practice, especially for students who felt anxious, underconfident, or hesitant to speak up.

The technical challenge

The hard part was not generating text. It was building a responsive interaction loop.

At the time, there was no clean off-the-shelf pipeline that combined:

speech capture
low-latency text generation
text-to-speech
facial expression mapping
embodied avatar feedback

So we built the full loop ourselves.

We got end-to-end latency down to under 1.2 seconds.

That mattered a lot. If the loop feels delayed, the experience stops feeling conversational and starts feeling like software. Once that happens, the learning effect weakens.

Results

The system led to a ~25% improvement in participant outcomes, measured through a mix of:

observed communication performance
self-reported student feedback
feedback from professors

The strongest signal came from non-native English speakers, for whom the system created a more forgiving environment to practice expression and confidence.

What started as a project focused on student-professor communication also opened up other directions, including:

communication training for staff
de-escalation-oriented use cases
early conversations around possible clinical applications

That was useful validation that the idea had value beyond the original setting.

Why this project mattered to me

This project changed how I think about intelligent systems.

It made one thing very clear: the model is only part of the experience. Timing, embodiment, trust, feedback quality, and emotional safety are equally important.

That insight shaped a lot of what came after. It is one of the clearest reasons I became interested in ambient AI, wearable interfaces, and systems that adapt to users instead of forcing users to adapt to them.

You can see that thread more directly in WearableAI.

What I'd do differently

I would make the architecture more modular.

The pipeline worked, but it was fairly tightly coupled. We optimized hard for latency, which helped us hit the 1.2-second target, but it also made the system harder to extend. Swapping models, adding languages, or changing individual stages required more rework than I would want in a longer-lived platform.

I would instrument learning outcomes more deeply.

We had positive results, but I would have liked more structured instrumentation around which interaction patterns produced the biggest gains, where users hesitated most, and how different feedback styles affected confidence and retention.

I would test more interaction variations earlier.

A lot of the product value lived in subtle choices: how the avatar responded, how feedback was delivered, and how emotionally safe the interaction felt. I would run a broader set of experiments on those experience variables sooner.

What I learned

HCI is not a cosmetic layer on top of intelligence. It is part of what makes intelligence usable.

This project was one of the clearest examples of that in my career. It taught me that if you want an AI system to help someone in a meaningful way, you have to design not just the output, but the conditions around it:

latency
embodiment
context
trust
emotional comfort
feedback quality

That lesson still shapes how I build products now.

WearableAI - applying similar ideas to ambient, voice-first systems on Ray-Ban Meta, iPhone, and CarPlay

Around the same period, I also worked on a more playful interactive installation that combined pose estimation and real-time avatar-style feedback. It was deployed at the Peggy Notebaert Nature Museum in Chicago, where visitors could stand in front of a camera and see butterflies appear on their bodies and move with them on screen.

It was a smaller project, but it reinforced something I still care about: technology can be technically sophisticated and still feel lightweight, intuitive, and delightful.