AI and XR: How Intelligent Systems Enhance Extended Reality
Extended Reality (XR) is the collective term for immersive technologies that bridge the physical and digital worlds, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). On their own, XR technologies create rich visual and spatial environments.
When combined with Artificial Intelligence (AI), these virtual environments gain the ability to listen, adapt, and respond dynamically to user actions. The fusion of spatial presence and machine intelligence creates flexible simulations that respond in real time rather than following rigid, pre-recorded scripts.
How AI and XR combine in practice
Integrating AI into immersive environments enhances several core capabilities:
Traditional virtual characters rely on branching dialogue trees where users select from pre-written multiple-choice options. When powered by conversational AI, non-player characters (NPCs) can understand natural speech, interpret context, and respond in character on the fly.
Speech-to-text models convert a user's spoken words into text instantly, language models determine the appropriate response, and text-to-speech engines vocalise that reply with natural inflection and synchronised lip movement.
AI systems can monitor user performance, hesitation, or decisions within a simulation. The environment can adjust its difficulty, introduce new events, or offer targeted coaching cues based on how the participant is performing.
In Augmented and Mixed Reality headsets, machine learning models recognise physical surfaces, walls, hands, and tools, allowing digital objects to interact convincingly with real physical surroundings.
A character that answers back
The same pattern in four settings. Rendered illustrations, not footage from a client build.
PTR's own conversational AI capabilities, "AI companions" and "Conversational agents," are the domain-grounded, guardrailed systems that sit in the "Language model" step above; the same underlying twin can be met through chat, voice, or a live avatar. See PTR's digital twins for how that spans text, voice and avatar surfaces.
Why the combination matters for practice-based learning
The primary value of uniting AI and XR lies in practice-based learning. In general terms, this combination brings together two essential elements:
XR places the learner in a convincing context (such as a clinic, a workshop, or a negotiation room), generating a sense of presence and situational awareness that flat screens cannot match.
AI provides an unpredictable, conversational counterpart that reacts realistically to tone, phrasing, and decision-making.
By combining both elements, learners can practise interpersonal communication, procedural workflows, and high-pressure decision-making in a repeatable environment without putting real people, patient safety, or valuable equipment at risk.
Current limitations and considerations
Organisations planning to combine AI and XR should keep several technical and practical factors in mind:
Fluid conversation requires near-instantaneous responses. Delays between a user speaking and a virtual character replying can disrupt the sense of natural dialogue. A 2025 study found most users preferred a character to show a small natural filler action, such as a glance or gesture, while it composed its reply, rather than a generic loading indicator (Gonzales et al, 2025).
Running detailed 3D graphics alongside real-time voice processing requires reliable computing hardware, low-latency network connections, or capable standalone headsets.
AI-driven characters must stay within defined scenario parameters and professional boundaries. Clear instructional design and technical constraints are needed to prevent characters from going off-topic.
Voice recordings and transcripts captured during immersive practice sessions must be handled under strict data protection and privacy standards.
Common misconception: it is not just "a chatbot inside VR"
Dropping a text chat window into a VR scene does not, on its own, create the effect people mean by "AI and XR." Research on embodied conversational agents finds that embodiment materially changes the outcome: participants who practised interpersonal skills with a VR-embodied conversational AI agent rated it more effective than the same AI reached through text alone (Cornell/Stanford, CSCW 2025). A separate comparison found an AI NPC tutor reached moderate to high co-presence but was still rated well below a human tutor on overall social presence (ACM SIGGRAPH MIG, 2025), a reminder that "AI plus XR" narrows the gap with a human demonstrator without yet closing it.
| Scripted virtual characters | AI-driven virtual characters | |
|---|---|---|
| Dialogue options | A fixed set of pre-written choices | Open-ended natural speech or text |
| Adapts to unexpected phrasing? | No, falls back to a default line | Usually, within its defined guardrails |
| Needs re-authoring for a new topic? | Yes, every branch is hand-written | Often just an update to the approved source material |
AI and XR combine when a language model decides what a virtual character says and does, and XR gives that character a spatial, embodied place to say and do it. Either piece alone falls short: XR without AI is a fixed script; AI without embodiment is a chat window.
Sources and further reading
- Young Merlin: an embodied conversational agent in virtual reality - ACM ICMI, 2016
- Social Presence in Virtual Reality: A Comparative Study of AI NPCs and Human Instructors - ACM SIGGRAPH MIG, 2025
- Comparing Text-Only and Virtual Reality-Embodied Conversational AI Agents for Interpersonal Skills Training - ACM CSCW Companion, 2025
- Behavioral and Symbolic Fillers as Delay Mitigation for Embodied Conversational Agents in Virtual Reality - IEEE Transactions on Visualization and Computer Graphics, 2025
PTR's digital twins · What Is AI? · AI for Training and Roleplay · XR overview