What is mixed reality?
Mixed Reality (MR) combines a live view of the physical environment with digital objects that can remain positioned in it and respond to it, rather than simply floating on top of a screen. An MR system builds a precise 3D understanding of the user's physical environment, allowing virtual objects to sit convincingly on real tables, hide behind real walls, and respond dynamically to physical obstacles and natural hand interactions.
How mixed reality works in practice
Mixed reality combines high-resolution environmental sensors, advanced computer vision, and real-time spatial computing to anchor interactive digital content into physical space.
Key hardware architectures and technical principles include:
- Video passthrough headsets: Enclosed headsets that capture the physical environment using high-definition external cameras and present that video feed onto internal screens in real time. Digital graphics are rendered directly into the camera stream with extremely low latency, creating rich colours and wide fields of view.
- Optical see-through headsets: Transparent glass or visor devices where users look directly at the physical world while digital light is projected onto see-through lenses (waveguides) into their eyes.
- Spatial mapping and meshing: Depth sensors scan the geometry of the room (such as floors, walls, furniture, and ceilings) to create an invisible 3D digital mesh. The system uses this mesh to calculate physics, ensuring a virtual ball bounces off a real desk or rolls across a real floor.
- Occlusion: A critical visual effect where physical objects block the view of virtual objects positioned behind them. For example, if a virtual character stands behind a real office chair, the chair naturally conceals the lower half of the character, creating authentic depth perception. Meta's own developer documentation describes this as a Depth API providing real-time, per-eye environment depth estimates, specifically to let developers implement this kind of dynamic occlusion. Meta developer blog, 2023
- Spatial persistence: Software anchors virtual content to fixed physical coordinates so that a digital dashboard placed on a specific boardroom wall remains in that exact spot when a user returns days later.
- Direct hand tracking and gesture control: Built-in cameras track fingers and joints in real time, allowing users to tap, push, rotate, and resize digital items naturally without holding physical controllers.
- Eye tracking and foveated rendering: Sensors monitor gaze direction to enable eye-based navigation and foveated rendering (a technique that renders the area where the eye is focused at maximum graphical sharpness while reducing detail in peripheral vision to optimise performance).
What it feels like to use MR
Using mixed reality feels like having responsive digital holograms integrated directly into everyday surroundings. Because the physical room remains visible (either through transparent lenses or real-time passthrough video), users maintain full spatial awareness and can easily talk and collaborate with colleagues in the same room.
Interacting with virtual objects feels intuitive because digital items respect the physical laws of the room. When reaching out to touch a virtual control panel floating beside physical machinery, the responsiveness of hand tracking creates an immediate sense of control without requiring complex controller bindings.
Where mixed reality is commonly used
Organisations deploy MR across complex operational, medical, and technical workflows:
- High-fidelity clinical training and medical simulation: Healthcare teams practise complex medical procedures on physical training mannequins while seeing holographic internal organs, blood vessels, and real-time patient vitals superimposed onto the mannequin.
- Engineering design and architectural prototyping: Engineering and architecture teams place 1:1 scale digital prototypes inside physical factory floors or construction sites to test ergonomics, spatial clearances, and maintenance access before building physical units.
- Complex assembly and quality assurance: Manufacturing operators follow interactive 3D holographic guidance that aligns directly with physical parts on the production line, verifying component alignment in real time.
- Multi-user spatial collaboration: Remote and co-located teams gather around a physical table to interact with a shared 3D holographic model, such as a facility layout, terrain map, or product design.
Mixed reality in practice at PTR
PTR builds mixed reality alongside its virtual and augmented reality work, describing itself as building "Virtual, augmented and mixed reality for realistic practice." MR follows the same practice-based approach PTR uses for VR, a safe environment, real pressure, the learner's own choice, and a debrief afterwards. What changes is where the practice happens: instead of taking the learner somewhere else, MR anchors the scenario in the room they are already in, so a holographic person can be met at a real bedside and a procedure rehearsed against real equipment.
Common confusions about MR
While both technologies keep the physical world visible, basic AR overlays static 2D or 3D graphics on a camera feed. MR creates interactive co-presence where physical and digital items interact through spatial meshing, collision physics, and occlusion.
Modern standalone headsets increasingly support both full VR and high-resolution MR passthrough within the same hardware device, allowing users to transition between completely virtual worlds and blended mixed reality environments.
Video passthrough allows dynamic digital compositing, but the view is rendered via camera sensors and digital screens, meaning lighting conditions and camera resolution can influence visual clarity compared to optical see-through glass.
A 2019 study presented at the ACM CHI conference interviewed ten AR/VR experts and reviewed 68 published papers, and found at least six different, partly competing notions of what "mixed reality" means, from a point on Milgram and Kishino's original continuum to a synonym for AR to a description of separate AR and VR users collaborating together. When a vendor or article uses the term, it is worth checking which of these senses it actually means.
Is mixed reality the right fit?
- Users must interact with digital models and physical objects simultaneously, such as a holographic scan beside a real desk.
- Collaborative in-person training is required, with participants seeing each other and shared virtual assets naturally.
- Situational awareness is critical, letting users navigate their real environment while interacting with spatial simulations.
- Hardware cost and complexity: MR headsets with depth sensors and passthrough cameras cost more than a phone-based AR app or a simple VR headset, and need more setup and calibration.
- Small or changeable spaces: Spatial mapping and persistence work best in a mapped, relatively stable physical space. Frequently rearranged rooms, or very small or cluttered ones, make occlusion and anchoring less reliable.
- Battery life and thermal limits: Passthrough cameras, depth sensors and rendering both a real and a virtual scene simultaneously are power-hungry, which limits continuous session length on standalone headsets.
- When plain AR is enough: If the task only needs an overlay, such as a label or an instruction floating near an object, without objects hiding behind real furniture or persisting across sessions, simpler and cheaper AR usually does the job.
To explore how mixed reality applications are built for enterprise and training, visit the XR overview page.
MR keeps the real world visible, like AR, but goes further by giving digital objects a working understanding of that room, so they can rest on real surfaces, hide behind real furniture, and stay in place between sessions. That deeper environmental integration, not just "AR with a nicer headset," is what actually separates it from standard AR.
Sources and further reading
- What is Mixed Reality? (Speicher, Hall and Nebeling) - ACM CHI Conference on Human Factors in Computing Systems, 2019
- A Taxonomy of Mixed Reality Visual Displays (Milgram and Kishino) - IEICE Transactions on Information and Systems, 1994
- Building for Mixed Reality on Meta Quest 3 - Meta Horizon OS Developers blog, 2023