Tangible & Embodied Interaction Framework
Redefining how humans control digital spaces by using the body as the interface.
What happens when movement and an everyday device become part of the interface instead of relying only on conventional screen-and-pointer interaction?
A smartphone could act as an embodied controller, mapping familiar physical movement into control of a virtual environment.
For my M.Des thesis at IIT Hyderabad, I built a functional prototype using smartphone sensor input through ZigSim / OSC to control virtual camera perspective and environment logic inside Unreal Engine.
The project established the interaction direction that still runs through my work: physical movement can become meaningful digital input. That thread later continued through spatial experiments and NudgeQ.
Research & Theoretical Grounding
The project began with a deep dive into Tangible and Embodied Interaction (TEI), embodied cognition, ecological perception, constructivism, and core HCI principles. Existing digital workflows—especially in 3D tools and filmmaking—were analyzed to identify friction caused by abstract controls, steep learning curves, and over-reliance on software literacy rather than human intuition.
The key insight: humans already understand movement, space, weight, and orientation—interfaces should leverage that instead of fighting it.
Concept & Interaction Hypothesis
I hypothesized that mapping real-world physical movements directly to digital actions would increase intuition, agency, and immersion. Instead of keyframes, sliders, or timelines, users would move to move the camera.
The phone became a tangible controller—its orientation, position, acceleration, and touch inputs acting as embodied inputs for controlling virtual cameras, lighting, and motion inside 3D environments.
Prototyping & Technical Execution
To test the concept, I built a functional interaction framework using:
- ZigSim to stream real-time sensor data from a phone
- ARKit for motion tracking and spatial accuracy
- Unreal Engine with Blueprint scripting for interaction logic
- Blender for early camera and lighting experiments
Physical phone movements were mapped to virtual camera translation, rotation, lighting direction, and impulse-based motion—creating a direct cause–effect loop between body and screen.
Virtual Installation & Cinematic Output
The framework was validated through a short cinematic experiment titled “Echoes of the Past.”
Every shot in the film was captured by physically moving the phone in real space—no keyframing, no timelines.
This demonstrated how embodied control can be used not just for experimentation, but for real creative output like cinematics, previsualization, and interactive installations.
Evaluation, Limitations & Iteration
User testing revealed strong gains in intuitiveness, immersion, and speed—but also surfaced real constraints. Precision control, physical fatigue, and system stability emerged as key limitations.
Rather than hiding these, the project documents them clearly, framing them as design challenges for future iterations rather than failures.
Remove unnecessary abstraction.
This project reshaped how I think about interfaces. It reinforced that good interaction design isn’t about adding smarter controls—it’s about removing unnecessary abstraction.
The work sits at the intersection of design, technology, and human behavior, and continues to influence how I approach XR, tools, and systems that demand learning, creativity, and flow.