OMOTE / Real-time Face Tracking & Projection Mapping

Omote is the Japanese word for face, or mask: the outward surface (omote) set against what lies beneath (ura). In Noh theater, a single static mask can carry sorrow, rage, and tenderness depending on the tilt of the head and the angle of the light. The project drew on that idea, asking what a face becomes when it can be rewritten in real time.

Initially conceived as a commercial pitch for a cosmetics brand that never moved forward, the idea was strong enough that we pursued it as a personal project instead. Three months of testing were enough to confirm what was possible, and that it deserved more time to be done properly.

I built the system from the ground up: tracking facial markers in real time through motion capture sensors, reading the position and orientation of the face from that data, rendering a CG face with an animated texture, and projecting the result back onto a living face with enough precision and little latency that the projection stopped looking projected. No comparable system existed at the time, so both the calibration approach and the tracking algorithm had to be built from scratch.

When the video was released, it reached seven million views within a few days. Also, a major aspect of that project was the time spent working alongside Hiroto Kuwahara, a master makeup artist whose instincts shaped so much of the project's visual language. That collaboration didn't end with Omote, and led to several more projects together.