Augmenting Deformable Surfaces

Augmenting Deformable Surfaces

I envy those students that create a video demonstrating their research results and then upload it to Youtube. It's a great a way to show others what have you been working on for all those years. Julien Pilet of Keio University, Japan, is one of those students. He recently uploaded two videos showing off his PhD thesis (done at the EPFL), that explores registering and augmenting non-rigid objects using only one regular camera:



AS you can see in the video above, not only the virtual EPFL sticker bends according to the contours of the shirt, its illumination model is adjusted as well (shadows cast on the shirt are cast on the sticker), and it handles occlusions nicely , most of the times. The only requirements are that the augmented object has a non-monotone texture, that it is locally planar and that it lacks holes.

In the next video, the illumination model is more obvious and a bit trickier to handle, since it now interpolates the shadow cast on a 3d model, from the 2d information in the image:



Pilet's work has been used in several projects. One of them you might recall if you were following AR for the last year. It's Camille Scherrer's "Magic Book":



According to Pilet's, he used standard Macbooks Pros to achieve decent rendering times. For example, handling occlusion on a 2.0Ghz single core computer, gets him a rendering rate of 18 frames per second for a resolution 360*288 pixels (slightly worse results are achieved with his illumination algorithms). So, AR of deformable surfaces is probably not coming soon to an iPhone near you, but Pilet's research is a step in the right direction.