There are touchscreen displays and 3D displays, but now there’s one that combines the two: a flexible screen that users can pinch, poke and stretch.

The screen, designed by Dhairya Dand and Rob Hemsley, both of MIT’s Media Lab, is calledObake, named after a shape-shifting spirit of Japanese myth. It’s comprised of a flexible rubber sheet laid over a set of actuators. The whole thing sits under a camera built from a Kinect projector that beams images onto the sheet. The camera also measures how deep the screen is poked into it or how far its stretched.

SEE ALSO: New Hologram Tech Sets 3D in Motion

If you pull on it, the sheet makes a little mountain. Poke it and it dents. Rub your finger on it and it senses the friction. Stretch it and it shows the distortion on the image.

All this is geared to making touch displays that are truly 3D. Current technology lets you see 3D images, but that’s an optical illusion rather than the real thing. And virtual reality allows users to manipulate objects with gestures, but a screen that you can actually touch and stretch is the next best thing to being there. Dand and Hemsley call it a 2.5 D display.

There’s another aspect to this work: The two told ExtremeTech that the current state of the art touch screens are still working with the same assumptions that guided the mouse. Even though point, drag and click has been replaced by touch, drag and tap, it all still assumes a 2D image.

A lot of 3D displays operate the same way, and don’t take advantage of the range of gestural controls. But the reason for that isn’t just that technically it’s hard. As computer users we have all collectively gotten used to certain ways of doing things with our hands: We pinch and zoom, tap, point and click. It’s become a part of our collective vocabulary.

It’s similar to the reason keyboards look the way they do. Keyboards mimic a typewriter, even though there is no logical reason to have a QWERTY keyboard on any computing device — there are more efficient ways to arrange keys, for instance. But the QWERTY keyboard was standard and it’s what most people knew how to use.

So creating a 3D display that used completely unfamiliar gestures would be self-defeating. Dand and Hemsley wanted to build something that was interactive, yet not so baffling that it is unusable.

That’s why they went with the design they did: the stretchable surface allows for gestures lots of people know already; at the same time it can go beyond the capabilities of a typical touch screen.

SEE ALSO: 3D Holograms on Your Phone, No Glasses Required

There’s a long way to go, of course, before this gets on to a smart phone. Actuators are relatively large, and there would have to be room for pressing in as well as extruding, and the need for a projector. Odds are this technology will first show up on tabletop applications.