A thought struck me the other day: with the “large” size of the iPad’s screen, why couldn’t I make physical objects that could be uniquely identified by the iPad? If possible, this could potentially allow uses of the iPad along the lines of larger, more sophisticated multi-touch surfaces – ReactiPad anyone?

Unlike most multi-touch surfaces, we can’t use computer vision to identify objects (since, well, there’s no camera under the display) but we do have the ability to track “up to” 10 fingers on the display at once. I figured that with, say, three specific points (which is what I’ll call “fingers” in this post) within a certain radius, a fair number of uniquely identifiable configurations would be possible by calculating the distance and angle between the points. (and even the orientation can be calculated) – add in four or five points and there should be a fair number of unique markers that can be created.

Not having an iPad when I started experimenting (they weren’t and, as I write this, still aren’t available) I whipped up a simple app for my iPhone that would tell me the number of touches it detected at any given time. This let me start to experiment with what objects would fool the device into thinking I had my fingers on the surface.  First try was some conductive foam from a box of ICs I had lying around- it worked! But only when I was holding it! I tried copper foil and mesh, and several other items before deciding to do some research online. I found that several people had created styli for the iPhone by using a damp sponge wrapped tightly in plastic. That was easy and after grabbing some vinyl gloves and sponge material, I was well on my way to physical objects. It took a couple of iterations to find that the spacing of the touch points had to be fairly generous on my test iPhone, so the objects will have to be pretty large at this stage, but this is a test afterall. I also chose to take a shortcut initially (to prove the viability of the idea without days of work) and chose to ignore the spacing and orientation of points and concentrate just on the number of points detected – this limited me to a 2, 3 and 4 point object due to size constraints.

Sponge Fingers

The Three and Four point object prototypes

Finished Prototypes

Having so many things to work with at my fingertips made they prototyping phase a breeze. As I mentioned, a pair of vinyl gloves and some sponge material came first, some nice, black art board second, and some wire to tightly close things up was the final ingredient.

After creating these ugly little prototypes, I was thrilled to find my iPhone test app was recognizing them properly over 80% of the time. When testing moving points (touchesMoved) instead of just starting point (touchesBegan) the accuracy jumped to near 100%. I had some viable objects to use! Now I needed something to test it with, so I customized a version of our first in-house iPad app, Akina HD. You’ll get to see it in action as soon as I grab a video! UPDATE: Here’s the video. Also, Emily Daniels (who narrates the video) made the ugly things into super-awesome art pieces in their own right.

I am anxious to see what others do with physical objects and the iPad. Integrating bluetooth connectivity into objects may provide some amazing interface possibilities, but with some refinement, even these “dumb” objects could make for some very interesting UI innovations. I’m thrilled to be able to carry around a multi-touch surface anywhere I go!