Core77

In the recent James Bond film Quantum of Solace there is a scene in which M, Bond and other agents share information and briefings around a multitouch table. Just three or four years ago this would have seemed as sci-fi as the now infamous scene in Minority Report, but this time it felt like MI6 was almost behind the curve.

From the work of Jeff Han to Apple's iPhone, Nintendo's Wii and slew of larger multitouch interfaces such as Microsoft's Surface and MultiTouch's Cell, the era of gestural interfaces is here. Physical and screen-based interfaces have collapsed into each other and both industrial and interaction designers have a whole new set of issues to grapple with.

Dan Saffer's latest book, Designing Gestural Interfaces maps out this new frontier. The positive side is that there are a range of exciting new interaction and product possibilities. The negative side is a potentially confusing mess of gestures, each specific to a brand or, worse, owned by one of them through irresponsible patent usage.

Saffer's previous book, Designing for Interaction: creating smart applications and clever devices, brought some order to the disparate fields and disciplines that comprise interaction design, reflecting Saffer's personal journey as an interaction designer:

"I tried just about every job one could try in what was then called 'New Media' before settling on interaction design around 2000," he says. "It just seemed to fit my personality and what I liked to do. I'm not much of a visual designer and not much of a coder, but I enjoy and understand those things and am interested in how things fit together. Those interests and just my personality seemed to gravitate towards this new design discipline."

Testing a prototype. Courtesy Stimulant.

Most interaction designers will sympathize with this—I still haven't found a way to explain to my mother what I do—but does it matter that this growing, changing area does not yet have a clear name?

"I'm of two minds about this question," replies Saffer. "In one sense, it doesn't matter; the work has to be done and interaction design is practiced by lots of people, whether they call themselves an interaction designer or not. On the other hand, if we want interaction design to be as known and respected as graphic or industrial design, it helps to call ourselves by what we do. Naming something gives it both a boundary and a power."

Designing Gestural Interfaces was born out of his own need to try and ground this fast-growing area so that clear design approaches could be developed.

"I wrote this book because, on my third gestural interface project, I was frustrated that I could find only bits and pieces of information about a subject—this subject—that was obviously important and growing rapidly. Why wasn't there a book about these things? I groused. How come I have no idea how big a touchscreen button needs to be? What is a capacitive touchscreen? What kinds of gestures can I use?" he writes in the preface.

Designing Gestural Interfaces is very accessible—I read it pretty much in one sitting—but what surprised me most of all was just how much there is already out there in terms of gestural interfaces and how long they have been around.

"I was as surprised as many people probably are, although for people like Bill Buxton, this topic is all old hat," says Saffer. "HCI academics have been thinking about and dealing with touchscreens and gestural interfaces for literally decades now. But I was also surprised at what individuals have been doing with the technology over the last few years, even though we're clearly still in the early days of its commercial use.

"I don't really have a good answer as to why it has all happened now. It's possibly any number of things: processor speeds, the availability of sensors, the gradual training of the population on touchscreens (via ATMs, kiosks, and other vending-style machines) and freeform gestures in bathrooms that has caused this sudden explosion of the technology. Certainly the unveiling of the Wii and the iPhone came like a bolt of lightening that ignited all the gasoline lying around.

"It's also possible we've reached our limit, from an interaction design standpoint, as to what the standard WIMP [window, icon, menu, pointing device] and desktop metaphor will get us. Once you move into the locations where you find a lot of touchscreens and gestural interfaces, those setups just seem clunky and unnecessary. You don't want to mess with windows when checking in at the airport, for instance."

In some ways it feels like Designing Gestural Interfaces is long overdue, in others it feels like the very first stake in the ground of trying to define an area that is only just emerging in the mainstream. The last 50 pages or so are an appendix documenting a wide range of already existing and possible future gestural interactions. Some of them are hand and face gestures whilst others use the entire body. How important is it to resolve these issues now rather than let them organically emerge?

"Well, the appendix wasn't meant to be a definitive guide to anything, just a help if you are stuck while designing and need some inspiration. I would hope that patterns would emerge organically and not be overly stifled by companies trying to protect their patents."

Playing Starcraft with multitouch. Courtesy Harry van der Veen and Natural User Interface.

Even though he says, "predicting the future is a sucker's game," Saffer has taken the plunge. Some of his predictions are cautious, but they do spot some trends, especially in collaborative workspaces and the potential blunders of gestural interaction design.

"New ways of manipulating data are really exciting, as are large collaborative interactive surfaces. Scary is just the misuse of the technology, which we're seeing now as EVERYTHING needs to be a touchscreen. I'm surprised I haven't seen a touchscreen toilet. I'm not sure we've reached the point of the 'bizarre misuse' yet, although the gestural toilet paper dispenser comes close. Touchscreen spreadsheets are probably right around the corner.

"There's a lot of experimentation going on and it's an exciting time, much like the web circa 1995-97, which, in one sense, is great. I can also see that experimental period becoming annoying for users who just expect their devices to work as they expect. We're still figuring out what this stuff is good for and it's going to take a while, perhaps a long while, before we figure out the things it's really best at. We've only just begun."

Metaphors have long been crucial to interfaces (and anyone interested in this area should read the work of Lakoff and Johnson), but apart from buttons, many gestural interfaces minimize or drop the metaphor altogether because the user is directly interacting with the content (pinching photos small, etc.).

"Metaphor will always play a role with our devices—it is impossible to use or understand them otherwise. As our devices get more sophisticated, metaphor will become even more important. I mean, look at your example—the user isn't 'pinching' anything, just bringing their fingers together, but we think of it as pinching to understand what is going on."

Saffer recently left Adaptive Path to form Kicker Studio, in part because he and his partners "knew that so much of the next generation of technology—touchscreens, gestural interfaces, ubicomp, haptics—all required a tight integration between visual, industrial, and interaction design to make the best products," he says.

"One of the reasons we created Kicker was this sort of deep integration, so the partners come from various design disciplines: industrial, visual, and interaction. We work on next-generation devices with embedded technology: responsive objects or 'things with screens'. Everything from consumer electronics to mobile devices to robots and interactive environments."

The challenge for Kicker and anyone else working in this area is to change over two decades of bad habits in which the screen and software side of a product use separate design processes and teams from the hardware side.

Microsoft Surface recognizes the phone on top of it. Courtesy Microsoft

"Never has it been so important for hardware and software to be integrated as it is right now. Touchscreens are a deep combination of hardware and software and with some gestural interfaces the hardware seems to dematerialize all together and we're left with our bodies as the sole input device. The physical sensors are so important and completely determine what is possible to control with our bodies. We're working on a gestural interface for a prototype media center right now and the camera we're using (created by our client Canesta) is so essential. There is an interplay of hardware and software, driven by behavior and code, that we probably haven't seen to this degree since the 1980s, when computing resources were painfully scarce.

"Most product design is done wrong. That's at the core of why we formed Kicker. Too many devices come onto the market where the interface and the physical product were clearly done by two separate groups that never talk to each other. Mobile phones, appliances, and consumer electronics are the worst for this. You have these beautiful forms or really interesting functionality that are then stuck with an ugly UI or in a black plastic box. One of the amazing things about the launch of the iPhone was that the commercials focused almost entirely on the UI, which I'd never seen elsewhere.

"I don't blame the manufacturers, really, or other design firms. They were set up to make beautiful objects or electronics or stoves. They weren't set up for this new world of networked objects. They've had to bolt on interaction design or industrial design somewhere. We wanted to have a design firm that was created from the outset to address the new challenges of responsive objects.

"By putting all the disciplines together, you're better able to tackle problems and limitations. 'We can't afford a button.' 'Ok, let's move it to the UI then.' Or: 'We need a physical switch for that.' 'Ok, let's design that in.'"

"It's been a tough time to start a company, but we think the vision is a good one and time is on our side. Tens of millions of devices with touchscreens and other gestural interfaces will be sold in the next few years. Hopefully we'll have designed lots of them!"

Outside of the personal gadgets we all carry around, one of the areas in which touchscreens have become most pervasive is self-service machines. Whilst service designers like Live|Work in the UK are trying innovate and refine the entire service experience, it is easy for other companies to slip into using technology due to misguided rationales.

Jive: a touchscreen and tangible interface device for elders. Courtesy Ben Arent

A recent documentary in Germany showed how many of these interactions were not only badly thought out, but also how much this can shift the work onto the customer, thus saving the company large sums of money under the guise of self-service choice and freedom. How should service/interaction/product designers navigate this space?

"The question is: In a world full of multi-touch and gestural interfaces for every aspect of what we do, what are the things we need to look out for to keep it human(e) and positive rather than one struggle after the next?" says Saffer.

"There are a lot of things, from both an ergonomic perspective, from a ethical perspective, and from a simple interaction design/usability perspective. From an ergonomic perspective, the question is: what can you have people do, especially in a public space, that isn't difficult or taxing to the human body. From an ethical perspective, it is about privacy and not having people appear foolish while performing their task. And from a design perspective, it's about focusing on the behavior and context of use to make the tasks as efficient and pleasing to do as possible."

Illuminate table for collaboration. Courtesy GestureTek

"Product designers are now service designers," argues Saffer. "We can't help it. Most of the devices we're making now are part of a network, or need to work with other devices, or at a minimum have a website. Our products are now services. Objects are now carriers of behavior. When we put a service design lens on our work, it changes how you think about the product, by seeing it as a piece of a larger ecosystem. We have to be certain we're applying touchscreens when and where appropriately. Like any technology, there are good uses for it, and bad uses for it—the designer needs to apply judgment when applying it."

Andy Polaine is an interaction designer, writer and Editor of The Designer's Review of Books. The Designer's Review of Books has published a full review of Designing Gestural Interfaces to accompany this interview.