Ric Holland's Blog

BILL BUXTON interview

Principal Researcher: Microsoft – Sketching User Experiences

Ric Welcome Bill and thanks for joining us in the Art of Making Marks. As Chief Scientist at Alias in the early days you did some very innovative work involving Wacom tables. Please tell us about your work then and now as the Principal Researcher for Microsoft and how your work has evolved.

Bill Very few people have ever taken much advantage of the stuff that was in the UD series of Wacom pen/tablets in terms of there capability to sense multiple devices on the tablet surface at a time. We started doing that very, very early in terms of being able to support two headed inputs. So what’s interesting is we built what was really the first digital airbrush. As far as I can see, anybody who has ever seen an actual airbrush much less used one has noticed that what passes for an airbrush on any computer paint program has no resemblance to a real airbrush or furthermore to employ the skills that one needs in using an airbrush. In terms of a visual language, you never ever see anybody using an airbrush without holding the frisket in your non dominant hand in order to dodge and mask the principle image. The visual language of an airbrush is a soft feathered edge on one side and a very hard edge on the other and that only can be achieved using a frisket or stencil in combination. There are some programs that give you very awkward ways to position the frisket using your mouse or stylus being the airbrush but we actually let you do the thing you couldn’t do with any other digital technique and that was to hold the frisket in your non dominant hand and hold the airbrush in your dominant hand and work in a very natural way. That was only possible using the UD series of Wacom tablets and there are other things too that we started to do where you could hold rulers or other constraints. For instance if you were doing a pivot point and rotating something in 2D or 3D you could push down with your left hand holding one device, say a puck in your left hand and specify the pivot point or the anchor point and then use your dominant hand holding the stylus to control the degree of rotation.

We started to develop these tools from about 1989/90 to really take advantage of Wacom’s dual sensor technology, though mostly experimental systems we actually did support some of this stuff in a product called Studio Paint when I was at Alias. There’s a video of that stuff on my web page. The problem with these sorts of things was that if you think about it the mouse took about thirty years from invention to the point that is caught on and became ubiquitous and so some of the ideas that we were doing back in 1989/90 are only just now becoming mainstream using other techniques like multi touch on devices like the Apple iPhone. The pinching type of gesture to scale things and rotate by grabbing with two fingers or two hands, we were actually doing all of that with Wacom tablets back then. Those techniques were all well known and we published and demonstrated them and so Wacom had a huge impact there.

So back in 1994 when I first joined Alias one of the first places I visited was Wacom headquarters in Japan and I met one of the co-founders, the person who did the hardware whose name is Murakami-san. He didn’t speak English and I don’t speak Japanese, he didn’t really know who I was and I’d never met him before and it started off as a typical Japanese business meeting, fairly formal with the guests on the one side but very quickly he started to get excited. I’d showed them a couple of things that I’d been doing and then the meeting just broke into chaos and all of the other people sort of looked like this never happens, you just don’t do this in Japanese meetings. He expressed to the interpreter something like, ‘I’ve been looking all my life to find someone like you! I’ve been having all these dreams of hardware and what the hardware can do but I’ve never found anybody who understood the idea from the software.’ I was saying to him back through the interpreter, ‘I’ve had all these ideas about how software could work but I’ve never found anybody who made the hardware.’ The meeting broke up completely at that point. In any formal business meeting in Japan where an executive of one company comes to see another for the first meeting this sort of thing just doesn’t usually happen and so this was not a typical meeting at all by this point. He then just grabbed me and took me up into the lab behind the doors where visitors normally wouldn’t see to show me some stuff that they were working on. He’d built things that so far have never been released which included things like an industrial designers full set of markers where he’d have a handful of styluses each one marked with a direct colour just like markers in a marker set so there was no menu, you just grabbed the marker you wanted and start using it. He also had an eraser that really was an actual eraser and you just used that on the tablet to erase whatever was going on. It was a remarkable concept at the time. There was this degree of inventiveness at Wacom that has never been commercialised and he was exploiting the ability of the technology to do more things. Just like now with the Intuos series, the tablet can tell if it’s interacting with a stylus or a puck and it can tell if it’s the airbrush stylus or the regular stylus. Basically you do that by changing the frequency in the coil. In those days in the UD series you changed the actual coil so the objects all had different resonance frequencies and he was pushing those types of systems to do very interesting things. He pulled out another system and he allowed me to videotape it so I actually have a videotape of this paint program and I actually have an article on my web page if you want to read about all this stuff.

Under Recent Work and Activities there’s a paper called Surface Intangible Computing, in preparing that paper I had the videotape which I’m happy to make available so you can take a look at it. I’ve also still got the paper written up in Japanese about this system and it was one of the first examples of what has now become called tangible or graspable computing, a new emerging area that this is one of the two first instances of. What we had was a paint program with no menu and it was just a whole screen and in those days the norm was a 12” CRT with these little tangible objects or what today are calling physical icons or ‘Phycons.’ One was a little filing cabinet, one was an ink pot, one was a stylus and one was an eraser and when you wanted to paint you painted with the stylus, when you wanted to pick a new colour you put the paint pot on the tablet and you pushed down on the lid and your paint palette came up and you moved it around to the colour you wanted to release with your brush and then you started painting with that. When you wanted to erase you just pulled up the eraser, a real eraser with a coil embedded in it. If you wanted to file something or retrieve something from the file system you put the little filing cabinet on the screen and you’d push the drawers which were spring loaded and so you held the whole menu, all the menus were held in your hands not on the screen and they were physical icons that represented the function, the same way that icons do on the screen. That project was extremely influential and then that led to a whole stream of research because I had a student at the time named George Fitzmaurice and we’d been working on graspable computing stuff and he saw that and some other work from Denmark which was unrelated to this of which I’d been involved. His PhD thesis is still the classic in this area and then a guy named Hiroshimishi, a Japanese guy who was a visiting researcher in my lab at The University of Toronto at the time took that whole concept to the media lab and really put it on the map. So the work out of Wacom had a huge impact both in technology but even the concepts that they were developing in terms of this whole other direction in computer science and you’ll see I write about it in that particular article. Surface and Tangible Computing, and the “Small” Matter of People and Design.

Ric Fantastic.

Bill The other thing that maybe is of interest and I know you’ve talked to Duncan and Duncan’s worked on tablets and he’s done a lot of the paint stuff in Maya but in terms of most of the pen/tablet work that was done at Alias that was actually done by my group. Duncan was an absolute creative genius and I’m not at all disrespecting his contribution. His ability to allow you to paint with these natural elements instead of just a colour you could paint with a 3D texture, and 3D elements and so on and so forth. It was just mind boggling. It’s still is mind boggling but there were a whole lot of other things that went on there in terms of developing techniques surrounding the tablet. You’ll find a paper called T3 which really pushed what you could do with two hands, a puck and a stylus, really quite far. The culmination and the commercial side of this work was perhaps the first application for the Tablet PC that was ever designed from the ground up to be pen based and take full advantage of Wacom technology and the Tablet PC. This project that I led was done in five months from no team, no concept to shipping in 2002 and was called Alias Sketchbook, now it’s called Autodesk Sketchbook Pro.

Ric Yes I love using that tool, it is very elegant.

Bill I was the lead on that project which came out of my research team at Alias. Gordon Kurtenbach was my PhD student and I think I first hired him when I became head of research at Alias. He became head of my research team and took on the role of lead engineer. His Thesis called T3 and George Fitzmaurice’s PhD thesis on grasping is full of stuff around innovation with Wacom technology. Radio Menus and Hot Box has driven the Alias stuff and were all developed for the Wacom tablet. Scott Rawlings and his group at Wacom as well as the folks in Japan were very, very helpful so there’s really a lot of history there and the reason why we were able to put Sketchbook together so quickly. It is still one of the most highly rated programs in terms of useability and user satisfaction. Also the reason why it had no bugs was because we were building on stuff that we already knew how to do as we’d spent years figuring out some of the basic stuff behind it. It’s a great example of how research can inform product development really effectively.

I wanted to prove to my colleagues and the executive at Alias that there was a different methodology to use when developing products where you actually employ a proper design process, a process that an industrial designer can use. I talk about that in my book Sketching User Experiences, getting the design right and the right design. Sketchbook was the first product I took from beginning to end following that process and also it’s one of the only products that ever came out of Alias that was designed from ground up and shipped on time, on budget, on schedule and certainly the only one that ever shipped on time and on schedule. We’d gone through a period of several years developing products that didn’t ship on time or failed or were withdrawn or were highly compromised and so the thing about Sketchbook was that it was the first product that from beginning to end followed that methodology where you really did the design up front and the business planning up front and how you’re going to build it up front and then you go ahead and build it.

Ric So you were innovating in both Process and Product design methodology?

Bill That’s right and the fact that we had no team and we didn’t know what we were going to build on July 1st and yet the product shipped on November 7th which is the day the Tablet PC was to be launched and we had no Tablet PC’s in house until very close to October and we had no deal with Microsoft until probably around September but we still made the press date for October 1st is a testament to that methodology.

Ric Brilliant.

Bill That’s not a well known story but we would never have been able to do that had we not been doing our development on Wacom tablets. We knew that the drivers worked and everything worked and we would never have been able to do it had we not had known a lot about the user experience or really understood the technology. When we saw this opportunity it was just like ‘man we’ve done our homework’ and it really was a team effort. There was about eight full time equivalent staff or less and it was actually five months from the beginning day to end shipping day. It was a great experience.

Ric Sketchbook continues to evolve, have you had much to do with version 2?

Bill No, in fact I think that what it has evolved towards is more of a paint program which was never the intent and I think that’s unfortunate. I would have liked to have seen it go a different direction but I’m not going to try and second guess the business reasons for that. It’s not for me to say what’s right or wrong. I would have liked to keep it as simple as possible so that you have as much user flow as possible. There are a few ideas in there that were pretty powerful, there was a zoom/pan tool that was never developed to reach its full potential and it started to go more and more like other paint programs rather than being very distinct. My view about that class of product is that you make a number of things very light weight and easy to use and then you design with clean interfaces so that the way you get real fluidity and personalisation is by the combination of the components that you use as opposed to building more and more features into a single application. That’s just a philosophy of design and I don’t believe that everybody should follow it and it’s really just my personal view. I would say that for the most part Autodesk have retained the heart of Sketchbook and they’ve kept it up to date and they’ve kept the quality there so who’s to complain.

Everywhere I go I see people using Sketchbook which reflects my view of the importance of sketching and how I’d seen it in the context of industrial design. Digital technology has enabled us to do all kinds of wonderful things but actually this has hurt the design process because of the lack of sketching. Designing on computers has tended towards people doing less and less sketching, so how could we make it so that we did more and more sketching? Make it better than paper because by having simple layers and being able to try a couple of different things without risking your entire sketch is a better experience. So I’m really quite happy with it.

Ric How do you find using it with the Cintiq 21?

Bill Yes the Cintiq has many positive things especially as they get larger and I think there’s nothing I can say that I don’t like about it for what one does with it. If you put a really powerful computer behind it it’s like a Tablet PC on steroids! The challenge with the Cintiq is that until it gets larger even than it is right now it’s difficult to do two handed stuff on it. Like everything it works for something and is bad for something else and so in no way am I’m saying is Cintiq a bad product just sometimes a pen/tablet is better, sometimes the Cintiq is better.

Working directly on the screen is really good and Wacom have done a very good job on the Cintiq in terms of the texture of the glass and minimising the problems of parallax. What you can’t do is work two handed with the stylus and puck and even if you could your hands are now obscuring your work whereas on a tablet the coordination actually to work indirectly is extremely quick to learn and then your hands never cover your work and therefore you can have different tools on your tablet without obscuring it. So there’s pros and cons to both and I think that it’s like no one really use the same kind of pen or the same kind of car and just because I don’t drive the same car as you doesn’t mean yours isn’t a great car, you just have different purposes. Maybe you live in the country and you need a four wheel drive and I live in the city so I drive a Smart car for example. Not that I own a smart car but just that I’m trying to make a point.

That’s the whole idea of a product line and I think for other people just having a tiny little Bamboo is what they should have and they don’t need anything more. I believe that what is going to happen is that pen based stuff is going to move increasingly out of just the graphic arts field. For example I own almost every eBook Reader there is. I have a Sony Reader and I have a thing called iRex which has the Wacom technology in it and I have the Amazon Kindle but I can tell you that for my day to day work the only one I use is the iRex, which is the least well known of all. I had one of the first ones to ship and it’s the most expensive and probably the one that’s going to fail in the market but it’s the only one I’ll use. My life involves a lot of reading and commenting on documents and so without the ability to annotate with a pen it’s just not an acceptable experience. Reading for me cannot be disassociated from the activity of marking up and annotating. On the Kindle I can do text notes like you can in Adobe Acrobat but I want to do graphics, circles, draw arrows and that sort of stuff and I want to be able to do it out on the dock in the summer in the sun which I can’t do on my Tablet PC. I carry the iRex with me everywhere. I also think on mobile phones touch technology is a great thing, I know as much about touch as I do about pen based input so this isn’t one against the other, they’re just different, and for me this touch phone versus style phone debate that’s going on right now is a totally bogus debate. It implies that it has to be just one or the other when in fact of course it depends on what you’re doing as to which type you should use and it’s absurd to have something that’s a palm sized application device with a screen that I can’t use as a post-it-note type of device. So again if you come back to the stylus and I can only write down a post-it-note instead of being able to just scribbling on the screen it’s just as absurd. If I just have only touch on my phone then I can’t take notes, I can’t write, I can’t do quick drawings and save the file and send on, and so forth, my fingers are just too damn fat to do anything but finger paint which is the work of a child, not of a professional.

Ric So of course you must be very well aware of Wacom’s developments in RRFC touch technology, what’s your opinion of this in combination with our EMR® pen-input technology?

Bill My first surface was a three foot diagonal display that I built back in around 1992 which had a high resolution digitising stylus, it wasn’t a Wacom because Wacom didn’t make this type of product but I used a transparent overlay system from a company called Scriptel and so the left hand had gesture recognition and you held the pen in the other hand so you’re talking about something I’ve been doing for about fifteen years.

Ric What I meant though is the actual technology we are bringing to market now.

Bill There is a number of people playing in that space right now. It’s already shipping on Dell. I haven’t actually used the Wacom. I don’t know whether you’ve got the palm rejection right or what kind of gestures, what kind of touch and so the answer is I haven’t used the Wacom technology yet and I’m interested in it. Somebody’s going to do it right and here’s the thing about Wacom. Wacom’s been an outstanding hardware vendor but Wacom has been at best a mediocre software company, that’s for your drivers and other things. You’d either farmed that out or not controlled it, and the support from the software side has not been good and it never has been and so as we move into touch coupled with pen this is absolutely not a hardware only problem. You can put the best touch surface and combine it with the best stylus technology and you still have an unusable system unless you’ve got the software layer. Wacom has never taken ownership of that and that is the challenge so I’ll be very curious to see how that goes. I track these technologies still very carefully and I’d love to get a sample of the units and actually try one out and see for myself.

Having said what I just said my area is in the software side of things which sits between the hardware and the devices and the user as well as the applications and Scott will tell you that I’ve been a total pain in the arse repeatedly in terms of asking for things and him having to say no or maybe or helping me. He’s been really good. He’s a very good friend and that notion of trying to push the company and help them gain the insights that will make them get the products right. It’s what I’ve always done and what I will continue to do because I don’t want to be in that business, I just want to have the right technologies available from the vendors which our software runs on. It’s as true now with Microsoft, as it was when I was with Alias.

The thing with Scott and he’s absolutely right, and as I was an executive at Alias I understand the things that he’s always been adamant about, was that at the expensive of doing something cool and innovative he wasn’t going to compromise the quality of the product and the brand. I think that’s absolutely the right decision and I think the real trick right now in terms of this particular space is that it’s really competitive and there’s a lot of threat in some ways to Wacom’s position if the touch stuff takes off and holds back the stylus, even if in many cases it shouldn’t a lot of times the market looks like a bunch of lemmings and just starts following. The iPhone has had a really large impact on what people think they need or what they think the future is and I think the iPhone is a wonderful piece of design and I also think it has some gaping flaws, the note taking and sketching difficulties and the lack of a stylus is not the least of them.

Ric It amazes me of Apple’s lack of attention to pen input in this whole area. It constantly frustrates me and my colleagues when presenting pen technologies for medicine let’s say and somebody says well you know I don’t want to use Vista, I’ll be using the Mac OS and so I have to say ‘Well I’m sorry that the experience is very different’.

Bill [Laughs] The thing is you should be able to be more blunt. Apple does no research, they have no research. They don’t push that. Every once and awhile they’ll acquire something. They acquired a company they called Fingerworks and if you read the PhD thesis of the technical guy from Fingerworks they acquired, it’s all about the work we did in Toronto. Steve has been very adamant about their iPhone and these types of things are not going to have a stylus, he’s made statements like ‘if God wanted us to use a stylus he would have given us skinny drawing fingers.’ Fine, he’s been very successful with it. I respect what they’ve done and I really enjoy a competitive marketplace because even by making such a strong statement in such a strong product you define the space and the value of looking at input and so I would say that my ability to get other approaches to input implemented was actually easier because of Apple so they helped me as much as themselves.

Ric Are you able to talk a little bit around the Microsoft’s Surface Technologies and your work in that space?

Bill I think the best I could say about surface is I think those technologies are going to develop pretty quickly or faster than anybody predicts. I think they’re going to become more cost effective faster than anybody thinks for a number of reasons but not the least of which is what’s going on with some of the technologies and the stuff I’m happy to put on record is probably along the lines of my recent talk that gives some vision about that space and is captured in an article I wrote about tangible and surface computing which was a key note talk I gave at the International Solid-States Circuits Conference in March 08. It’s not a deeply technical paper because the audience didn’t know much about applications but it’s just trying to talk about here’s some interaction concepts where I think things are going in pure science and how we’re going to get back to the computers and how this might help shape how you approach design of inner line circuitry.

Ric I saw a very good video presentation from Microsoft late last year addressing health and medical markets which took a number of user scenarios and outlined the opportunities for Microsoft’s Surface technology in that space.

Bill I know the video you speak about and yes we do a lot of those types of envisioning videos to try and frame ideas because it’s not about any single product, it’s about the larger ecosystem and so the value of those types of videos is to show the devices in a larger context and the relationships to the other technologies that would be part of the ecosystem. Even when working on an individual product you’re always keeping in mind the larger ecosystem, the value system and the conceptual framework in which it’s going to exist. Those types of videos are futuristic but it’s actually trying to say okay so if we’ve got ubiquitous computing and we have these things we carry around with us and we have these venture networks and everything is embedded in the environment, what are they really and how do they work together? What does it really look like? and I’d say that Medicine is one of the areas that is of very high interest to the company, I’d say it’s a priority.

Ric Our CEO Yamada san has a similar vision as to the partnerships and how our role as an interface solutions organisation in those ecosystems you suggest may manifest further down the track and he has set the organisation on a path to work in harmony with these sorts of visions of the future.