Ric Holland's Blog

NUI (Research Archive)

Razorfish ports DaVinci interface to Microsoft Kinect

Wikipedia NUI =

Ric Holland – PhD Research at UNSW CoFA (University New South Wales, College of Fine Art)

Dr Phillip George Academic Supervisor – Senior Lecturer, Media Arts, University NSW CoFA
Prof Ross Harley Academic Supervisor – Head of School, Media Arts, University NSW CoFA

Masahiko Yamada Industry Supervisor – CEO Wacom

Prof Judy Kay External Academic Advisor – Head of IT – Computer Science, Sydney University
(CHAI) Computer Human Adaptive Interaction Research Group
Australian Research Council Research Fellow, President AIED Society

Darrall Thompson
External Academic Advisor – Director of Teaching – School of Design,
Architecture and Building. University of Technology Sydney
Dr Jeffrey Crass Director, Design Centre Enmore & Dean, TAFE NSW Higher Education

Industry advisors

Russell Brown (Global Creative Director Adobe)
John Derry (Co-author Corel Painter)
Bill Buxton (Global Research Microsoft)
Duncan Brinsmead (Chief Research Scientist Autodesk)
Scott Rawlings (Global Product Planning Wacom)
Joel Bryant (Global Product Manager Wacom)
Glenn Tsunekawa (Global Product Manager Wacom)
Ron Cobb (Visual Futurist)
Syd Mead (Visual Futurist)

PhD Title – “21st Century Mark Making
A survey of the human computer interface, applications and implications.”

An international exploration of 21st Century Mark Making – applications
and implications of expressive human computer interfaces in relationship
to their traditional counterparts. Traditional 2D & 3D expressive art tools
and techniques recorded and compared to their digital counterparts,
building a foundation for future predictions in Expressive NUI innovation.

Statement of Topic and Aim

From my own knowledge, skills and experience I propose to chart with the help of global experts in specialized fields of art, design and computer science the past, present and future of expressive mark making techniques. I will be looking for where the ‘gaps’ are to faithfully connect humans with Natural User Interfaces and Applications that empower highly developed skills and creativity. I will create a range of instructional content and interviews from the world’s most celebrated practitioners and masters for publication.

NUI – ‘Natural User Interface’ is a fundamental shift in thinking that has been occurring within the academic field of computer science and is mirrored in software development, industrial design and web development/design industries in the forms of User Centered Design and Usability. These methodologies embrace the notion that humans interact with technology best when it ‘fits’ their needs and motivations in a timely way. These principals have been adopted globally as best practice and are continually being researched and refined. For developing computer interfaces that respect people’s deep specialized skills four topics must be considered. These are -

1. Motor Sensory Skills
2. Cognitive Skills
3. Social Context
4. Emotional Responses
(‘A prosthesis to the skills of the artist’ – Bill Buxton)

This applies to disciplines such as Art, Design, Photography, Sculpture, Animation, Music, Video, etc. Expressive mark making is a primal human urge. To illustrate ideas and emotions, to communicate and document social history are only just a few of the motivators over the centuries. From early cave paintings to the great works of modern time, mankind has striven to interpret the world through his/her own eyes using tools and techniques that continue to evolve and innovate. A very significant change in that evolution has occurred only relatively recently with the invention of digital technology.

Although digital mark making has matured somewhat over the past 20 years it is still only in its infancy. Illustrators, designers and concept artists have pushed their results from software programs like Photoshop, Painter, Sketchbook Pro, etc. to impressive levels of sophistication and 3D ‘sculpting’ applications like Mudbox and ZBrush which form the basis for expressive organic work. But traditional tools and techniques still remain the last bastion for the ‘true’ artist. For art works to be perceived fully by the viewer as ‘tangible’ it is preferred that the brush strokes are ‘real’ or the object can be touched. An interesting current trend is for an artwork to be conceived digitally and then made ‘real’ via a blend of digital and traditional media techniques.
(ie. – Jeremy Sutton) This is an example of traditional and digital tools and techniques becoming irrelevant as separate disciplines but perceived by the artist as just common tools available in the process of making marks.

But what of the many and varied traditional techniques that have evolved from so much enquiry and discovery over centuries of human endeavor? How close are we really to totally synthesizing analog to digital in the field of expressive mark making in such a relatively short time? I propose that this question is unanswered and so I intend to chart with the help of global experts in specialized fields of art and the computer science the past, present and future of expressive mark making techniques in relationship to Nature User Interface development.

Method/Approach –interviews with leading artists, designers, photographers, 3D sculptors, Videographers, etc. from digital and non digital disciplines. These interviews will form the basis of further discussions within global communities and at my blog which currently contains 3 years of field research for my first book ‘The Art of Making Marks’ – Wacom’s celebration of 25 years as a leader in Natural User Interface technology.

Through practical exploratory sessions with traditional artists/instructors I will catalogue and cross reference traditional tools and techniques to digital tools, techniques and NUI interfaces.

My diagram indicates traditional to digital convergence. The point of intersection represents where digital and traditional have completely merged together so human skills, traditional and digital technologies have formed a new professional standard moving forward. Some fields have already converged and some are closer than others and it is my intention to focus the outcome of my thesis to fully understand the ‘gaps’ and chart likely courses for the future.

The Art of Making Marks book represents a significant summary of field research towards “21st Century Mark Making – a survey of techniques and the human computer interface, implications and applications.”
References – Painter 11 Creativity – Jeremy Sutton, Sketching User Experiences – Bill Buxton,

Review of literature and relevant practice –

Scott Jenson, The Simplicity Shift
Designing innovative consumer products takes both passion and perspective. Passion to overcome the hundreds of problems that always crop up and perspective to know which problems are the most important to solve.

David Bayles and Ted Orland, Art & Fear
Observations On the Perils (and Rewards) of Art making.

Steve Krug, Don’t Make Me Think

The title of the book is its chief personal design premise. All of the tips, techniques, and examples presented revolve around users being able to surf merrily through a well-designed site with minimal cognitive strain.

Dr Helen Sharp, Interaction Design: Beyond Human-Computer Interaction
A distributed cognition analysis typically involves examining the distributed problem-solving that takes place including the way people work together to solve a problem.

Donald A. Norman, Things That Make Us Smart: Defending Human Attributes in the age of the machine
A Human-Centered Technology – The human mind is limited in capability. There is only so much we can remember, only so much we can learn. But among our abilities is that of devising artificial devices – artifacts- that expand our capabilities. We invent things that make us smart.

Bill Moggridge, Designing Interactions
Why a Mouse? Who would choose to point, steer, and draw with a blob of plastic as big and clumsy as a bar of soap? We spent all those years learning to write and draw with pencils, pens and brushes., Digital Painting Techniques: Practical Techniques of Digital Art Masters
Digital Painting Techniques is a compilation of tutorials written by various talented digital artists. It is broken up into eight chapters with roughly five to seven tutorials apiece, and a gallery at the end.

David Cole, Complete Digital Painting Techniques
Simulate and replicate traditional art-creation techniques – from dry brush and wet-into-wet to crosshatching and graffiti – using digital applications. Featuring step-by-step instructions using popular programs such as Adobe Photoshop and Corel Painter, this encyclopedia of techniques is a bookshelf staple for designers, illustrators and fine artists alike.

Cher Threinen-Pendarvis, The Photoshop and Painter Artist Tablet Book: Creative Techniques in Digital Painting
Learning to actually draw and paint with a graphics tablet has barely been covered. If you’ve been a traditional artist for years, you still need basic exercises to learn how to handle the “digital brush.”

Scott Spencer, ZBrush Digital Sculpting Human Anatomy
Whether you are sculpting human or alien characters with ZBrush, your ability to create convincing results with an efficient workflow is extremely dependent on your knowledge of sculpting anatomy. As any established character artist will tell you, the education you get through sculpting your first human body following anatomically correct guidelines is one of the most important learning processes you will go through.

Mike de la Flor and Bridgette Mongeon, Digital Sculpting with Mudbox: Essential Tools and Techniques for Artists
Explores digital sculpting in its entirety from the art studio, to computer, through to production. Traditional sculpting techniques and digital sculpting in Mudbox with comprehensive tutorials. Interplay between Mudbox and other 3D/animation programs like 3ds Max and Maya – export Mudbox models -Use Mudbox to prepare rapid prototypes, build/mill armatures, and create hybrid sculptures.

Robert Barrett, Life Drawing: How to portray the figure with accuracy and expression
Clear chapters which demonstrate how to study figure drawing from a wide variety of approaches. It does not limit to one particular procedure but offers many ways to get into the subject, all of which it demonstrates via very fine drawings/sketches. Very strong on structural approaches.

Lark Books, The Figure in Clay: Contemporary Sculpting Techniques by Master Artists
Nine artists are featured and all use a wide variety of techniques: coiling, pinching, slab, casting, and molds – alone or in combinations to create some of the best modern ceramic sculpture being produced today. Each of the featured artists writes an introductory essay explaining their philosophy, work and techniques.

Jose Arguelles, The Transformative Vision: Reflections on the Nature and History of Human Expression.
Arguelles gets right down to the root causes of the crisis of modern civilization. He calls the feminine element psyche (breath or soul) and the male element techne (skill). Since the Renaissance techne has come to dominate life, giving us a purely mechanical existence at the exspense of psyche, which has been relegated to the unconscious. Arguelles shows how art has manifested this split, culminating in the sterility of modern art today.

Michael Benedikt, Cyberspace: The First Steps
Cyberspace has been defined as “an infinite artificial world where humans navigate in information-based space” and as “the ultimate computer-human interface.” These original contributions take up the philosophical basis for cyberspace in virtual realities, basic communications principles, ramifications of cyberspace for future work and play places.

John Carroll, Making Use: Scenario-Based Design of Human Computer Interactions
Traditional approaches to the complexity of the design process via abstraction, treating design problems as if they were composites of puzzles have given way to Scenario-based design uses concretization. A Scenario is a concrete story about use. Scenarios are a vocabulary for coordinating the central tasks of system development—understanding people’s needs, envisioning new activities and technologies, designing effective systems and software, and drawing general lessons from systems as they are developed and used. Instead of designing software by listing requirements, functions, and code modules, the designer focuses first on the activities that need to be supported and the allows descriptions of those activities to drive everything else.

Alan Cooper, About Face: the Essentials of Interface Design
Primarily directed at software interface design, its principles could also easily apply to other disciplines, such as architecture and industrial design – anyone concerned with how people use products.

Anthony Dunne and Fiona Raby, Design Noir: The Secret Life of Electronic Objects
Beneath the glossy surface of official design lurks a dark and strange world driven by real human needs. A place where electronic objects co-star in a noir thriller, working with like-minded individuals to escape normalisation and ensure that even a totally manufactured environment has room for danger, adventure and transgression.

Paul Dourish, Where the Action Is
Introduction to the world of phenomenology, sociology and philosophy as pertaining to Human-Computer Interfaces. Arguably “social computing”, “tangible computing” and “embodied interaction” add up to a construct that can effectively inform the design of new HCI devices.

Matt Fuller, Software Studies
Computer scientists, artists, designers, cultural theorists, programmers, and others from a range of disciplines each take on a key topic in the understanding of software and the work that surrounds it. These include algorithms; logical structures; ways of thinking and doing that leak out of the domain of logic and into everyday life.

George Lakoff and Mark Johnson, Metaphors We Live By
Deconstruction of the term “conceptual metaphors”, and the complex way in which they interact to structure our experience of reality. The authors examine how common ways of speaking and thinking actually reflect a relatively coherent metaphorical system, suggesting philosophical and political implications.

Brenda Laurel, The Art of Human Computer Interface Design
1990 human-computer interface design. Wide range of opinions, experiences and conclusions on what really works (and what doesn’t) on interfaces back then.

Malcolm McCullough, Digital Ground
The many disciplines that support interaction design: psychology, architecture, cultural anthropology and technology.

Bonnie Nardi, Context and Consciousness: Activity Theory and Human Computer Interaction
A provocative compendium of theoretical expositions and methodological examples for the application of activity theory in the design of collaborative computing systems and the implications of activity theory to instructional design/architecture, knowledge management and all that fun stuff.

Bonnie Nardi and Vicki O’Day, Information Ecologies: Using Technology with Heart
Perspectives on how technology is affecting our society in both an anthropological as well as a sociological view catering to both technical and non-technical individuals.

Don Norman, The Invisible Computer: The Design of Everyday Things
Strives to support the design of better information appliances due to the complexity of the computer coupled with creeping featurism. Human centered design must be used to overcome increasing complexity.

Jennifer Preece, Yvonne Rogers and Helen Sharp, Interaction Design: Beyond Human-Computer Interaction
Interaction Design is a meat and potatoes book about HCI. Rather than focusing on the software that drives the application, the book analyzes how users actually interact with the system. This interaction is what ultimately will determine whether a system is successful or unproductive.

Jef Raskin, The Humane Interface
Goes outside the realm of currently-used computer systems, and introduces ideas that can’t immediately be put to good use. But that is necessary to get a complete picture of the concepts. (Not to mention the help that it might give to someone who decides to go about designing an all-new computer or operating system.

Nathan Shedroff, Experience Design 1: A manifesto for the creation of experiences
An attempt to define this field in as inclusive a way as possible. It covers a broad cross-section of media with examples and theory.

Edward Tenner, Why Things Bite Back—Technology and the Revenge of Unintended Consequences
From medicine and natural disasters to plant and animal pests to machinery and software to how better running shoes lead to more injuries. In short what we have here is a warning: we are not as smart as we think we are. We are not as completely in control of our lives as we would like to believe. We are in danger of really screwing up the works at any time.

John Tahckera, In the Bubble
A loosely structured conversation with many voices, a freestyle rush into 10 clusters of ideas on how designers – architects, industrial designers, artists, engineers, urban planners and others – should be thinking about today’s big design issues, including sustainability, needless complexity, and the frenetic pace of the social and business worlds.

Edward Tufte, The Visual Display of Quantitative Information
A simple proposition: graphs and graphics that represent statistical data should tell the truth.

Lawrence Erlbaum Associate, Human Computer Interaction
Defines the psychology of human-computer interaction, showing how to span the gap between science & application. Studies the behavior of users in interacting with computer systems.

View more presentations from Javier Gonzalez Sanchez.

Crayon Physics Deluxe from Petri Purho on Vimeo.

Nathan Shedroff-Keynote: Meaningful Innovation Relies on INteraction and Service Design from Interaction Design Association on Vimeo.

Stereoscopic 3D displays explained simply

Cardboard Anaglyph glasses are probably the easiest way to make 3D images is to separate the right and left image using colors. The image has two color “layers”, and you separate the layers using glasses that has blue/red lenses (or cellophane paper, in the cheapest glasses). This is called Anaglyph 3D, it’s cheap and easy to do as you don’t need a new TV and the glasses are very cheap. The problem is that you lose colors in the image. It simply looks bad.

Polarizing 3D glasses is a better technique uses polarized lenses. Again, you display two images, each using a different polarization, and the glasses filter the image for each eye. This technique (used in Disney World and Universal Studios for example, using two synchronized projectors) allow full-color viewing. But it also have quality issues, it can cause eye strain, headaches and nausea, tilting your head may ruin the 3D effect, and it can costly (you need two projectors. On LCD displays, it’s difficult to do this because LCDs already use polarization to view a normal image).
Active 3D glasses

XpanD Active-Shutter LCD glasses are the new breed of 3D TVs and projectors make use of a simple idea – you display images for the left and right eye alternatively – once the image for the left eye, and once for the right. Now all you have to do is wear glasses that block each eye in sync with the display, and you get 3D. Active Shutter 3D glasses main advantage is that the image looks great – just as they look in 2D on the same display. If you refresh the screen fast enough, it also feels good and there are much less headaches and nausea than with polarizing glasses.

Active-3D is very costly, though. The display must refresh the screen fast enough – at least 60Hz for each eye, which means 120Hz for the display itself. The glasses are also expensive – they have to include 2 LCDs, and batteries. And you also has to synchronize the display to the glasses (usually using Infra-red).

In short – you must get a new TV (or projector) that supports active-shutter glasses. It is projected that within a few years, most TVs on the market will support this new 3D technology, and most TV makers (including Sony, Samsung, LG, Toshiba, JVC and Panasonic) have already announced (or are selling) high-end models that are 3D-enabled.
No-glasses 3D (Auto-Stereoscopic)

Auto-Stereoscopic displays display different pixels to each eye, using optics (lenses or barriers) to direct the correct pixels to each eye. The nice part is that you don’t need to wear any glasses to experience the 3D image! But there are many problems with these new kinds of technologies. Basically there is just one location you can be in order to view the 3D correctly (just one viewpoint). You can add more viewpoints, but each viewpoint actually requires two ‘dedicated display’. This means that if you want 10 places from which you can view the 3D, you need to be able to produce 20 sets of displays – that’s a lot of pixels. Another issue is that the display is always in 3D. You can’t view a 2D image. There are some solutions to that, too (for example Sharp is using 2 sets of LCD layers – one for 3D and one for 2D, and you can turn off the one you do not want to use).

These technologies are starting to appear now commercially, but don’t hold your breath for a 3D TV for home use that will not require glasses. It won’t happen in the near future…
Volumetric displays: real 3D images

The most straightforward way to create a 3D display, is to actually creating it in 3D. These are called Volumetric displays, and scientists are working on all sorts of way to create them, which usually involves lasers and rotating or vibrating mirrors. A couple of years ago researchers create a volumetric display using a mirror that is rotating very very fast, and a projector that projects an image on it… this creates a 3D image that you can view all around (360 degrees). There was actually a prototype display built on this principle:
Volumetric display prototype using spinning mirrors and a projector photo
3D Cameras

Obviously, you have to shoot content in 3D if you want to view it in a 3D TV (although there are technology that takes a 2D film and turns it into 3D). Basically you have to shoot the same scene from 2 cameras, and make sure that they are in sync and have a fixed distance between them. Several companies are working on professional 3D movie cameras, that have 2 lenses built-in in the same camera, which obviously makes everything easier.

Back in 2009 Fujifilm released the first ‘home’ digital camera that can shoot photos in 3D. We hope that more companies will follow suit, and shooting 3D photos and movies will be easier and more affordable.