Interactive 3D: Using It, Designing It, and Teaching It

After several false starts, interactive, real-time 3D graphics (IRT3D) is finally starting to happen. Advances in the processing power and graphics capability of inexpensive desktop computers have created the first generation of consumer-grade systems capable of generating three-dimensional graphics of satisfying speed and quality. Now the user can truly interact with these advanced representations in real time, creating powerful new possibilities for the visualization of digital information.

While the first beneficiaries of this new capability have been those who play games on their computers, the potential of IRT3D to transform other aspects of human interaction with computers is being demonstrated more and more. From eCommerce to educational software, the rate at which IRT3D continues to improve is matched only by the demand of the market for real-time content. Media trends forecaster Jon Peddie & Associates estimates that by the year 2007 there will be more than one million 3D Web sites and over half a billion 3D-enabled Web browsers.

It isn't an overstatement to suggest that we may be watching an evolutionary leap in the progression of how computer images are created. And all of this is important to us as designers and educators because it not only presents us with exciting new methods to create expressions of information, but also with new skills that we must challenge our students to investigate and master.

Defining IRT3D
Unlike traditional sprite graphics, such as bitmapped images, 3D graphics are based on mathematical descriptions of the surfaces of the objects that they depict. Specifically, the surfaces are described in terms of polygonal geometry, which "models" or approximates the surface as a blanket of polygons. While bitmapped sprites can generally be rendered at a much faster rate, the advantage of a geometric image is that the perspective from which it is viewed can infinitely be changed.

Animation, on a computer or otherwise, consists of a series of slightly transformed images played in sequence fast enough that they appear to be a single, fluidly moving image. This is possible by virtue of a phenomenon called "persistence of vision" by which physiological components of the brain will "blur" successive images, so long as the images occur at or beyond the threshold rate of around eight frames per second (fps). In practice though, ideal results are realized at around 24 fps. Thus, 3D animation consists of a series of slightly transformed images, modeled from geometry, observed at a sufficient rate to look like real motion.

When the various images that make up an animated sequence are created in advance of their being viewed, they are said to be "pre-rendered." These often take the form of digital videos, which typically feature extreme detail.

Conversely, animated images rendered at the exact moment that they are viewed are called "real time." The trade-off of generating images in real time at rates sufficient for animation is generally the relative lack of detail that they can feature.

Animation that plays in a predefined sequence is said to be "linear." Pre-rendered 3D is generally linear. However, when we combine the ability of a computer to create 3D graphics in real time with its ability to be programmed, we have the ability to create 3D graphics that interact with the user in a responsive and logic-driven manner.

Using IRT3D
In looking at the actual use of IRT3D as a medium, there are six categories that hold particular promise. It is particularly useful to bear these categories and their distinctions in mind when beginning an exploration of IRT3D design methods:

1. Games: This category consists of entertaining content that involves the activity of play within a goal-based, competitive, and challenging structure. IRT3D has been used in games for some time already, to great effect.

2. Non-Game Entertainment: Active or passive entertainment experiences that are not goal-oriented. The most common example would be the emerging medium of the "interactive narrative."

3. Informative: Activities that are driven by specific educational goals. Content in this category is authored in a manner that teaches rather than merely delivers knowledge. Examples are educational software lessons and interactive instructions.

4. Reference: Content that is categorized or indexed in order to accommodate its retrieval. Unlike "informative" content, reference material is presented in whole based on query. Examples include maps, directories, and simulations.

5. Persuasive: Content that seeks to influence the user, such as to buy something or to assimilate a point of view. Examples include commercial and political content.

6. Aesthetic: This category includes artistic or experimental expression.

Most of my work with IRT3D in the past year has been to develop a formalized approach to teaching it. There is little foundational pedagogy in this area other than what can be synthesized from the various disciplines that it covers. Still, I have pieced together and tested in the classroom "laboratory" what seems to be a promising starting point. Using a series of key concepts as the framework for a quarter-long (three-month) studio class, I have been able to achieve significant results from my students (see "IRT3D: A Template Curriculum").

IRT3D: A Template Curriculum

Workflow and Tools
IRT3D authoring consists of two primary components: asset creation and programming, in that order. The first step in comprehending the authoring process is to understand this workflow, the various tools available, their roles at each step, and the specific tools being used.

Modeling and Texturing for IRT3D
The overriding consideration in creating assets for IRT3D is to craft them to accommodate the intense performance needs of real-time 3D rendering. Key among these are low-polygon modeling techniques, as well as desktop computer-friendly textures.

Transforms and Transformations
The ability to manipulate the position, orientation, and scale (PSR) of an object in 3D space is the key to being able to program it to do practically anything of use. The transform object is an abstraction of the complex matrix math that remembers the PSR of a 3D object, while the transformations are functions that allow the transform to be manipulated transparently. Understanding what each is and how it works is the crucial first step to learning how to program IRT3D.

2D to 3D Interface Approaches
Creating visual and conceptual metaphors for the user to interact with objects in 3D space is a field of study in itself. We explore this in terms of both visual and kinetic devices. Visual devices include the various on-screen elements that the user can manipulate to affect the 3D world and the objects in it, such as "dashboard" controls at the bottom of the screen. Kinetic devices include those that involve user interaction with hardware to affect the 3D world, such as mouse gesturing, picking, and keystrokes.

Cameras and Perspective
This introduces the cinematic element. Aside from the technical chores associated with creating and manipulating one or more cameras in 3D space, there is now the design consideration of a psychological perspective.

Lighting
Also a cinematic element, but steeped in technical considerations. Lighting is one of the most taxing features of real-time 3D rendering and must be used judiciously. Still, it remains one of the most important and evocative elements available in creating drama and mood, focusing attention, and revealing/concealing geometry.

Groups and Hierarchies
Grouping allows multiple objects to be "joined" together to transform as one, while hierarchies allow objects to take on a "parent-child" relationship. Both groups and hierarchies have very profound influences on how objects within them will behave. Used strategically, they can facilitate complex control without the need for complex programming, as well as allowing composite animation.

Effects
As more and more 3D functionality finds its way to the graphics processors (GPU), more and more "effects" will become available in real time. Such hardware-based effects offer a powerful means of achieving stunning and seamless visuals without the need for a more powerful central processing unit (CPU). Two common examples are particle systems and fogging, each of which can be used to create dramatic and useful visual effects.

Designing IRT3D
It is just as important to have good authoring tools to create IRT3D content as it is to have computers capable of delivering it. One of the primary reasons IRT3D may well succeed now is that established companies have brought relatively easy and broadly applicable IRT3D tools to market, all of which allow the authoring of IRT3D content, using 3D content from existing tools.

In fact, the production of IRT3D taps the existing knowledgebase and functionality of established tools in each step of the workflow. For instance, a typical workflow might include creating 3D models and scenes in Alias|Wavefront Maya, texturing surfaces using Right Hemisphere's Deep Paint 3D, processing sounds and digital video in Adobe Systems' Premiere, and then authoring interactive 3D with these assets in Macromedia's Director Shockwave Studio.

Remember that the authoring tools at the end of the workflow are key to the process. Increasingly powerful, flexible, functional, multi-platform, object-oriented, and multi-output programs have now made a relatively easy task of what has been the historically elusive task of creating viable IRT3D.

While this new breed of authoring tools has allowed designers to explore new ways of creating 3D visual experiences that look and act more like real life, new design challenges do not end there. One particularly acute example is in the area of Human Computer Interface (HCI) for IRT3D. Traditional computer interface devices such as the mouse, track-pad, and screen are all 2D devices and fall somewhat short as intuitive and effective means of placing the hand of the user in the virtual 3D world of the computer.

While the initial work in IRT3D HCI research was focused on interfacing virtual 3D worlds with 2D devices, it has recently expanded to explore engaging more of the senses, so as to make the interface experience itself more natural and intuitive. One example is the work to create IRT3D interface devices that include haptic response, so that a user not only negotiates virtual 3D objects and space through the device, but feels them as well.

[Editor's note: Marty Plumbo will present a half-day workshop, "The Design and Programming of Interactive 3D for Educators," on Sun., July 27, at Syllabus2003.]

comments powered by Disqus