The Trial The Trail: Building a VR Drama

Josephine Anstey, Dave Pape
Department of Media Study, University at Buffalo
Buffalo, New York 14260

Abstract. This paper discusses the production of interactive fictional experiences for immersive virtual reality systems such as the CAVE(r). We are primarily concerned with techniques for setting the user at the center of these works, so that his or her actions and emotions become part of the story that is unfolding. As we build these interactive fictions, we borrow heavily from the field of dramatic practice.

1 Introduction

In this paper we discuss the process of planning The Trial The Trail, an interactive drama for immersive, projection-based, virtual reality systems such as the CAVE(r). Our planning stage has been heavily influenced by the lessons we learnt building a previous project The Thing Growing, which was designed to engage the user in an emotional relationship with a computer-controlled character in the context of a fictional narrative. The Thing Growing was built at the Electronic Visualization Laboratory between 1997 - 2000. Work continued on a Japanese version in 2001 at the University at Buffalo with the help of NTT’s InterCommunication Center. The Thing Growing has shown as a work in progress and a finished piece in the US, Europe and Japan.

Our basic research question is, how can VR fiction become as compelling a forum for story-telling as traditional media? In novels, radio, theater, film or television, readers/listeners/viewers identify with a protagonist and in so doing rehearse and explore their own reactions to condensed dilemmas of a political, social, ethical, and/or emotional nature.[11] Computer-based fiction should be able to remove the middleman, make the interactor the protagonist, and allow that exploration to be direct. This paper begins with a brief description of the VR systems we use and why we think immersive VR is an exceptional platform for the creation of first person experiences. Then we discuss our organizing principles and hypotheses at the start of The Thing Growing project and describe how those initial ideas evolved during its production. Finally we explain how we are applying these lessons to the The Trial, The Trail; and sketch out the new hypotheses we are bringing to that project.

2. Projection-Based Immersive VR

A CAVE [3] is a room-sized, virtual reality theater. Computer generated images are rear-projected onto the walls and front-projected onto the floor. A tracking sensor on the head allows the system to correctly calculate the 3D perspective of the real-time graphics as the user looks about and moves through the virtual environment. To create a 3D stereo effect the user wears active or passive 3D glasses, the graphics system creates an image for each eye, the user's brain fuses the two images. The user carries a wand with a joystick for navigation and buttons which can be programmed for interaction. This wand is also tracked so that the system knows where the user's hand is. In a CAVE interactors can see their own bodies in the virtual world, and therefore immediately and automatically assess their own size relative to virtual objects. There are very few CAVEs open to the public for the exhibition of art work since they are expensive and have a large architectural footprint. Therefore we also show our VR drama on smaller, lower-cost, one screen systems.[13] We prefer to use a system which has a tracking sensor for both of the user's hands.

Immersive VR systems make possible not only first person perspective, but first person experience, and, we believe, can offer a uniquely powerful interactive fiction medium. The user is inside the virtual world with the other characters, rather than in the rather God-like position of viewing tiny characters on a monitor and manipulating both them and an avatar of him/her self. This fits precisely with the kind of drama we are interested in producing where the user is central to a story which she does not control - like any traditional protagonist she must react to unfolding events. The other characters are not only peers of the user in terms of size but have a physical presence, they approach the user, they can get too close, they can invade her personal space, they can run away, they have body language.

3 The Thing Growing: Organizing Principles

When we started to plan The Thing Growing, in the Fall of 1997, we had four organizing principles. One - the specific story we wanted to tell would be the main driver for our working process. Two - work on interactivity would take precedence over the creation of complex graphics. Three - we would build around a very simple core narrative. Four - we would create a virtual character to act opposite the user to draw her into the story.

3.1 Specific Story

Anstey'sbackground is in prose fiction and experimental video art. She believed that VR was a medium which could better communicate a specific story idea she had found difficult to write in a traditional prose form. The essence of this story was to recreate the experience of being in a dysfunctionally claustrophobicrelationship; the emotional twists as the "other" is by turns loved and hated, experienced now as separate and interesting, now as unbearably intrusive; a bad relationship which is none the less compelling and hard to escape. Our goal was to make the user one of the two main protagonists of our story, so she would live out this relationship (albeit phantasmagorically) flavoring it with her own emotional patterns. So by “story” we mean a simulation with a purpose and a narrative arc which the user takes part in. We do not give the user information as in a role playing game about their character, motivations and goals, rather they enter our story space as themselves. However, the narrative, the graphics, the interaction, the sound, the computer-controlled characters are all constructed with a view to stimulating the user to experience the dilemmas inherent in this specific story.

3.2 Interactivity trumps Beauty

Given finite resources, we decided to focus our time and attention on the responsiveness and interactivity of our VR environment not on gorgeous, highly resolved 3D graphics. Many of the VR art works we had seen firmly prioritized the visual. Interactive elements were added last, and typically an expert user was necessary to explain the interactivity and even the best way of navigating the environment. This would not work for the kind of interactive drama we were planning. Instead the environment and the computer characters needed to be smart enough to explain the interactivity and guide the user through the experience themselves.

We also hypothesized that high visual resolution of landscapes, buildings and characters is only one axis of the sense of reality that a virtual environment can provide. By reality we mean the extent to which the user feels present and immersed in the virtual environment, two qualities that have been studied by Mel Slater et al.[19] Writing about Disney's VR project Aladdin, Pausch et al say "We suspect that the limited believability of our first system's characters is due to low fidelity."[14] In contrast we decided to construct simply rendered characters and environments which would respond to the user intelligently. The choice here is to make the objects in the space read symbolically rather than literally and is analogous to a simple set design in a theater. Scott McCloud suggests that viewers can more easily identify with simply drawn, iconic, cartoon characters.[10] In the same way we believed that a simply designed main character would be able to stand in for any significant other in the user's life (spouse, sibling, parent, child) so that elements of the user's own relationships would seep into the emotional narrative.

3.3 Simple Core Narrative

One much hyped strength of computer-mediated fiction is the possibility of non-linear narrative and multi-form plots. Janet Murray suggests digital techniques could present kaleidoscopic stories revealed from multiple points of view which, coupled with the encyclopedic capacity of computers, would offer the possibility of the creation of almost infinitely rich and detailed narrative epics.[12] Brenda Laurel's describes an Interactive Fantasy System, in which she proposes using intelligent agents to develop and change the plot on the fly.[6] However, it has proved very hard to create such ambitious works in practice.

We decided to build our interactive drama around a very simple core narrative for two reasons. First, we considered that it was too time-consuming and expensive to create a multiplicity of complex scenes in VR. Talin also makes this point in relation to the construction of "high-production-value graphical adventure" games.[20] Second we did not find elaborate hypertext stories cathartic or dramatically satisfying, and that was the kind of experience we were pursuing. Our story only involved one character and a few sets. We thought that this kind of simplicity was in our favor; that making one believable, responsive character that really engaged the user was challenge enough.

3.4 Virtual Character

Our prime objective was to make the user a protagonist in our virtual drama. Given the nature of our basic story - a love-story of sorts - we needed a computer-controlled character to act as the other. Although we knew several research teams had focused on using intelligent agents for interactive, narrative experiences, including the Virtual Theater for Children [5], theAlive Project [1], theOz Project [7], we felt that they addressed the user too cerebrally to be useful role models for our application. In these projects the user is invited to create fiction with the agents; to direct them; or to marvel at simulations of autonomy and personality. We wanted to provoke a more basic emotional response in the user. Our approach to building the intelligence for the application was to focus on the agent's character and responsiveness to the user rather than on its own set of intelligent and autonomous behaviors. It would be an actor/agent giving a dramatic performance which would include simulating emotions and following the arc of our narrative. Our hypothesis was that that simulated emotional behavior would stimulate emotional responses in the user.

4. The Thing Growing - Evolution of Principles through Practice

This section will detail how the specific story idea drove the production process in terms of the narrative, the visual design, and the animation and intelligence of the computer-controlled character. We will describe how some of our organizing principles were refined and extended by our practice, and how we discovered the importance of an iterative design process and Wizard of Oz experiments.

4.1 Narrative

The core purpose of The Thing Growing was to recreate for the user the sense of being trapped in a claustrophobic relationship with a clinging and demanding other, the Thing. A difficult relationship, by definition, is composed of both positive and negative emotions, and so we needed to stimulate ambivalent feelings about the Thing. We evolved the narrative structure as our basic strategy for doing so. In order to keep authorial control over pacing and the creation of surprise we based The Thing Growing on a dramatic arc with a distinct beginning, middle and end.[4] Act One introduces the environment and characters (computer-controlled and human), and an inciting incident reveals the protagonists' dilemmas or goals. The inciting incident is the Thing being released from a box by the user and instantly falling in love with the user. The user's first experience of the Thing is that it is charming and loving. Act Two contains the Thing's clinging and dysfunctional attempts to manipulate the user into loving it - and is designed around a central trope - the Thing teaching the user a dance. Users become aware of, and react to, the growing unpleasantness of the Thing. Act Three contains the resolution. First, a new element is added, to wit the Thing's aggressive cousins who disapprove of the relationship between the user and the Thing and become a common enemy that theyhave to faceThe Thing is by turns brave and helpful, and in need of protection. The resolution is the user's decision whether or not to terminate the relationship (and the Thing.) 

Each act was also subdivided into a beginning, middle and end of its own. This structure provides the user with constantly changing clues and suggestions on how to act and incidents to act against, and imparts a forward thrust to the experience which is essential to maintaining interest. Each scene in each act also has an underlying goal to stimulate an emotion or sequence of emotions in the user. Surprises (the Thing jumping out the box); reversals of fortune (the Thing's fawning love turning to annoyance); and events with unanticipated results (killing cousins too efficiently leads the Thing to chastise the user); are all designed to elicit the current emotional goal. 

Fig. 1. A user dances with Thing in the CAVE - Avatar of User with Thing and cousins

We believe that this adaptation of a traditional dramatic arc worked very well. Although the overall experience was quite tightly structured, at a finer level of granularity there were multiple ways in which the user could interact with intelligent agents or smart environments within each scene. The agents or environments had their own goals within the dramatic structure. It quickly became apparent that the narrative was the most useful constraint not only on the user's actions [6] but on the necessary responses that we had to program into the Thing.Within this framework the user/protagonist had a certain amount of agency but as in life, as in novels, she had nothing like total power or absolute free will. 

In the course of our production we rediscovered a problem described by Kidsroom team - narrative-based, interactive experiences should avoid dead ends where the user can get stuck unless she performs a specific action.[2] Therefore our narrative moved from scene to scene either as a result of the user's actions, or because a time-limit had been passed. In VR it is trivial to move the user from one virtual scene to another - the equivalent of cutting from scene to scene in a film. However it can also be completely disorienting since there are no established VR conventions for understanding what has happened. Working on this piece we came to believe that in immersive VR the scene transitions have to be fore-shadowed in order to work - the user has to be aware that a change is going to happen, and have an idea why its going to happen. 

4.2 Visuals, Animation and Reality

"Whether something seems real or not doesn't just depend on whether it simulates the way light bounces around in a room, but whether it simulates correctly how the object and the environment respond to our activities." [17]

As we built The Thing Growing we focused on aspects of realism distinct from photo-realism. Our graphics were non-photorealistic: the simple visuals were designed to augment the psychological and emotional ambiance we were establishing. However, we did make the movement of our characters life-like by animating them with motion capture techniques. And we concentrated on the development of intelligent characters and smart environments that would respond appropriately (in the context of the narrative) to the user.

The Thing Growing's visual design was childlike but psychologically responsive. At the beginning the visuals lull the user into a false sense of security and agency; the sun is shining as she passes through a cartoon world; doors open as she approaches; she is given a key to open the box which contains the Thing and sets the narrative going. Later, as the story gets darker, the innocent visuals contrast with an increasingly hostile, world: the Thing hounds and bullies the user; rocks move and morph to trap her. Finally the user is dropped into a violent red environment, where she must make life or death decisions. But this environment is a deliberate caricature of a shooting game environment and here the user is prodded to deliberate on her actions whether they be aggressive or passive.

The Thing itselfis a collection of unconnected pyramids representing its head, body, arms and tail. It is animated both by motion capture and procedurally. For the motion capture, we used the CAVE's tracking sensors to collect position and orientation data that was fed to the models of the Thing's body parts - the result was a strong illusion of a dragon-like being with life-like gestures. The procedural animation determined where the Thing stood or moved. It could move relative to the user, staying just in front, at the side, right in the user's face. It could move autonomously - ignoring the user, running away. The Thing's gestures were captured in relation to its dialog so that the Thing's body language also reflects its mood and the meaning of the phrase. When the application is running the Thing's brain picks the next appropriate animated movement and phrase. We were very pleased with the results of this animation system. It was flexible, fast and seamless as the character moved between different actions.

4.3 Computer Controlled Character

The Thing was our first attempt at creating a believably responsive virtual agent. Janet Murray mentions three computer-based characters that have operated successfully because they acted within specific contexts, Eliza; the chatterbot Julia; and Parry the paranoid agent.[12] In each case the limitations of the character in the Turing sense are masked by its personality traits and establishing context. Users tend to explain inconsistencies by forming a complex, human-mind-based model of the character's interior processes. We believe that the Thing was successful because it operated in a very similar way. The narrative provided a context within which its actions and goals were clear. Its willful and somewhat psychotic personality maintained a recognizable emotional logic. The Thing is dominating because of the exigencies of the story and constantly tells the user what to do. Although the user may become annoyed at the bossiness of the Thing, she is less likely to attribute her frustration to the limits of the program than to the character she is dealing with.

The Thing perceives the interactor through the tracking system. The dancing trope maximizes the meaning of the information that we receive from this system and is also a crucial element in building the emotional relationship between interactor and agent. The interactor is taught dance steps by the agent. As he dances we take information from the tracking sensors and joystick, interpret them and program the Thing with a logical response in the context of the story. The Thing praises or criticizes dance steps appropriately; knows when to teach a new step and when to go over one that the interactor is doing poorly; runs after an interactor that is trying to escape the interminable dancing; remonstrates with an interactor who refuses to move. That the Thing knows what they are doing makes it easier for users to believe that it is sentient.

The Thing has a human voice; we recorded about 500 short phrases that encompass the narrative possibilities of our story. Many of the phrases are similar in meaning but phrased slightly differently so that we avoided a robotic repetition. It has four moods, manic, happy, sad and mad. Many of the pre-recorded phrases were recorded in each of the moods, so it can be saying the same thing but the emotional value of the interaction will be different. The Thing, as a coherent actor, is created by a passage through narrative fragments, where each fragment is an action made up of a short spoken phrase and an accompanying animation. The effectiveness of the character depends on seamless and appropriate transitions from fragment to fragment. Visually this is done by interpolating between one motion captured sequence and the next. To make the character flexible and able to respond in a timely manner, each of its actions lasts from one to five seconds, and it also is possible for the Thing to interrupt an action abruptly when necessary. Logically and conceptually, a finite hierarchical state machine architecture, based on the narrative arc, is used to trigger events in the world, and the Thing's actions. The selection of the appropriate action is informed and determined by the passing of time, the user's location and interactions, and the Thing's internal logic and moods.

Although the Thing speaks to the user, it does not hear him or respond to his utterances. The simple graphics and fact the Thing was not human in form, did not encourage the expectation that it could hear. The Thing's high-handed personality also helped to cover this technical limitation. Nevertheless, many people did talk back to it and serendipitously logical conversations occurred. Our observations suggest that people do respond emotionally as they try to cope with the virtual creature and that they internalize a psychological model of it. One interactor who believed she was dancing well, suggested the Thing was just trying to tease her when it criticized; another described the creature getting in "my face, making me mad;" another said the character was very manipulative. [21]

4.4 Iteration & Testing

As we built our application it became clear that an iterative creative process is vital for the construction of virtual drama. Creating successive drafts of the project and testing it with a variety of users not only honed the interface, but clarified how well the story was being communicated. By chance during the building of The Thing Growing, we created a Wizard of Oz stage in which a networked human augmented the Thing's intelligence. Data gathered from this stage enabled us to pinpoint much more clearly how we needed to build that intelligence.

Fig. 2. First Version of Thing with Avatar of User – Final Version of Thing

During production problems emerged with every aspect of the VR application, graphics, interface, concept. A graphics problem was that some people were unable to put together a coherent picture of the computer-character from the its abstract motion-captured body parts. Out-lining each body part in black and adding eyes to more clearly identify the head was our solution. An interface problem was to clearly indicate to people that they had to move their bodies during the dancing sequence, and that the Thing "knew" what they were doing. Some users, especially expert users, thought that they should use the joystick and buttons to dance rather than their own tracked arms and head. To solve this we inserted some new behavior at the very beginning of the dance sequence. The Thing announces that it will show the user how to dance, then says that they must "loosen up a little first." It instructs the user to wave her arms above her head. The program performs checks to make sure they are using their body rather than the joystick controls to move, before going on to the first dance step.

The main concept problem was to make the user's feelings towards the Thing ambivalent enough - in our first version everyone disliked the Thing and shot it when they got the chance. We added a section where the Thing copies the user's movements, clarified the evilness of the Thing's cousins by having them tear off one of the Thing's arms, and changed all the Thing's dialog in the final act to emphasize its need of protection. In 2001 we made a Japanese language version of The Thing Growing, and also incorporated some new code that logged the user's behavior. This code enabled us to gather statisticsduring the period of time that the application showed at the ICC museum in Tokyo in the summer of 2001. During this show, half the people killed the Thing, and half let it live..

5 The Trial The Trail - Planning

The Thing Growing is a cultural work not a scientific experiment. It shows in museums, galleries, festivals and conferences, and our assessment of the project is anecdotal. However, from observation and feedback we believe it was successful and we are basing our next project The Trial The Trail on lessons learnt from The Thing Growing.

5.1 Specific Story

The Trial The Trailwill be an interactive quest; an absurdist fantasy based on elements from Tarkovsky's Stalker, Lewis Carrol's Alice Through the Looking Glass, and Monty Python and the Holy Grail. The user will embark on a guided journey through this warped yet familiar landscape. As she proceeds her actions and interactions will be logged, interpreted psychologically, and used to determine the outcome of her quest. The Thing Growing's story line determined that we should build the kind of agent whose neurotic ticks tended to obscure the limitations of its intelligence, not vice versa. The desire to manipulate the user emotionally led us to use the kind of narrative structure that is more typical of drama. InThe Trial The Trail, we are again trying to lead the user on an emotional and rather introspective journey. The psychological terrain that we are interested in exploring encompasses ambivalent notions that our actions and choices lead where we want them to; questions of what is more important - the journey or the goal; issues of what our desires are and whether they can be gained. As we go forward we will apply the lessons and procedures that evolved during the production of The Thing Growing, while remainingready to throw out any method or assumption that does not work in our quest to create an interactive quest.

We believe that the familiarity of the quest tradition will be a crucial element that will help us build a more complex narrative this time around. BecauseVR story-telling does not yet have a lexicon of common practices that make people at home in the environment, several theorists and practitioners suggest the most enjoyable experience comes when the user has something familiar to hang onto, some clue about how the she should proceed to interact with this new environment. Brenda Laurel was the first to suggest "A system in which people are encouraged to do whatever they want will probably not produce pleasant experiences."[6] Randy Pausch et al. suggest that users need a background story and a goal.[14] Janet Murray says, "Highly ritualized interactions can actually increase the participant's freedom, rather than limiting it, by offering them more choices for coherent action."[12] In The Thing Growing we used the dominant character of the Thing to explain the story and what the user should be doing. The Trial The Trail also has a guide character but we are planning to rely more heavily on a familiar story to make the user feel comfortable and empowered to act within the virtual environment. Our assumption is that these initial familiar surrounding structures will enable the user to cope with and enjoy the interactive drama more intensely, and then keep the user with us when the experience splits from the familiar.

5.2 Narrative.

Michael Mateas argues that an interactive drama needs at least two computer-controlled characters, besides the user, to carry the story out of narrative cul-de-sacs.[8] Although we were very pleased with The Thing Growing, we realize that it relied heavily on one main computer-controlled character, that character's speech, and one main activity between users and that character, dancing. This narrow focus reflected the particular story we were telling but also ran a higher risk of seeming rather sterile and limited to some users. The Trial The Trail's narrative contains more scenes, more interactive tropes, and more characters, but will still be structured around a core narrative arc. 

Our planning stage has included working on two story-boards of this narrative. In the current storyboard [16], the opening scene takes place at a barred gate into the zone. In the inciting incident a virtual guide reveals the quest - if the user reaches the top of a grain elevator in the middle of the zone, she will get her heart's desire. The journey is difficult - distractions and dangers will meet them. After this explanation the guide helps the user get into the zone.

The middle section contains a series of interactive challenges which encourage interactivity and emotional response. These include a series of distractions which will encourage the user to dally instead of getting on with her quest; a reed bed which responds physically and musically to the user's movements; a pool which she can ripple; and a fire/particle system which she can sculpt. There will also be test-like challenges which include the opportunity to rescue a helpless creature from evil by fighting a sentient morning-star; and the problem of what to do with this creature who, once rescued, attaches itself clingingly to the interactor, preventing her from going further, and reaching her goal. 

These challenges are designed as psychological traps, and we are interested in seeing if we can design traps that will work for both the cynical and the romantic user. For example the romantic may be very excited to rescue a helpless victim, but then finds herself mocked for her attempt. The cynic may find her cynicism amplified by this trap, but may then be puzzled when the guide reappears and insists that the rescue attempt was simply illusory, a figment of the user's imagination stimulated by the zone. We also hope to draw the cynic towards a more optimistic naivety as she plays with the distractions.

Fig. 3. First Version of The Trial The Trail with higher resolution images

The end of the drama is determined by a psychological profile of the user gathered from her responses to the challenges. This will determine if her heart's desire is Life or Death. Both choices are problematic. In the Life ending the user flies over the entire environment - all the virtual characters gather around her praising her. The Life ending is both desire and temptation. You, but only you, are alive. Everything else is subservient to you. You are master of all. Nothing opposes your will, for there is nothing else. In the Death ending darkness falls, glowing wil “o” the wisps float downwards to be extinguished by contact with the floor or the user. The Death ending is both desire and temptation. You no longer have to struggle and fight. Everything is peaceful. You don't have to do or say anything. You no longer exist.

Although we will maintain a narrative arc with a first act that introduces the goal, a second that introduces hindrances for the user trying to reach the goal, and a third that ends the quest, we are planning to be more flexible with the order of the scenes in the second and longest section. We want the user to feel as free as possible to move through the virtual environment and choose what she does next. In order to maintain a rising curve of interest, a typical dramatic strategy is to put the protagonist in situations which are progressively riskier and riskier, with lulls between each crisis, until the final moment of truth arrives.[11] This strategy is based on having linear control over time. The author of interactive environments does not have such control, VR is an omni-directional space. Yet allowing the user to wander about too freely, negotiating the scenes in the middle section in any order, may mean that we lose intensity and introduce boredom. At this stage in our planning we intend to address this in two ways. At our most manipulative we will be able to make our virtual environment very foggy - therefore if the user doesn't wander in the right direction - the fog can roll in and we can slap the next interactive situation down right in front of her. At our most subtle we will there will be multiple ways of entering each scene which take account of which scenes the user has already encountered. 

5.3 Computer Controlled Characters

As already mentioned we are designing The Trial The Trail so that it does not rely so heavily on one virtual character to do the work both of "telling" the story and stimulating the user's actions and emotions. Instead these disparate tasks are spread over a number of agents and smart environments. An intense and rather mournful guide figure will take on the role of explicating the story, explaining the interface, setting the user's goals and intervening to move the drama from one scene to another. The work of stimulating the user emotionally will be taken on by the agent she must fight, the agent who becomes the too-clinging creature she has saved, and the distracting, smart elements in the environment as a whole. Other agents will be used to set scenes and moods.

As in The Thing Growing the job of the agents will be to interface with the user and assemble fragments of the narrative into a compelling dramatic form. However, in order to handle multiple agents and a more complex narrative we decided that we needed a more advanced agent architecture. We have started to collaborate with AI researchers on the creation of an actor/improviser agents for The Trial The Trail. The agent will be built using SNePS -- The Semantic Network Processing System - a knowledge representation/reasoning system developed by Stuart Shapiro and the SNePS Research Group at the University at Buffalo.[18] The long-term goal of the group is the design and construction of natural-language-using computerized cognitive agents. Our actor/improviser agents must guide the human user through virtual locations, moral choices, and emotional states - although the guidance may not be apparent. The agents must react believably, and in character. To this end they will be designed to understand natural language, to dissemble if they do not understand, and to talk back to the interactor.

The SNePS agent's knowledge base will be the narrative, in the most expansive sense of that word, comprising information about the VE, and all that it needs to guide/persuade/manipulate the user to see, hear, feel and do in order for her to experienced our drama. The agent will also have a model of the user, that will enable it to interpret the user's emotions and psychological state at any given point in the narrative, and more effectively mold the narrative to give the user a unique experience. Part of the research process is to determine whether we want one artificial intelligence which then provides the reasoning for each embodied agent and for the entire virtual environment, or whether each embodied actor or part of a “smart environment” has its own intelligence aspect. In the latter case, we may also need a dis-embodied director agent to oversee the entire development of the plot. 

Additionally we will need to add speech recognition software to the VR display system. Although the primary mode for the agent will be to understand the speech input and reason from it, it will also have methods for dealing with speech that it does not understand. The creation of a strong personality for each character will help determine character- and context-appropriate ways for covering lack of understanding. Current speech generating software does not do a good job of creating an appropriate emotional tone [12] - so the agent will not use synthesized speech software, instead just as for The Thing Growing it will pick an appropriate verbal response from a pre-recorded library. Our planning schedule includes Wizard of Oz tests with users in June 2003. In these tests a human takes the place of the intelligent agent and we can more nearly pinpoint how we need to build the agent.

5.4 Visuals

We started the planning process of The Trial The Trail in Fall 2001, made a basic story-board, created some 3D models, assembled a preliminary VR draft of the world, and started work on the behavior of the fighting morningstar. In May 2002 we showed a small fragment as a work in progress and began to realize that our first draft was not communicating our story. It was too literal and game-like; a DisneyQuest! Whereas we want to provoke dis-ease about the whole question of quests and desire. The rest of 2002 was dedicated to thinking more carefully about the series of moods that the want to maneuver our users through, and the kinds of graphics that may do this. 

The first models that were built were quite highly resolved, with photorealistic texture mapping. The walls of the zone were stone, its entrance was an elaborate arch. The first story-board called for a castle in the center which would contain the room where the user gets their heart's desire. The connotations of these graphics were closer to a Hollywood version of the Arthurian legend rather than Tarkovskyian angst or Monty Pythonian absurdity. In our second story-board we have returned to The Thing Growing principle of making simple models that can be read symbolically. Although we do want to trap the user into reacting in a very direct way to our story - to let themselves be carried away. We also want the visuals to recall the fact that this is a constructed reality, that their emotions are like-wise constructed, that all is suspect and should be analyzed by the brain. In dramatic terms we are interested in allowing the user to maintain a certain Brechtian distance from our plot - to collude in the springing of the traps, but to recognize the traps and the underlying story themes.


Fig.4. Second Storyboard -  Simpler Graphics of Factory, Tree, Interactive Pools

Our animation methods for the virtual actors should remain fairly similar to The Thing Growing. However, our first models of the other characters for The Trial The Trail were also problematic. We wanted them to be more humanoid than the Thing, but they became too stiff and too defined. In another project, PAAPAB, we developed characters clothed with a cloth-like material that will swirl as we use the motion capture to animate them. We have decided to use this as the basis for the new characters.At present we are still debating what to do with the faces - our current plan is to give each character a set of stylized masks which reflect their emotions, and change as their emotions change. We are also developing some procedural animation techniques, for example in a scene where a sentient morningstar fights the user and must be able to respond, dodging, feinting and attacking in real-time.

6 Conclusion

As we continue with the production of The Trial The Trail our four organizing principles remain intact, although they have evolved. We continue to maintain that the specific content of a VR drama must be the major driver of the production process, and doubt the efficacy of elaborate theories and methods that attempt to build generic actors, or story-generating systems. Nevertheless we believe that certain conceptual and technical tools can support that process.

Our principle of using a core narrative remains - the narrative should be broken into discreet scenes. Although some scenes may be re-ordered on the fly there should be a dramatic arc that provides the story with a beginning, middle and end. Attention must be paid to transitions between scenes in order for a coherent story to emerge. We are beginning to hypothesize that there may be two main types of scenes - those that contain the interactive or informational content, and those that are transitional. The combination of the two will allow for the greatest flexibility. We have also learnt that we must prepare and test iterative drafts of the project in order to hone the interface and content.

We continue to believe that intelligent agents are a vital part of the process. Our interactive drama is intended to create an unfolding story around the user which the agents establish, populate, maintain and influence. The scenes of the narrative can be broken down into small fragments, and the virtual actors reassemble the fragments to accomplish this goal. Building better agents, building agents that can take in human speech as input, has a high priority for our current project. We have also learnt that building a Wizard of Oz stage into the production is vital for the efficient building of intelligent agents.

Finally, we are still inclined towards simple non-photorealistic visuals, and to prioritize interaction over graphics. Increasingly photorealistic graphics signify the games industry and trap the work in connotations that effect the content. We want the graphics to serve the story we want to tell, not to trip unwanted metaphorical associations.


We would like to acknowledge the support of the Electronic Visualization Laboratory, University of Illinois at Chicago, the NTT InterCommunication Center, the Department of Media Study and the Department of Computer Science, University at Buffalo. We would like to thank Amit Makwana and Sara Nohejl.


    Blumberg, B., Galyean, T.: Multi-Level Direction of Autonomous Creatures for Real-Time Virtual Environments, In the Proceedings of SIGGRAPH 95, ACM Press, Los Angeles, CA, (1995) 47-54.

    Bobick, A., Intille, S., Davis, J., Baird, F, Pinhanez, C., Campbell, L., Ivanov, Y., Schütte, A., Wilson, A. : The KidsRoom: A Perceptually-Based Interactive and Immersive Story Environment. In Presence Vol. 8, Num. 4. MIT Press, Cambridge MA (1999) 367-391

    Cruz-Neira, C., Sandin, D., DeFanti, T. A.: Surround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVE,Computer Graphics Vol. 27, Press (1993), 135-142.

    Glassner, A., Wong, C.: Fiction 2000:Technology, Tradition, and the Essence of Story. In SIGGRAPH 99 Conference Abstracts & Applications, ACM Press, Los Angeles, CA (1999) 161

    Hayes Roth, B., Brownston, L., Sincoff,E.: Directed Improvisation by Computer Characters, Stanford Knowledge Systems Laboratory Report KSL-95-04 (1995)

    Laurel, B.: Computers as Theater. Addison-Wesley, New York (1993)

    Mateas, M.: An Oz-Centric Review of Interactive Drama and Believable Agents, Technical Report CMU-CS-97-156, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA. (1997)

    Mateas, M.: A Preliminary Poetics for Interactive Drama and Games. In SIGGRAPH 2001 Electronic Art and Animation Catalog, ACM Press, Los Angeles, CA (1999) 51-58.

    Mateas, M., Stern, A.: Towards Building a Fully-Realized Interactive Drama, The Fourth International Digital Arts and Culture Conference, Providence, Rhode Island,(2001) pages?

    McCloud, S.: Understanding Comics, Tundra Publishing, Northampton, MA, (1993)

    Mckee, R.: Story: Substance, Structure, Style, and the Principles of Screenwriting. Harper Collins, New York (1997)

    Murray, J.: Hamlet on the Holodeck, The Future of Narrative in CyberSpace. Simon & Schuster, New York (1997)

    Pape, D., Anstey, J., Dawe, G.: A Low-Cost Projection Based Virtual Reality Display, In the Proceedings of SPIE, Vol. 4660, SPIE - The International Society for Optical Engineering, San Jose, CA, (2002) 483-491

    Pausch R., Snoddy S., Taylor R., Watson S., Haseltine E.: Disney's Aladdin: First Steps Toward Storytelling in Virtual Reality. In: Dodsworth Jr., C. (ed): Digital Illusion. Addison Wesley, New York (1998) 357-372

    Roussou, M.: Incorporating Immersive Projection-based Virtual Reality in Public Spaces. In the Proceedings of 3rd International Immersive Projection Technology Workshop, Vol. T52. Springer Verlag, Berlin, Heidelberg, New York ( 1999) 33-39

    Storyboard at:

    Sandin, D.: Digital Illusion, Virtual Reality and Cinema. In: Dodsworth Jr., C. (ed): Digital Illusion. Addison Wesley, New York (1998) 3-26

    Shapiro, S. C. and the SNePS Implementation Group: SNePS 2.6 User’s Manual, Department of Computer Science and Engineering, University at Buffalo, (2002)

    Slater, M., Wilbur. S.: A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence Vol 6,Num 6. MIT Press, Cambridge MA (1997) 603 - 616

    Talin.: Real Interactivity in Interactive Entertainment. In: Dodsworth Jr., C. (ed): Digital Illusion. Addison Wesley, New York (1998) 151 - 159

    Video clips at: