The Trial The Trail: Building a VR Drama

Josephine Anstey, Dave Pape
Department of Media Study, University at Buffalo
Buffalo, New York 14260

This paper discusses the production of interactive fictional experiences for immersive virtual reality systems. We are primarily concerned with techniques for setting the user at the center of these works, so that his or her actions and emotions become part of the story that is unfolding. As we build these interactive fictions, we borrow heavily from the field of dramatic practice.
In this paper we discuss the process of planning The Trial The Trail, an interactive drama for immersive, projection-based, virtual reality systems. Our planning stage has been heavily influenced by the lessons we learnt building a previous project The Thing Growing [1], which was designed to engage the user in an emotional relationship with a computer-controlled character in the context of a fictional narrative. The Thing Growing has shown as a work in progress and a finished piece in the US, Europe and Japan. Our basic research question is, how can VR fiction become as compelling a forum for story-telling as traditional media? In novels, radio, theater, film or television, readers/listeners/viewers identify with a protagonist and in so doing rehearse and explore their own reactions to condensed dilemmas of a political, social, ethical, and/or emotional nature.[11] Computer-based fiction should be able to remove the middleman, make the interactor the protagonist, and allow that exploration to be direct. In this paper we discuss our organizing principles and hypotheses at the start of The Thing Growing project and describe how those initial ideas evolved during its production. Then we explain how we are applying these lessons to the The Trial, The Trail; and sketch out the new ideas we are bringing to that project.

2 The Thing Growing: Organizing Principles
When we started to plan The Thing Growing, in the Fall of 1997, we had four organizing principles. One - the specific story we wanted to tell would be the main driver for our working process. Two - we would build around a very simple core narrative. Three - we would create a virtual character to act opposite the user to draw her into the story. Four - work on interactivity would take precedence over the creation of complex graphics. The essence of this story was to recreate the experience of being in a dysfunctionally claustrophobic relationship. So by "story" we mean a simulation with a purpose and a narrative arc which the user takes part in. We do not give the user information as in a role playing game about their character, motivations and goals, rather they enter our story space as themselves. However, the narrative, the computer-controlled characters, the graphics, the interaction, the sound, are all constructed with a view to stimulating the user to experience the dilemmas inherent in this specific story. As we built our application it became clear that an iterative creative process is vital for the construction of virtual drama. Creating successive drafts of the project and testing it with a variety of users not only honed the interface, but clarified how well the story was being communicated. By chance during the building of The Thing Growing, we created a Wizard of Oz stage in which a networked human augmented the Thing's intelligence. Data gathered from this stage enabled us to pinpoint much more clearly how we needed to build that intelligence.

2.1 Core Narrative
One much hyped strength of computer-mediated fiction is the possibility of non-linear narrative and multi-form plots. {12] {6] However, it has proved very hard to create such ambitious works in practice. We decided to build our interactive drama around a very simple core narrative for two reasons. First, we considered that it was too time-consuming and expensive to create a multiplicity of complex scenes in VR. [20] Second we did not find elaborate hypertext stories cathartic or dramatically satisfying, and that was the kind of experience we were pursuing. Our story only involved one character and a few sets. We thought that this kind of simplicity was in our favor; that making one believable, responsive character that really engaged the user was challenge enough. The Thing Growing recreates for the user the sense of being trapped in a claustrophobic relationship with a clinging and demanding other, the Thing. A difficult relationship, by definition, is composed of both positive and negative emotions, and so we needed to stimulate ambivalent feelings about the Thing. We evolved the narrative structure as our basic strategy for doing so. In order to keep authorial control over pacing and the creation of surprise we based The Thing Growing on a dramatic arc with a distinct beginning, middle and end.[4] Each act was also subdivided into a beginning, middle and end of its own. This structure provides the user with constantly changing clues and suggestions on how to act and incidents to act against, and imparts a forward thrust to the experience which is essential to maintaining interest. Each scene in each act also has an underlying goal to stimulate an emotion or sequence of emotions in the user. Our narrative moves from scene to scene either as a result of the user's actions, or because a time-limit had been passed, to avoid any deadends where the user can get stuck unless she performs a specific action.[3] We also discovered that it was important to handle transitions between scenes so that the user does not become conceptually lost.

Fig 1. User dancing with Thing - Avatar of User and Thing meet Thing's Evil Cousins

2.2 Computer Controlled Character
Our prime objective was to make the user a protagonist in our virtual drama. Given the nature of our basic story - a love-story of sorts - we needed a computer-controlled character to act as the other. Although we knew several research teams had focused on using intelligent agents for interactive, narrative experiences [5] [2] [7], we felt that they addressed the user too cerebrally to be useful role models for our application. In these projects the user is invited to create fiction with the agents; to direct them; or to marvel at simulations of autonomy and personality. We wanted to provoke a more basic emotional response in the user. Our approach to building the intelligence for the application was to focus on the agent's character and responsiveness to the user rather than on its own set of intelligent and autonomous behaviors. It would be an actor/agent giving a dramatic performance which would include simulating emotions and following the arc of our narrative. Our hypothesis was that that simulated emotional behavior would stimulate emotional responses in the user. Our observations suggest that people do respond emotionally as they try to cope with the virtual creature and that they internalize a psychological model of it.[12] One interactor suggested the Thing was just trying to tease her when it criticized; another described the creature getting in "my face, making me mad;" another said the character was very manipulative. [21]

The narrative provided a context within which the Thing's actions and goals were clear. Its willful and somewhat psychotic personality maintained a recognizable emotional logic. The Thing is dominating because of the exigencies of the story and constantly tells the user what to do. Although the user may become annoyed at the bossiness of the Thing, she is less likely to attribute her frustration to the limits of the program than to the character she is dealing with. The Thing has a human voice; we recorded about 500 short phrases that encompass the narrative possibilities of our story. Many of the phrases are similar in meaning but phrased slightly differently so that we avoided a robotic repetition. It has four moods, manic, happy, sad and mad. Many of the pre-recorded phrases were recorded in each of the moods, so it can be saying the same thing but the emotional value of the interaction will be different. The Thing, as a coherent actor, is created by a passage through narrative fragments, where each fragment is an action made up of a short spoken phrase and an accompanying animation. The effectiveness of the character depends on seamless and appropriate transitions from fragment to fragment. The selection of the appropriate action is informed and determined by the passing of time, the user's location and interactions, and the Thing's internal logic and moods.

2.3 Interactivity v Graphics
Given finite resources, we decided to focus our time and attention on the responsiveness and interactivity of our VR environment not on gorgeous, highly resolved 3D graphics. Wehypothesized that high visual resolution of landscapes, buildings and characters is only one axis of the sense of reality that a virtual environment can provide.[17] By reality we mean the extent to which the user feels present and immersed in the virtual environment.[19] Writing about Disney's VR project Aladdin, Pausch et al say "We suspect that the limited believability of our first system's characters is due to low fidelity."[14] In contrast we decided to construct simply rendered characters and environments which would respond to the user intelligently. Scott McCloud suggests that viewers can more easily identify with simply drawn, iconic, cartoon characters.[10] In the same way we believed that a simply designed main character would be able to stand in for any significant other in the user's life (spouse, sibling, parent, child) so that elements of the user's own relationships would seep into the emotional narrative.

3 The Trial The Trail - Planning
The Trial The Trail is also built around a specific story.It is an interactive quest based on elements from Tarkovsky's Stalker, Lewis Carrol's Alice Through the Looking Glass, and Monty Python and the Holy Grail. The user will embark on a guided journey through this warped yet familiar landscape. As she proceeds her actions and interactions will be logged, interpreted psychologically, and used to determine the outcome of her quest. The Thing Growing's story line determined that we should build the kind of agent whose neurotic ticks tended to obscure the limitations of its intelligence, not vice versa. The desire to manipulate the user emotionally led us to use the kind of narrative structure that is more typical of drama. In The Trial The Trail, we are again trying to lead the user on an emotional and rather introspective journey. The psychological terrain that we are interested in exploring encompasses ambivalent notions that our actions and choices lead where we want them to; questions of what is more important, the journey or the goal; issues of what our desires are and whether they can be gained. We believe that the familiarity of the quest tradition will be a crucial new element that will help us build a more complex narrative this time around. Because VR story-telling does not yet have a lexicon of common practices that make people at home in the environment, several theorists and practitioners suggest the most enjoyable experience comes when the user has something familiar to hang onto, some clue about how the she should proceed to interact with this new environment.[6] [14] [12]

3.1 Narrative.
Michael Mateas argues that an interactive drama needs at least two computer-controlled characters, besides the user, to carry the story out of narrative cul-de-sacs.[8] Although we were very pleased with The Thing Growing, we realize that it relied heavily on one main computer-controlled character. The Trial The Trail's narrative contains more scenes, more interactive tropes, and more characters, but will still be structured around a core narrative arc. Our planning stage has included working on two story-boards of this narrative. In the current storyboard [16], the opening scene takes place at a barred gate into the "zone". In the inciting incident a virtual guide reveals the quest - if the user reaches the top of a grain elevator in the middle of the zone, she will get her heart's desire. The journey is difficult - distractions and dangers will meet them. After this explanation the guide helps the user get into the zone. The middle section contains a series of interactive challenges which encourage interactivity and emotional response. These include a series of distractions which will encourage the user to dally instead of getting on with her quest; a reed bed which responds physically and musically to the user's movements; a pool which she can ripple; and a fire/particle system which she can sculpt. There will also be test-like challenges which include the opportunity to rescue a helpless creature from evil by fighting a sentient morning-star; and the problem of what to do with this creature who, once rescued, attaches itself clingingly to the interactor, preventing her from going further, and reaching her goal. The end of the drama is determined by a psychological profile of the user gathered from her responses to the challenges. This will determine if her heart's desire is Life or Death. Both choices are represented as temptations. Life is equated with domination, Death with submission.

Although we will maintain a narrative arc with a first act that introduces the goal, a second that introduces hindrances for the user trying to reach the goal, and a third that ends the quest, we are planning to be more flexible with the order of the scenes in the second and longest section. We want the user to feel as free as possible to move through the virtual environment and choose what she does next. In order to maintain a rising curve of interest, a typical dramatic strategy is to put the protagonist in situations which are progressively riskier and riskier, with lulls between each crisis, until the final moment of truth arrives.[11] This strategy is based on having linear control over time. The author of interactive environments does not have such control, VR is an omni-directional space. Yet allowing the user to wander about too freely, negotiating the scenes in the middle section in any order, may mean that we lose intensity and introduce boredom. At this stage in our planning we intend to address this in two ways. At our most manipulative we will be able to make our virtual environment very foggy - therefore if the user doesn't wander in the right direction - the fog can roll in and we can slap the next interactive situation down right in front of her. At our most subtle we will there will be multiple ways of entering each scene which take account of which scenes the user has already encountered.

3.2 Computer Controlled Characters
The Trial The Trail will have multiple agents. An intense and rather mournful guide figure will take on the role of explicating the story, explaining the interface, setting the user's goals and intervening to move the drama from one scene to another. The work of stimulating the user emotionally will be taken on by the agent she must fight, the agent who becomes the too-clinging creature she has saved, and the distracting, smart elements in the environment as a whole. Other agents will be used to set scenes and moods. As in The Thing Growing the job of the agents will be to interface with the user and assemble fragments of the narrative into a compelling dramatic form. However, in order to handle multiple agents and a more complex narrative we decided that we needed a more advanced agent architecture. We have started to collaborate with AI researchers on the creation of an actor/improviser agents for The Trial The Trail. The agent will be built using SNePS -- The Semantic Network Processing System - a knowledge representation/reasoning system developed by Stuart Shapiro and the SNePS Research Group at the University at Buffalo.[18] Our actor/improviser agents must guide the human user through virtual locations, moral choices, and emotional states - although the guidance may not be apparent. The agents must react believably, and in character. To this end they will be designed to understand natural language, to dissemble if they do not understand, and to talk back to the interactor.

The SNePS agent's knowledge base will be the narrative, in the most expansive sense of that word, comprising information about the VE, and all that it needs to guide/persuade/manipulate the user to see, hear, feel and do in order for her to experienced our drama. The agent will also have a model of the user, that will enable it to interpret the user's emotions and psychological state at any given point in the narrative, and more effectively mold the narrative to give the user a unique experience. Part of the research process is to determine whether we want one artificial intelligence which then provides the reasoning for each embodied agent and for the entire virtual environment, or whether each embodied actor or part of a "smart environment" has its own intelligence aspect. In the latter case, we may also need a dis-embodied director agent to oversee the entire development of the plot. Additionally we will need to add speech recognition software to the VR display system. Although the primary mode for the agent will be to understand the speech input and reason from it, it will also have methods for dealing with speech that it does not understand. The creation of a strong personality for each character will help determine character- and context-appropriate ways for covering lack of understanding. Current speech generating software does not do a good job of creating an appropriate emotional tone [12] - so the agent will not use synthesized speech software, instead just as for The Thing Growing it will pick an appropriate verbal response from a pre-recorded library.

3.3 Visuals

Fig. 2. First Version of The Trial The Trail

We started the planning process of The Trial The Trail in Fall 2001, made a basic story-board, created some 3D models, assembled a preliminary VR draft of the world, and started work on the behavior of the fighting morningstar. In May 2002 we showed a small fragment as a work in progress and began to realize that our first draft was not communicating our story. It was game-like; too like a DisneyQuest!

Whereas we want to provoke dis-ease about the whole question of quests and desire. The rest of 2002 was dedicated to thinking more carefully about the series of moods that the want to maneuver our users through, and the kinds of graphics that may do this. In our second story-board we have returned to The Thing Growing principle of making simple models that can be read symbolically. Although we do want to trap the user into reacting in a very direct way to our story - to let themselves be carried away. We also want the visuals to recall the fact that this is a constructed reality, that their emotions are like-wise constructed, that all is suspect and should be analyzed by the brain. In dramatic terms we are interested in allowing the user to maintain a certain Brechtian distance from our plot - to collude in the springing of the traps, but to recognize the traps and the underlying story themes.

6 Conclusion

As we continue with the production of The Trial The Trail our four organizing principles remain intact, although they have evolved. We continue to maintain that the specific content of a VR drama must be the major driver of the production process, and doubt the efficacy of elaborate theories and methods that attempt to build generic actors, or story-generating systems. Nevertheless we believe that certain conceptual and technical tools can support that process. Our principle of using a core narrative remains - the narrative should be broken into discreet scenes. Although some scenes may be re-ordered on the fly there should be a dramatic arc that provides the story with a beginning, middle and end. Attention must be paid to transitions between scenes in order for a coherent story to emerge. We are beginning to hypothesize that there may be two main types of scenes - those that contain the interactive or informational content, and those that are transitional. The combination of the two will allow for the greatest flexibility. We have also learnt that we must prepare and test iterative drafts of the project in order to hone the interface and content. We continue to believe that intelligent agents are a vital part of the process. Our interactive drama is intended to create an unfolding story around the user which the agents establish, populate, maintain and influence. The scenes of the narrative can be broken down into small fragments, and the virtual actors reassemble the fragments to accomplish this goal. Building better agents, building agents that can take in human speech as input, has a high priority for our current project. We have also learnt that building a Wizard of Oz stage into the production is vital for the efficient building of intelligent agents. Finally, we are still inclined towards simple non-photorealistic visuals, and to prioritize interaction over graphics. Increasingly photorealistic graphics signify the games industry and trap the work in connotations that effect the content. We want the graphics to serve the story we want to tell, not to trip unwanted metaphorical associations.

We would like to acknowledge the support of the Electronic Visualization Laboratory, University of Illinois at Chicago, the NTT InterCommunication Center, the Department of Media Study and the Department of Computer Science, University at Buffalo. We would like to thank Amit Makwana and Sara Nohejl.


    Anstey, J., Pape, D.: The Thing Growing: Autonomous Characters in Virtual Reality Interactive Fiction, In the Proceedings of IEEEVR Virtual Reality 2000, IEEE Computer Society Press, New Brunswick, NJ (2002) 71-78
    Blumberg, B., Galyean, T.: Multi-Level Direction of Autonomous Creatures for Real-Time Virtual Environments, In the Proceedings of SIGGRAPH 95, ACM Press, Los Angeles, CA, (1995) 47-54.
    Bobick, A., Intille, S., Davis, J., Baird, F, Pinhanez, C., Campbell, L., Ivanov, Y., Schütte, A., Wilson, A. : The KidsRoom: A Perceptually-Based Interactive and Immersive Story Environment. In Presence Vol. 8, Num. 4. MIT Press, Cambridge MA (1999) 367-391
    Glassner, A., Wong, C.: Fiction 2000:Technology, Tradition, and the Essence of Story. In SIGGRAPH 99 Conference Abstracts & Applications, ACM Press, Los Angeles, CA (1999) 161
    Hayes Roth, B., Brownston, L., Sincoff,E.: Directed Improvisation by Computer Characters, Stanford Knowledge Systems Laboratory Report KSL-95-04 (1995)
    Laurel, B.: Computers as Theater. Addison-Wesley, New York (1993)

    Mateas, M.: An Oz-Centric Review of Interactive Drama and Believable Agents, Technical Report CMU-CS-97-156, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA. (1997)

    Mateas, M.: A Preliminary Poetics for Interactive Drama and Games. In SIGGRAPH 2001 Electronic Art and Animation Catalog, ACM Press, Los Angeles, CA (1999) 51-58.

    Mateas, M., Stern, A.: Towards Building a Fully-Realized Interactive Drama, The Fourth International Digital Arts and Culture Conference, Providence, Rhode Island,(2001) pages?

    McCloud, S.: Understanding Comics, Tundra Publishing, Northampton, MA, (1993)

    Mckee, R.: Story: Substance, Structure, Style, and the Principles of Screenwriting. Harper Collins, New York (1997)

    Murray, J.: Hamlet on the Holodeck, The Future of Narrative in CyberSpace. Simon & Schuster, New York (1997)

    Pape, D., Anstey, J., Dawe, G.: A Low-Cost Projection Based Virtual Reality Display, In the Proceedings of SPIE, Vol. 4660, SPIE - The International Society for Optical Engineering, San Jose, CA, (2002) 483-491

    Pausch R., Snoddy S., Taylor R., Watson S., Haseltine E.: Disney's Aladdin: First Steps Toward Storytelling in Virtual Reality. In: Dodsworth Jr., C. (ed): Digital Illusion. Addison Wesley, New York (1998) 357-372

    Roussou, M.: Incorporating Immersive Projection-based Virtual Reality in Public Spaces. In the Proceedings of 3rd International Immersive Projection Technology Workshop, Vol. T52. Springer Verlag, Berlin, Heidelberg, New York ( 1999) 33-39

    Storyboard at:

    Sandin, D.: Digital Illusion, Virtual Reality and Cinema. In: Dodsworth Jr., C. (ed): Digital Illusion. Addison Wesley, New York (1998) 3-26

    Shapiro, S. C. and the SNePS Implementation Group: SNePS 2.6 User's Manual, Department of Computer Science and Engineering, University at Buffalo, (2002)

    Slater, M., Wilbur. S.: A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence Vol 6,Num 6. MIT Press, Cambridge MA (1997) 603 - 616

    Talin.: Real Interactivity in Interactive Entertainment. In: Dodsworth Jr., C. (ed): Digital Illusion. Addison Wesley, New York (1998) 151 - 159

    Video clips at: