Publication Type: Individual Poster
Abstract: Previous studies show young children utilize social cues signaled by caregivers to facilitate word learning. When two agents (the mother and the child, etc.) engage in a social interaction, the complex dynamic flow of the interaction depends on the child’s reaction to the mother’s behaviors which in turn will influence the mother’s response actions to the child’s reaction. Without interfering with the interaction, we cannot control the child’s responses nor the mother’s actions. And, without systematically manipulating some variables in the interaction, we cannot perform the kinds of experiments necessary to determine underlying mechanisms, leaving us with description alone.

The present study proposes a new paradigm using virtual reality technologies to de-couple complex social interactions. We build virtual agents that act as either a simulated caregiver or a simulated learner, and interact with human subjects to simulate mother-child interactions. By pre-programming virtual humans’ actions, we control one agent in the interaction and measure the behaviors of the other agent (human subjects, etc.).

Experiment 1 studies how learners utilize social cues to associate object names with their referents. We developed three virtual teachers with different social abilities, which form three learning conditions. There are 12 trials per condition and three virtual everyday objects are displayed on a virtual table at each trial. The virtual teacher in the first condition just utters object names without signaling any embodied social cues. In the second condition, a virtual agent looks toward the objects while naming them while the virtual teacher in the third condition points to and gazes at objects in speech production. We measure and compare the performances of human learners (adult subjects as a first step) in these three conditions.

Experiment 2 studies how learners’ reactions shape the behaviors of language teachers. The virtual environment is the same with the first experiment except that three different kinds of virtual learners were implemented. Subjects are asked to interact with virtual learners and teach them the names of a set of objects. They can point to and move those objects on the virtual table in real time through a touch screen mounted on the monitor. The three virtual learners demonstrate different engagement levels and sensitivities to a teacher’s actions and attention. We compare and analyze the behavioral data of real subjects, such as what they speak and how many times they point to or move objects, when they interact with different virtual learners.