Development of an Intelligent, Real-time, Heart Rate Sensitive Virtual Reality System
ABSTRACT
Children with autism, a neurological disorder, have difficulties carrying out normal social interactions. The most widely accepted treatment for autism involves behavioral intervention programs which are costly and not easily accessible. A virtual reality system which can imitate current intervention paradigms while taking the emotional state of the children into account would be beneficial. Such a system could lower costs and increase accessibility of intervention programs. As a preliminary step, we created a virtual avatar that responded on the anxiety level of a subject as indicated by his heart rate. A heart rate sensor was designed and an interface was developed between the sensor and the avatar. The subject was exposed to a stimulus to increase anxiety level. Based on the subject’s heart rate, the avatar gave a response regarding his anxiousness. This study demonstrated that a virtual reality system can detect a subject’s anxiety level and provide appropriate feedback.
INTRODUCTION.
One in 110 children in the United States is diagnosed with autism [1] and there is still no known cure [2]. Autism is a neurological abnormality that disrupts the brain’s development of social, behavioral, and communicative skills [1]. The most common treatments are behavioral intervention programs which can dramatically improve the lives of the children and their families by attenuating the dynamic changes of behavior expressed by these children when they are placed in uncomfortable situations [3]. Because of the expensive nature of these intervention programs and lack of trained behavioral therapists, many children with Autism do not receive proper treatments [3]. In order to lower the costs and make intervention more accessible, a new avenue of research is being explored to develop autonomous systems that can assist in such intervention.
Children with Autism tend to cooperate better with autonomous systems than humans because of the simplicity and predictability of their interactions [4]. Robots and virtual reality systems in particular have been shown to have great potential in such interactions [4, 5]. With this in mind, virtual reality was selected to be the autonomous system of choice for this study because of the widespread accessibility of home computers.
The primary goal of this study was to create an intelligent system with the ability to intrinsically adapt its behavior with respects to external stimuli and present individualized reinforcement. One of the stimuli best suited for autism intervention is variations in psychophysiological signals representative of the emotional states experienced by a subject [6]. Because children with Autism exhibit deficits in communicating emotions, having a system that can identify and adapt itself according to the child’s emotional state will be highly beneficial.
MATERIALS AND METHODS.
Hardware and Software.
Heart rate, measured in heartbeats per minute, was the principal psychophysiological signal used as a stimulus for the virtual reality system. In order to calculate heart rate based on blood flow to the fingertip, a heart rate sensor was assembled according to a modified circuit diagram (Fig. 1)[7]. The sensor functioned by measuring the transmission of infrared light. When an obstruction (e.g. a human finger) was present between the infrared light source (SFH487) and a matching receiver (SFH309FR), the amount of light transmitted changed. This changed was inversely proportional to blood flow. The reaming portion of the circuit mainly amplified and filtered the signal from the receiver. This device was extremely susceptible to electrical noise from movement so the transmitter and receiver were mounted onto a clothespin to reduce shake. An Arudino Board [8] was used to interface the sensor with the task computer.
Figure 1. Heart rate sensor circuit diagram.
Various modules were written in different languages in order to read and analyze the signal from the sensor. The signal processing module filtered the input signal to remove noise and calculated the heart rate. The supplementary methods describe how these modules were developed. The calculated heart rate was fed as an input to a virtual reality programming software in which a virtual avatar was created. The avatar gave different verbal responses on a subject’s anxiety level as indicated by his heart rate. Heartbeat palpitations, or increased heart rate, are indicators of increasing anxiety level [9].
Experiment to test the interface.
A pong-style computer game, where a subject hit a ball with a paddle and the computer player returned the ball with a hit of its own, was played while wearing the heart rate sensor. The game had three difficulty levels which altered the size of the ball, size of the paddle, and the speed of the computer player. There were three trials per difficulty level and the heart rate during each trial was monitored in three one-minute intervals. Before each trial, a two minute baseline reading was obtained where the subject was not exposed to the gaming environment. Then during the trials, the heart rate was compared to the baseline, and the virtual avatar gave responses every minute. The nature of these responses varied based on whether or not the trial heart rate exceeded certain predefined thresholds which are outlined in the supplementary methods. Th is simulation was executed in order to show a working interface between the heart rate sensor and the virtual environment. Changes in the difficulty level were designed to alter the anxiousness of the subject, and the avatar was to respond accordingly. Due to time and logistical constraints, only one subject took part in this experiment, and he was not diagnosed with Autism.
RESULTS.
The average heart rate during the baseline and trial sessions was compared in order to assess the effectiveness of the pong game as a stimulant of anxiety (Fig. 2). The baselines between difficulty levels were different because heart rates were monitored at different times throughout the day.
Figure 2. The average heart rate for the baseline and trial sessions at each diffi culty level was compared. The trial heart rate was always lower than its respective baseline heart rate. The error bars represent standard deviation.
While the participant was playing the pong game, the virtual avatar’s responses were recorded (Table 1 and Fig. 3). As the anxiety level increased, verbal response also changed according to Table 1. The most numerous response was 0 where the avatar declared that the subject was not anxious.
| Response Number | 0 | 1 | 2 | 3 |
| Response | “You’re doing fine.” | “Let’s slow down.” | “You seem to be getting anzious.” | “You’re stressed out!” |
Table 1. Actual response assigned numerical values for easier representation in Figure 3.
Figure 3. The virtual avatar gave different responses based on the subject’s average heart rate over each minute of each trial at each difficulty level in the pong game.
DISCUSSION.
Although having the heart rate sensor mounted on a clothespin did alleviate some noise, it did not completely clean up the output signal from the sensor. Voltage spikes from electrical noise caused the instantaneous trial heart rate to exceed the baseline heart rate. In order to account for these occurrences, the avatar was coded to respond only if increased heart rate lasted for about 30 seconds. Voltage spikes resulted in a false indication of increased heart rate but because they did not last for 30 seconds, they had minimal effects on the virtual avatar’s response. This demonstrates an effective interface between the heart rate sensor and the virtual environment.
Although previous studies have shown that cardiovascular activity and anxiety are linked [10], this relationship was not strong in this study. In comparing the differences between trial and baseline heart rates between the difficulty levels, we saw significant differences between Difficulties 1 & 2 and 1 & 3 but not 2 & 3 (Table 2). This led us to conclude that modifications in the pong game to induce anxiousness in the subject were insufficient.
| Differences (trial – baseline heart rates) between difficulty levels | Difficulty1&2 | Difficulty1&3 | Difficulty2&3 |
| p-values | 0.0336 | 0.0089 | 0.5596 |
Table 2. Using a two sample t-test, the average differences between trial and baseline heart rates between difficulty levels were compared.
The task presented to the subject, the pong game, was not strenuous enough to cause noticeable increases in heart rate compared to a baseline. In fact, the task seemed to relax the subject instead of making him anxious seeing as the average trial heart rate was consistently lower than average baseline heart rate (Fig. 2). This is consistent throughout the results seeing as the virtual avatar’s most numerous response was that the subject was not anxious.
CONCLUSION.
This study showed that a psychophysiological signal, heart rate, can be used as a stimulus for an intelligent virtual reality system. A virtual avatar was able to provide varying responses on the anxiety level of a subject as expressed through changes in the subject’s heart rate. Furthermore, as exhibited by the results, the entire interface between the heart rate sensor and the virtual environment was functional and versatile. It took into account voltage spikes from noise and was capable of giving three distinct responses on the subject’s anxiousness.
For intelligent virtual reality systems to be widely used, more tests need to be conducted to gauge the effectiveness of the systems. A commercial heart rate sensor should also be employed because it can provide cleaner signals. One crucial step to improve this overall experiment is to design the trials to maximize the differences between baseline and trial heart rates. This can be achieved by changing the stimulus to which the subject is exposed. Instead of a pong game, the subject could watch a video clip to induce shock, thus increasing heart rate. In addition, more trials should be conducted to increase the confidence levels of the results. Other questions that need to be asked before long term deployment of this technology include:
What is the most optimal stimulus to which the virtual reality system could adapt?
If multiple stimuli are applied, how could they be interpreted to return the best output possible?
How feasible is a system that adapts a task instead of just returning comments?
While heart rate is a possible psychophysiological signal to use as a stimulus, other signals such as perspiration levels, cardiovascular and electromyogram activities can be used as indicators of emotional state [10–13]. By finding the advantages and disadvantages of these different indicators, they can be linked together to create one conglomerate where the benefits of all the signals can be used to provide the stimulus for the intelligent virtual reality system. Doing so would improve the overall performance of the system and make it more viable to be used as part of autism intervention paradigm.
In this study, a virtual avatar gave responses while a task was being undertaken but in the future, there should be a push towards having a system where the task itself is adapted. In context of the pong game, a future virtual reality system could be used to adapt the game difficulty in real-time to increase the subject’s engagement. Having such a system can lead to new innovations in areas such as autism intervention where self-adaptive systems can increase the engagement and reduce the frustration of autistic children. This would cause the children to learn better from the intervention programs.
ACKNOWLEDGMENTS.
I would like to thank my PI, Dr. Nilanjan Sarkar, for his consistent support in this project. I would also like to thank Dr. Uttama Lahiri who provided me with the utmost assistance throughout this endeavor. In addition, I want to acknowledge the Robotics and Autonomous Systems Laboratory for providing me the necessary materials to complete this study. I would also like to acknowledge the School for Science and Math at Vanderbilt for giving me the opportunity to carry out this research, and Dr. Chris Vanags for his guidance in this project. The project described was supported by Award Number R25RR024261 from the National Center for Research Resources. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center for Research Resources or the National Institutes of Health.
REFERENCES.
- “Introduction to the Arduino Board,” Arduino, 2008.
- D. Zieve, et al., National Institute of Health, (2010).
- M. E. Dawson, et al., J. T. Cacioppo and L. G. Tassinary, Eds. New York: Cambridge Univ. Press, 295 (1990).
- A. Pecchinenda, et al., Cognit. Emotion, 10, 481 (1996).
- J. T. Cacioppo, et al., Handbook of Emotions, M. Lewis and J. M. Haviland-Jones, Eds. New York: Guilford, 173 (2000).
- J. F. Papillo and D. Shapiro, J. T. Cacioppo and L. G. Tassinary, Eds. New York: Cambridge Univ. Press, 456 (1990).
- C. Rice, Center for Disease Control and Prevention, (2009).
- L. Tarkan, New York Times, (2002).
- S. J. Rogers, J. Clin. Child Psychol., 27, 168 (1998).
- “The Aurora Project Introduction,” University of Hertfordshire.
- J. Groden, M. S. et al., Focus Autism Other Dev. Disabil., 20, 244 (2005).
- G. J. Compton, Pico Technology.
- K. Dautenhahn, I. et al., Proc. IEEE Int. Symp. Comput. Intell. Robot. Autom., 577 (2003)
SUPPORTING INFORMATION.
Posted by Printess Berry on Thursday, August 27, 2015 in May 2012.
Tags: Autism programming, heart rate, real-time, Virtual reality