Visuo-haptic multisensory learning enhances encoding and recall of Rey-Osterrieth complex figure shapes

Abstract

The objective of this study is to measure the impact of multisensory learning incorporating an original haptic modality on human visual memory via a behavioural experiment using the Rey-Osterrieth Complex Figure Test（ROCFT）. Recent studies have reported that children with difficulties in letter or figure perception have weaker visual perception processes. The scientific evidence on the effect of the learning support using haptics is required. Adult participants （n=52） drew a ROCF image through copy and recall, and then re-learned the image using a three-dimensional ROCF plate. Two groups were formed : the V-H group relearned the image using two modalities （haptics and vision）, whereas the V group was only visually explored. The second task was administered 24 hours after the first session. The data was analysed by two-way ANOVA with the scores on ROCFT as a dependent variable and the different modalities and scores before and after the learning as independent variables. In addition to the total scores on ROCF, scores in each subunit divided into three were also analysed in the same way. The result suggested the validity of visuo-haptic multisensory cognitive integration on visual memory

 

Introduction

Developmental dyslexia is often associated with weakness in visual perceptual processing skills. In the study of Goto et al. (2010), children with developmental dyslexia scored lower in visual memory tasks. Interventions with such children include haptic learning, such as making letters with clay to stimulate the haptic system.

 

The interaction between the visual and haptic systems explains the improved performance in visual recognition tasks with haptic learning. Haptic learning has the same effect as visual learning in visual recognition tasks (Ambrose & Cheong, 2011). Also, Helbig and Ernst (2007) found that learning with two modalities (visual and haptics) can increase the accuracy in memory of the shape and the size. Therefore, the interaction between these two systems suggests a positive outcome of visual pattern learning using haptic exploration. 

 

Some studies have confirmed the interaction and integration between haptic and visual systems using functional magnetic resonance imaging (fMRI). James et al. (2002) observed activation in the lateral occipital cortex and middle occipital cortex regions in fMRI while subjects were touching three-dimensional clay object recognition tasks with their hands. The same brain regions were activated when the subjects visually explored the objects after touching them with their hands. They concluded that haptic learning can facilitate brain activation in the visual area through object presentation system in the lateral occipital cortex which haptic and visual areas share. 

 

Nishino and Ando (2008) suggested that the human brain function of three-dimensional object recognition is based on the lateral occipital cortex which overlaps with haptics and visual. In this model, the haptic and visual images created from various inputs based on bottom-up processing are integrated into the lateral occipital cortex. This is consistent with what Lacey and Sathian (2014) found. With this model, when the familiar object is haptically explored, the images are recalled in the prefrontal cortex through the lateral occipital complex while the unfamiliar object is touched, the image is recalled in the lateral occipital complex supported by special image processing from the somatosensory cortex to intraparietal sulcus.

 

Nishino and Ando (2008) and Lacey and Sathian (2014) both believe that the top-down processing in the lateral occipital cortex plays an important role in cross-modal input recognition. Based on the model, tactile input combined with visual input can facilitate concrete visual recalling resulting in better object images. Thus, it is expected that independent haptic imagery can improve visual recognition in subjects who have difficulties with visual object recognition tasks.

 

In the study of Kalénine et al. (2010), they have studied object recalling with haptic learning by comparing two preschool groups, one with visual and haptic learning, and another with only visual learning. They found that combined visual and haptic learning facilitated better performance in recalling morphological features of shapes including square, rectangle, and triangle. The result suggests visual and haptic exploration improved object recalling following visual and haptic dual coding.

 

However, the study of Kalénine et al. (2010) does not apply to the Japanese language as they only studied the impact of haptic learning in simple/concrete object recognition while Japanese letter learning requires complex visual pattern memory. Thus, Rey-Osterrieth Complex Figure Test (ROCFT) which is used in reading and writing disorder assessment was chosen in this study to investigate the impact of haptic learning on visual recognition. ROCFT can measure visual memory with a reproduction score. In addition to our original ROCFT, we have created a 3D ROCFT with a 3D printer and used this in haptic learning tasks. We have chosen ROCFT as a memory test as the Japanese language has many letters compared to languages with alphabets resulting in Japanese dyslexic children having lower reproduction scores of ROCFT due to significant visual information processing load. To accurately produce and reproduce complex ROCFT figures, attention to the entire figure as well as details, visual pattern recognition, visuospatial recognition, and planning are required. The purpose of the study is to measure the impact of haptic learning on visual memory by investigating the impact of haptic learning with the three-dimensional ROCFT on reproduction performance.

 

Ⅰ.Method 

１．Subjects

Subjects were 52 adults including 16 males and 36 females with an average age of 35.7 years old. The standard deviation was 9.1. The subjects were randomly divided into 2 groups. Both groups have completed the second exploration following the first reproduction task. During the second exploration, the first group (V-H group) consisted of 8 males and 18 females with an average age of 36.0 and SD 8.46 explored the stimuli both using vision and haptic and the second group (vision group) consisted of 26 adults with 8 male and 18 female, average age of 35.0 and SD 9.73 only used vision to learn objects. We also confirmed with a t-test that both groups do not have differences in performance comparing the reproduction performance before the second exploration task ((t50) = -1.29, p= 0.20, ns). This study referred to the study of Yamashita (2007) which concluded there is no difference in performance between sexes and among ages 18 to 55 and has chosen healthy males and females without previous and current significant neurological and psychological conditions. They compared the reproduction performance of 144 healthy Japanese adults following the standardised ROCFT procedure as well as the standardised grading procedure of Osterrieth (1944). Also, the Japanese version of FLANDERS (Okubo et al., 2014) and the Semmes-Weinstein monofilament test (Bell-Krotoski et al., 1995) were used to confirm the dominant hands and hand and finger sensory of subjects. The process of this study was approved by the ethical review board of Kyoto University.

 

2．Stimuli

We have made our stimuli based on the three-dimensional data created with Fusion 360 from Autodesk. The size of the stimuli was B5 as shown in Figure 1. ROCFT is a visual memory test (Table 1) published by Rey (1941) and standardised by Osterrieth (1944). Subjects will first copy a figure, then will draw the same figure by recalling it after 3 minutes. Visual recognition and memory are assessed based on the recalled figure and accuracy in position. The highest score is 36 with the most accurate recalling. This is often used as a screener for learning disorders (Kayamura & Kayamura, 2008) as it can assess not only visual recognition and memory, but as well as several neuropsychological functions, such as specificity, executive function, and visuo-constructional ability. This study followed the method suggested by Osterrieth (1944) to administer and grade recalled figures. In addition to the total score of ROCFT, we have added 3 other categories based on the feature of ROCFT incorporating the Organisation Scoring System (OSS) suggested by Chervinsky et al. (1992). The categories include an exterior unit with protruded triangles and crosses, an interior unit with lines and crossings, and a partial unit with small and unique symbols. OSS analyses composition strategy by looking at whether the Ray figure is reproduced using 6 sections. This study does not consider composition strategy however we have categorized 6 sections divided by OSS into three divisions based on the location and features. 

We have made the height of three-dimensional ROCFT 0.5mm based on Hayashi et al. (2003) which concluded that the best height for Braille is 0.5mm with the highest accuracy. The width of the line is 1 mm painted black which allows both visual and haptic perception. 

 

Figure 1. Three-dimensional ROCFT stimuli were created based on the three-dimensional data of ROCFT with a three-dimensional printer.

 

３．Test procedure

1．1st ROCFT reproduction

First, subjects copied the three-dimensional figure followed by three minutes break. Then, the first reproduction task was administered, in which subjects were asked to draw the ROCFT figure on blank paper by recalling the image. 

２．2nd Exploration 

After the first reproduction task, the second exploration was done. 52 subjects were divided into 2 groups. V-H group touched the ROCFT figure with their three (index, middle, and ring) fingers for three minutes. The aim was to explore the figure with finger sensation using a wider. V group has only visually perceived the figure. 

3．2nd ROCFT reproduction

The 2nd reproduction task was completed 24 hours after the second exploration. Karni et al. (1994) reported the effect of sleep on visual memory improvement, therefore this study incorporated 24 24-hour gaps in between the tasks to measure the enhanced performance after sleep.

 

Table 1.  Two-way analysis of variants between V-group and V-H-group

 

4．Analysis

The data was analysed with IBM SPSS Statistics 27 software. Two-way analysis of variants was used to measure the difference in the performance between the two groups. The total score of ROCFT as well as other sub-scores including exterior, interior, and a partial unit score. Also, a t-test was used to measure the impact of the second exploration on the figures in two groups. With a t-test, we compared the performance before and after the exploration. 

 

Ⅱ.Results

The performance after the second exploration improved in both groups after the second exposure with the total score of F=13.78, p=0.00, and p<0.01 as well as performance comparison before and after the second exposure with the exterior uni of（（t  25）=－8.85  p=0.00、p<0.01） in V-H group and （（t  25）=－4.46   p=0.00、p<0.01）in V group.

 

 

 

Ⅲ．Discussion

Based on the analysed data, there was a positive relationship between haptic and visual learning with enhanced visual memory in the three-dimensional ROCFT figure reproduction task. Helbig and Ernst (2007) reported enhanced three-dimensional perception with combined visual and haptic learning by integrating information among multiple modalities. Thus, better learning is expected when visual learning is complemented with haptic perception, however, there are some influential factors to consider. For example, visual memory can be facilitated by multimodal learning or the complex figure information was organised during input with active learning (haptic exploration in this study).  

 

Moreover, the positive impact of haptic exploration on visual memory was only seen in the exterior and partial unit, not in the interior unit. The exterior unit is the outline of the ROCFT figure. The skills to pay attention to the entire figure and visuospatial skills to understand the location of each component are required to recognise the outline of complex figures. Haptic learning facilitated figure recognition by following the outline with hands. A partial unit is a small independent unit whose shape, and location can be recognised by touching each of them. Even though there was no difference seen in the interior unit score between the two groups, this unit requires attention to embedded figurative features by inhibiting attention to outline. Furthermore, the way subjects touched the figures mattered. In this study where subjects used their three fingers to couch the three-dimensional ROCFT figure, there was a larger area of their hands exposed to the figure which facilitated sensory input of outline and characteristic features of the figure. On the other hand, figurative features composed of complex liner features in an interior unit require more visual attention to understand the compositional features than haptic exploration with hands and figures. Such limitation of haptic learning needs to be considered in clinical settings where learning complex figures, such as Kanji. This can be compensated through writing the letter with their fingers in the air before the haptic learning by bringing attention to the interior liner compositional feature.

 

This study will contribute to intervention in children who have difficulties with visual recognition and reproduction. Ogino et al. (2019) also indicated that visual memory difficulties in ROCFT assessment contribute to reading and writing disorders. Moreover, Awaya et al. (2012) and Haruhara et al. (2005) reported a low score in the reproduction task of ROCFT from studies on children with writing difficulties, especially with Kanji. The positive impact of haptic learning in this study indicates the possibility of the use of three-dimensional letters as an intervention tool for children with reading and writing disorders to improve written expression by compensating for their visual perception weakness. 

 

References

1. Goto, T., Uno, A., Haruhara, N., Kaneko, M., Awaya, N., Kozuka, J., Katano, A. (2010). Visual Function, Visual Perception, and Visual Cognitive Function in Children with Developmental Dyslexia. Phoniatric and Logopedic Medicine, 51(1), 38-53.
2. Philip, A.P., Cheong, L.S. (2011). Effects of the Clay Modeling Program on the Reading Behavior of Children with Dyslexia: A Malaysian Case Study. The Asia-Pacific Education Researcher, 20, 456-468.
3. Reales, J.M., Ballesteros, S. (1999). Implicit and Explicit Memory for Visual and Haptic Objects: Cross-modal Priming Depends on Structural Descriptions. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 633-644.
4. Helbig, H.B., Ernst, M.O. (2007). Optimal Integration of Shape Information from Vision and Touch. Experimental Brain Research, 179(4), 595-606.
5. James, T.W. (2002). Haptic Study of Three-Dimensional Objects Activates Extrastriate Visual Areas. Neuropsychologia, 40(10), 1706-1714.
6. Nishino, Y., Ando, H. (2008). Mechanisms of Brain Function in Object Recognition Based on Three-Dimensional Shape. Japanese Journal of Cognitive Neuroscience, 51(2), 92-330.
7. Simon, L., Sathian, K. (2014). Visuo-haptic Multisensory Object Recognition, Categorization, and Representation. Frontiers in Psychology, 5, 730-741.
8. Solène, K., Edouard, G. (2011). The Visual and Visuo-haptic Exploration of Geometrical Shapes Increases Their Recognition in Preschoolers. International Journal of Behavioral Development, 35(1), 18-26.
9. Oginobu, Y., Kawasaki, S., Okumura, T., Nakanishi, M. (2019). Developmental Progress and Scale Structure of the Rey-Osterrieth Complex Figure Test in Childhood. Journal of the Biomedical Fuzzy Systems Association, 21(1).
10. Yamashita, H. (2007). Normative Data of Rey-Osterrieth Complex Figure in Japanese Adults. Psychiatry, 49(2), 155-159.
11. Osterrieth, P.A. (1944). The Test of Copying a Complex Figure: Contribution to the Study of Perception and Memory. Archives of Psychology, 30, 206-356.
12. Okubo, M., Suzuki, G., Michael, E.R. (2014). Reliability and Validity of the Japanese Version of the FLANDERS Handedness Test. Japanese Journal of Psychology, 85(5), 474-481.
13. Judith, A. (1995). Threshold Detection and Semmes-Weinstein Monofilaments. Journal of Hand Therapy, 8(2), 155-162.
14. Rey, A. (1941). The Psychological Examination in Cases of Traumatic Encephalopathy. Archives of Psychology, 28, 215-285.
15. Kayamura, T., Kayamura, T. (2008). Development of Accuracy and Construction Strategies in Copying the Rey-Osterrieth Complex Figure. Journal of Mukogawa Women’s University, 55, 79-88.
16. Chervinsky, A.B., Mitrushina, M., Satz, P. (1992). Comparison of Four Methods of Scoring the Rey-Osterrieth Complex Figure Drawing Test for Age Groups of Normal Elderly. Brain Dysfunction, 5, 267-287.
17. Hayashi, M., Kamoda, M., Fujimoto, H. (2003). Study on the Shape of Easily Recognizable Braille. Ergonomics, 39(3), 117-122.
18. Karni, A., Tanne, D., Rubenstein, B.S., Askenasy, J.J.M., Sagi, D. (1994). Dependence on REM Sleep of Overnight Improvement of a Perceptual Skill. Science, 265, 679-682.
19. Awaya, N., Haruhara, N., Uno, A., Kaneko, M., Goto, T., Kozuka, J., Magi, R. (2012). Application of Kanji Writing Training Method Using Auditory Method in Children with Developmental Dyslexia. Higher Brain Function Research, 32(2), 294-301.
20. Haruhara, N., Uno, A., Kaneko, M. (2005). Experimental Kanji Writing Training for Children with Developmental Dyslexia: Effectiveness of Training Methods Based on Cognitive Function Characteristics. Phoniatric and Logopedic Medicine, 46(1), 10-15.