Peppy:A Paper-Based Augmented Reality Application to Help Children Against Dysgraphia
ABSTRACT
Over the years, researchers have found an increase in learning problem among young children especially related to writing. Among these is a drop in dexterity or developmental dysgraphia due to a lack of fine motor skills. Also discovered is that handwritten activities on paper help work out these problems and with the advancement in technology in today’s day and age, better combative measures can be taken against these problems. With Peppy, a mobile application using augmented reality (AR), our aim is to fight these problems using technology as well as paper by augmenting it into something that is both interactive and useful for child development. Although educational AR applications already exist, they don’t focus on improving children’s fine motor skills using paperbased exercises. Peppy brings enjoyable, thought-provoking and intriguing paper prototypes consisting of colouring, games, and puzzles to life through AR.
Introduction
Dysgraphia, the disorder of written expression is characterized by writing skills (that) are substantially below those expected given the person’s chronological age, measured intelligence, and age-appropriate education. People with this ailment generally suffer from poor motor planning rather than linguistic impairment. It is estimated that the prevalence of developmental writing disorders is about 7-15% among school-aged children [8]. Therefore, it has become paramount that combative solutions be found. Handwriting can be improved through several mechanisms, one of the most common being handwriting drills [3]. However, static, non-interactive content coupled with the requirement of sustained attention for handwriting performance [5], among other factors, means that a more interactive invention needs to be developed. With more evidence in favour of the conventional wisdom that puts paper and pen as central to handwriting efficacy [10], there is a need for a hybrid system that augments paper-based interventions. This might be effective in combatting developmental dysgraphia as augmented reality engages learners in rich spaces for learning [2], alongside providing an optimal medium through which a system can be developed to maintain both the efficacy of paper for developing handwriting while adding interactivity. In education, AR has been widely used with several learning approaches such as game-based learning where learners are engaged with authentic experiences, problem-based or even simulation-based systems which allow for maximal impact. Studies have shown that the use of AR provides attractive learning experiences and unique educational benefits [4] as well as makes tangible user interfaces (TUI) for learning affordable and easily accessible. Systems such as IoT-lens and the ARCA environment by the University of Essex aim to tackle different sections of academic approaches [11] using AR. AR has also been used in tandem with paper-based interfaces as shown by Bonnard et. al [1], where geometry was taught through paper objects. With spatial Augmented reality, work has been done within the enhancement of development of fine motor skills where Mueller’s studies show us how AR can be used as both a motivational and instructional paradigm [9]. Thus, we note that AR provides us with several advantages which range from motivation to a more efficient transfer of knowledge. This helps children combat developmental dysgraphia and helps in learning about the subject material they will interact with.
User Research
To understand the issue in greater depth, we used contextual Inquiry (CI) with semi-structured interviews of 7 teachers (t) and 3 parents (p). We visited 3 different schools and observed 60 children aged 3-5, while they were doing the activities provided by us which included dot-to-dot, colouring, and maze solving, Fig. 2. These activities are part of occupational therapy to help improve handwriting skills. Our focus was on finding the problems associated with paper-based media for child development, especially in improving hand dexterity. In half of the classes we first gave a tutorial on how to solve a particular activity and in the other half, we had them attempt the activities without explicit directions. We asked teachers not to actively help children in order to find their strengths and weaknesses. During our CI we collected and analysed 100 sample activities attempted by the children in order to fully gauge and/or corroborate the drop in fine motor skills.
The activity samples were marked on 4 aspects, Conformity: how well does the student follow visual guidelines, Completion: does the child complete the task, Quality: how neatly are the lines and the colouring done and Comprehension: how well does the student understand the task. All worth 5 points. They were judged by one person only who followed a marking scheme different for both groups of students in order to ensure consistency. Since half of the children were not given a demo and the teachers were told not to help, comprehension and conformity measured how much the children understood about the activity. A score of 5 was given to those who performed the task in the most optimal way expected from children of this age while 1 was given to those who attempted to do something in the right direction. Quality and completion analysed whether the children were able to make smooth, unbroken lines and complete the task with neat colouring.
The schools were divided into two groups: “elite school” (expensive private schools that use smart technology in classrooms) and “non-elite schools” (affordable schools where technology is not used in the classrooms). The students’ scores in “non-elite schools”, compared to that of “elite schools” were consistently higher with an average of 12.83/20 while the elite schools had an average of 9.81/20. We noted that these activities aimed at children seemed to lack interactivity and failed in maintaining their attention as the completion rate was low. This led to an overall lack of quality, Fig. 3, especially in “elite schools”, Fig. 4. The “non-elite” school, however, did relatively better, Fig. 5. The lack of dynamism also lent itself to lower transfer of knowledge as we noted that students also scored lower on the measure of conformity. This seemed largely due to the lack of visual cues or dynamic guidance that persists with traditional paper-based activities. We furthered our inquiry with teachers and parents and found that almost all teachers except two within the elite school agreed that they noticed a considerable drop in dexterity for children as compared to previous years. Teachers also felt that the prevalence of touch surfaces could be a contributing factor, as it is compounded with the reduction of attention spans within the current student body by “keeping them distracted” as said by t2. According to t5, “children are more interested in watching things on multimedia rather than working on boards.” Nonetheless, they felt that that the use of paper-based activities was key in helping prevent developmental dysgraphia and the current state of paper-based activities was not at par with the children’s expectations. The teachers also felt it important that a “different” system be developed to help not only attract children’s attention but also challenge them to improve their concentration and cognitive strength.
​
​
​
Design and Implementation
After user research, we went through the process of brainstorming and paper prototyping and after multiple iterations, we came up with Peppy to help children’s fine motor skills, cognitive functioning, attention span and learning. There are three known types of learning modalities: visual, auditory, and kinaesthetic [7] (a person's awareness of the position and movement of body parts by means of sensory organs in the muscles and joints), so we also incorporated video, audio, and touch to make the application easier to understand and more interactive. We also decided to create an activity book that would be of 20 - 30 different exercises for the children to practice. We needed to develop mechanisms that would maximize scores within conformity, completion, quality, and comprehension. To maximize conformity, we added a video tutorial, a screen with the finished activity and a scoring mechanism. Each activity has a video tutorial that shows how it can be done through distinct steps which can be followed in order to successfully complete the activity. This would also help with comprehension. A finished model of what the activity page would look like is displayed while the child carries out the activity, so they get an idea of how to complete the activity successfully. For the scoring, models that closely resemble the complete activity are used as image targets, so scoring is done based on a similarity between the attempt and the “ideal” activity. To maximize both completion and quality we used our scoring mechanism. As the children would begin to move through the steps of the activity, the scoring mechanism responds with increased scores while providing encouragement through auditory responses. Further incentives in the form of the AR experience and the mini-game, at the end, help in motivating the children to complete the activity so as to unlock these incentives. Moreover, the AR experience is designed to maximize comprehension to help the child visually understand how the activity translates into the real world hence increasing their knowledge through the addition of information within the AR model.
​
We chose a system consisting of the following processes: Once the book is acquired (Fig. 6), it gives instructions on how to download the application. The application has a simple interface that comprises of clearly distinguished items which will have page numbers and a small preview of the activity. Upon selection, a video tutorial plays which shows the users the steps needed to complete the activity after which there is a screen which shows the finished activity along with options to scan said activity or receive help. Once the AR camera is used, there are two possible outcomes. If the activity does not achieve the completion goal the users will receive output in form of stars to keep score and encouraging messages as pop-ups designed according to the activity along with audio to motivate the children to keep working. If the desired quality of the activity is reached, the outcome is the augmentation of the object in AR. The AR has additional features like facts about the object and audio tailored to the object in order to enhance learning. An option to play a minigame related to said activity is then present on the augmented reality screen through which the user would get the reward and incentive to complete newer exercises. All these user interface screens are illustrated in Fig. 7 on the next page
​
We developed the application for an Android mobile operating system, using libraries available for augmented reality within the Unity game engine. We had to select activities that would have a wide range of difficulty but also would make it easy enough for children to complete them correctly. Consequently, for our first exercises, we chose a dot-to-dot colouring activity of a cardinal bird since the purpose of the activity is to help with the aforementioned problems. Colouring and dot-to-dot books provide good paper pencil activities for fine motor and eye-hand development [6]. The choice of such a model helped us make the experience more realistic and immersive and thereby more interesting for children. We developed a connected system where every object on the paper is transferred to the digital world (mobile application), where it is then augmented in 3D in the real world, after which the said object is used in a mini-game. We aim to ensure that the children remain incentivized to complete the activity and by extension, increase the amount of time and concentration they put into the task. Alongside this, we also hope the process becomes more interactive and playful, all the while providing a cognitive challenge as well.
​
​
Evaluation and Results
The participants of our user testing consisted of 21 children (C) of ages 3-6 from different schools, environments and habits along with 16 parents (P) and teachers (T), different from those in user research at either their homes or in school at different times. It consisted of a briefing, prequestionnaire, the activity divided into tasks and a post-questionnaire. We used the constructive interaction method (users are required to work in pairs to discuss and make a mutual decision on how to proceed) in order to note the dynamics of the teacher-child and parent-child partnership required when using said system. The users were given an overview of the purpose of the book and the application without revealing how to exactly perform the tasks. The questionnaire had queries related to their environment, previous knowledge of AR and questions about the children’s habits of using technology and writing. These questionnaires were largely filled by the adults. The users were given a scenario where they had received/bought the activity book and had to discover how to use it to its full extent. They were then given their tasks one at a time. If the users were stuck, they were given approximately 3 minutes before being given a hint. After the testing users were given a post-questionnaire on the usability and design of Peppy and its future aspects.
​
When asked about the fine motor skills of children, even this group of parents and teachers either agreed that they had deteriorated or had no opinion, but none disagreed with the notion. According to the parent’s feedback, children with an average age of 4.5 spent an average of 2 hours per day using technology and only 1 hour a day on paper-based activities on average. On evaluating the design, we asked both adults and children if they liked the cover, colour, look of the book and the application, to which most of the test users had a positive response. Most users had not used AR before and so when some of them skipped over the instructions on the book and went directly to the activity page, they were confused about its connection with the application. The overall usability scores given by the users remained middling with an average of 7.2/10. Some found the application “highly intuitive” as mentioned by T6 while others felt that it requires “more explicit instructions and guidance” according to P1, P7, and T3. However, users felt that the application was effective and highly pleasing as the children were interested in and enjoying the 3D object. C3 even wanted/tried to “pet” the cardinal and C16 asked if he could “play it at home as well.” The parents and teachers also thought of this as useful and a good way of helping children, noting the idea as very “engaging” as stated by P8, and according to T1 “if done right could help a lot of parents and teachers” while further agreeing that this system would be useful as a combative measure for developmental dysgraphia. Of our 16 participants, 94.7% of the users said that the application and book would be useful as a means of improving the hand dexterity of children, and 89.5% would like to use the application again. One of the teachers, T10, even said that she “can’t wait to see this application at school level in my classroom”.
​
Despite its successes, our study faces some limitations that must be improved. We noted that in our limited time span we received a positive reaction from all the teachers we interviewed regarding the use of AR in order to help improve the fine motor skills of children. They largely agreed to its efficacy as both a motivational and instructional tool but to confirm our research so far, we require a future study that follows students who show symptoms of developmental dysgraphia and decrease in dexterity and fine motor skills and use Peppy over an extended period of time (approximately 3 months). This would help us in determining Peppy’s effectiveness and prove the usefulness of hybrid systems in the realm of child development and early education.
Conclusions
We looked at the issue of developmental dysgraphia and its prevalence within children. We corroborated the lack of interactivity within paper-based media and its effects through our user research in which we found low to middling levels of interest and efficacy among the children when it came to attempting the paper-based activities. We presented a hybrid solution which would incorporate AR in order to solve these issues while adding more utility in the form of an active tool for positive reinforcement and a mechanism for better transfer of knowledge. In our testing, we found that this system shows great promise with agreement from our users that it could be highly effective not only as a combative measure for developmental dysgraphia but also as an active learning tool during child development. With refinements in both the interface with respect to user complaints with lack of affordances and experience, and with a reduction in error rate of scoring, Peppy could have applications both in the home and in the classroom due to its mix of paper benefit and interactive learning of technology. According to T14, “When technology goes hand in hand with paper, we can achieve our goals”. In the future, we hope to make Peppy an everevolving platform, one which continuously develops newer learning exercises and educational materials that could then be made more interactive using augmented reality.
References
[1] Bonnard, Quentin, et al. "Paper interfaces for learning geometry." European Conference on Technology Enhanced
Learning. Springer, Berlin, Heidelberg, 2012.
[2] Bronack, Stephen C. "The role of immersive media in online education." The Journal of Continuing Higher Education
59.2 (2011): 113-117.
[3] Crouch, Alyssa L., and Jennifer J. Jakubecy. "Dysgraphia: How It Affects a Student's Performance and What Can Be
Done about It." TEACHING Exceptional Children Plus 3.3 (2007): n3.
[4] Dayang R. A. Rambli, Wannisa Matcha, and Suziah Sulaiman. 2013. Fun learning with AR alphabet book for preschool
children. 2013 International Conference on Virtual and Augmented Reality in Education 25: 211-219.
[5] Feder, Katya P., and Annette Majnemer. "Handwriting development, competency, and intervention." Developmental
Medicine & Child Neurology 49.4 (2007): 312-317.
[6] https://www.edubloxtutor.com/fine-motor-skills-handwriting/
[7] Jeffri, Nor Farzana Syaza, and Dayang Rohaya Awang Rambli. "Design and Development of an Augmented Reality
Book and Mobile Application to Enhance the Handwriting-Instruction for Pre-School Children." Open Journal of Social
Sciences 5.10 (2017): 361.
[8] Katusic S. K., Colligan R. C., Weavers A. L., Barbaresi W. J. (2009). Forgotten learning disability - epidemiology of
written language disorder in a population-based birth cohort (1976-1982), Rochester, Minnesota. Pediatrics 123 1306
1313. 10.1542/peds.2008-2098
[9] Mueller, Florian, et al. "Hanging off a bar." CHI'12 Extended Abstracts on Human Factors in Computing Systems.
ACM, 2012.
[10] Valderrama Bahamóndez, Elba del Carmen, et al. "Analysis of children's handwriting on touchscreen phones."
Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services.
ACM, 2013.
[11] Wang, Minjuan, et al. "Augmented reality in education and training: pedagogical approaches and illustrative case
studies." Journal of Ambient Intelligence and Humanized Computing 9.5 (2018): 1391-1402.