Children born with cleft lip and palate (CL/P) disorder go through several years of corrective surgeries, dental procedures, and jaw correction. At a very young age, starting from 3 years and above, these children suffer from varying degrees of speech impediment among other problems. Even though it is one of the most common birth defects recorded in the world, the treatments are not equally accessible to all the children suffering from it. Based on socio-economic backgrounds and the geographical location, the necessary facilities may or maynot be available to these children.
The ongoing technological developments in this field are very sophisticated, but are also expensive and inaccessible to most speech clinics. By introducing a robotic solution that tackles the problem in a micro level and caters directly to each CL/P child, their families and their speech therapist, we bring the technology to people’s doorstep in not just an accessible way but also in a very tangible way.
Our design solution introduces Buddy, a robot for CL/P children that can provide contextual speech therapy in a gamified story-building format using audio and visuals. To motivate and encourage the children socially, Buddy provides a community setting for the children to build adventures together with their family and friends in a collaborative environment. Within these story building sessions the children are prompted to use and practice tough speech therapy words and vowels. Buddy provides real time visual feedback as a reference to improve articulation of words and guiding the children in enunciating the tough words.
To gain a deeper understanding of the problem, we conducted contextual inquiries with two craniofacial experts in speech pathology - Dr. Rita Patel and Dr. Steven Lulich, and interviewed a person born with the CL/P disorder and the parents of a CL/P child. We also performed literature reviews of over 8 papers related to speech therapy and the psychological impact of CL/P on the children and their families, reviewed over 11 clinical video documentaries included in The clinicians guide to treating cleft palate speech and referred to over 3 video diaries of CL/P children.
What we found was very profound -
While standard speech therapy includes frequent practice of difficult vowels and words, kids find it difficult to stay committed to regular practice sessions and track self-improvement. Since the children lack a clear understanding of the proper tongue position in relation to the right sounds that they should make when they speak, they are unable to gain a clear sense of how to articulate vowels and tougher words.
It is currently also hard for therapists and the parents of the child to track and evaluate the efficiency of the homework.
We were introduced to a newly developed biofeedback volumetric image system that uses ultrasound to capture the oral movement of a person while they are speaking to provide a real-time visual feedback of their speech. A sophisticated tool that provides efficiency in diagnosing, evaluating, and correcting the speech performance of the CL/P children, but this system is not readily available at most clinical facilities.
%20copy.gif)
Social engagement plays an important role in the psychological development of CL/P children.
Our interview with the CL/P participant revealed that she had grown up feeling isolated and different from everyone else. Our conversations relayed the importance of having strong familial and social support in the development of such children. Without proper help, these children are more likely to build a complex around this problem and feel alienated from the community around them for being born different.
In the current process, the speech therapist provides homework for the CL/P children, the purpose of which is to help speed up the therapy process and improve the pronunciation of tougher words. The homework usually consists of articulation of vowels and difficult words that have been introduced in the clinic. Using Buddy, our storytelling robot we will bring a higher level of engagement, tangibility, and dynamism in the homework sessions by leveraging the robot’s friendly personality and its ability to provide gamified visual and audio feedback. Buddy’s story building format also brings an equal participation from the family and friends, which provides a platform for the CL/P children to be socially engaged and feel encouraged as they learn. The importance of this is highlighted by the conversations we have had with our CL/P participant.
Buddy would be introduced to the child with CL/P by the therapist in the clinic. Buddy will then identify and remember the child as the primary user and the child will get to take Buddy home and introduce the robot to his/her family and friends. During the story building time, Buddy will project a child-like personality and initiate conversations within the group by providing a story prompt. Using a combination of audio and visuals drawn from a central library of animated resources, Buddy will engage the group in the storytelling activity.
-01.png)
These are therapy words that the speech therapist assigns to the child for homework practice. Buddy will introduce these magic words at the beginning of every story-building session.
When the child pronounces them correctly, these words
will act as triggers that move the stories along. E.g. Opening magical trap doors, casting a spell, and so on.
Buddy will facilitate the story-building process within the group (CL/P child, family & friends) using story prompts.
As we learned through our research, this type of collaborative engagement with their family and friends can help the CL/P children navigate the psychological issues that come with the social anxieties of having CL/P.
Buddy will provide the CL/P child with a real-time visual biofeedback for proper speech every time the child pronounces the magic words.
This feature is inspired by the biofeedback technology that has proved to be beneficial in clinics, but is inaccessible for children in their homes. With Buddy we are looking to develop a more accessible format of visual feedback for these children.
During the clinical visits, the speech pathologist will be able to access the recorded speech patterns to track improvement that the child makes and for further evaluations with the child and the parents.
-2.jpg)
I am a second year master's student graduating from the HCI/d program at Indiana University Bloomington. Before becoming a designer, I worked as a character animator for five years during which I envisioned and created character performances for TV shows and Movies. The skill I gained from this professional experience helped me bring our robot Buddy's personality alive. Over the last two years, I have worked on several products spanning across customer experiences in digital products, CRM softwares, and provided strategic design solutions for both customer facing and B2B product companies like Salesforce and Fourkites Inc.
Buddy is the current big project in my pipeline, which is an innovative solution in development for a problem that is common but is generally overlooked.
I'm Kathy Li, a second-year master student in HCI/d program at Indiana University Bloomington. During these two years, I was lucky to work for the City of Bloomington, IN, eBay, and musical.ly as a designer. Prior to joining the master program, I had experience in shipping two features and one new app. The new app, live.ly, became #1 in AppStore(US) a week after launch and hit 2 million downloads within a month. Meanwhile, after working for musical.ly as the first product designer hired by the leaderboard, I became more interested in building successful products and technological entrepreneurship opportunities.
In our first prototype which we demonstrated at the 2017 Fall Symposium, we built the visual components to simulate the social interaction of Buddy. We tested the look and voice of Buddy as it engaged with patrons at the symposium. We also prototyped a basic visual biofeedback system that could recognise pitch and provide feedback on how to correct the pitch, and used a smartphone for the facial simulation of Buddy.
We created character facial animations for Buddy using AfterEffects and complied the dialogue and acting into an android application to project Buddy's social interaction capabilities.
We used a Mac to mirror the smart phone screen and operate the animation instances to simulate the illusion of conversations with the patrons.
We designed the voice interaction of Buddy to resonate with children and their families in a comfortable way. To create the voice of Buddy, we collaborated with Char Wozniac from the Cognitive Development Lab at Indiana University, an expert in child psychology and baby talk. During our recording sessions we explored effective ways to encourage and motivate children when providing story prompts in the speech activity sessions and the language and voice modulation of Buddy when it communicated errors or success in pronunciation of difficult words.
To ensure a delightful experience for the child and his/her family, we designed a natural and intuitive social interaction for Buddy. During the conversation, Buddy will turn the head towards the person who talks and hold their gaze. The head movement includes turning right and left, and also looking up and down. At the same time, Buddy will also recognise the primary user as the child with CL/P and focus its attention on the child instead of the family or friends when it provides its visual feedback. In the first prototype, we used an Arduino circuit and a joystick controller to 'Wizard of Oz' the interaction.
During the story telling activity, Buddy provides a visual reference for the child to follow as he/she pronounces the 'magic words' (therapy words), as a form of speech instruction. When the child pronounces the words, an animated feedback of visuals play in the form of a celebratory response for right pronunciation.
When the pronunciation is not satisfactory, Buddy encourages the child to repeat the magic word by saying - "Let's try again, together!", and provides a visual enunciation reference for specific parts of the word.
The figure to the right demonstrates Buddy's visual feedback for the magic word 'SWEET'.
The body of our prototype consists of a wooden frame that was laser cut and assembled to hold our circuit and battery pack in place. We have two servo motors driving the 3D printed head and neck of our robot to move in two degrees of motion - up/down, and right/left, that we can control using a joystick. The base of our robot revolves on a slip ring to display how Buddy will be able to rotate in all angles and interact with people when placed in between a group. In our next prototype, we are working to make Buddy's structure lighter and with a softer outer body to make Buddy more approachable for children.
In our next iteration, we hope to develop Buddy’s capabilities in identifying and engaging with group participants that comprise of both the CL/P child and his/her families. We also plan to integrate a voice recognition and recording system to help therapists track improvements that the child makes in the homework sessions.
Assistive physical artefacts such as a mirror and a ‘speech buddy articulation tool’ will provide children with a better base for efficient homework practice. The tangibility of a robotic structure will make this feature more accessible and contextual to homework practice.
In this phase we will explore ways in which the speech therapist can easily access the data collected by Buddy on the progress that the child has made so far. This will be valuable for the therapist to structure and define the therapy process further.
We will design and develop Buddy to recognise and identify multiple people in the group that it interacts with.
We will evaluate Buddy with our user groups to identify how to make the interaction more secure and how much control the parents of the child will have in deciding when Buddy will record and evaluate the activities.
Our long term goal is to make Buddy accessible, affordable and contextual to the needs of CL/P children in other countries where speech therapists might not be available.
We see the scope of Buddy expanding into language and accent learning and other speech disorders in future.
We plan to bring in more support in the further development of Buddy's physical build to help us move to the next phase.
We will be replacing the current smart phone feature for the face with an actual screen and developing the required back end code to support the implementation
To fully integrate Buddy's personality into the robot and to provide a platform of story prompts, we will be developing a complete library of animations that can be accessed for contextual responses by Buddy.
We will also be developing and integrating the biofeedback system completely into Buddy's interactions. At this stage we expect to have Buddy recognise the primary users (CL/P children) and distinguish them from the secondary users (family and friends)
To further expand on Buddy's capabilities, we will be integrating a speech recognition software. In future, we plan to use machine learning to provide a more sophisticated feedback and data to the human speech therapist for further evaluation.