Vedant Das Swain

LetterBoard

Improve rapid prompting method for autism through a responsive letterboard

Health Ubiquitous Reflection

Jan, 2017 - May, 2016

Introduction

The Rapid Prompting Method (RPM) was developed by an Indian mother, Soma Mukhopadhyay, to teach her non-verbal austistic son, Tito, to communicate. The main objective of RPM is to help “verbal and written expression”, i.e. to read, write and type.
Letterboards are key artifacts to facilitate learning through RPM. During initial phases the letterboard is used by educators (teachers and/or parents) to conduct lessons. In order to augment the evaluation method, by making it more data-driven we designed a prototype smart letterboard that captures inputs as well as logs performance to responsive interface in realtime.

My Role

Constructing this instrument was the core component of our coursework in Design of Interactive Environments. The initial user research involved all group members participating. The division of labour for the development phase was more specific. Broadly, it was built on 3 pillars; hardware, network and interface.

Development

In order to keep the prime artefact, the letterboard, similar to the traditional form we decided to take an embedded systems approach. Our hardware was rigged with circuits and augmented with an Arduino that communicates in realtime with a monitoring dashboard. In my role, I had to setup an intermediate server as well as the communication interfaces on the both the hardware and the UI side. Additionally, I was also responsible for determining the feasibility of other connected devices - that could help monitoring - during the early stages of development.

Prototype Designer

In the early stages of designing the interface of the dashboard I helped mockup the interaction flow in the low-fi to mid-fi span. This includes playing with the idea of syncing the output of two differnt sensors (eye tracking and touch) to represent a child's performance.

Note: These roles are not necessarily exclusive to me nor are they exhaustive in describing everything I did in the project

The Problem

How RPM Works

We were fortunate to get access to a private english lesson of an autistic child. The three major players of a session like this are the student, the facilitator and the parent. In some cases the latter two can be the same. Based on this, we can describe the communication between the players as follows:

Practitioners of this method are often self-taught or acquire this skill through workshops, or through other facilitators. Parents, for example, pick this up from facilitators, so that they are equipped to conduct lessons on their own. Our understanding is that continuous use of this method can lead to children communicating unassisted via the letterboard and eventually even move onto typing.

Observations

One of our major observations was the asynchronous relation between the child's cognitive ability and their motor movements, and how this influences their performance with during an RPM lesson.
In the case we observed, the child was in the verbal band of the autism spectrum. This helped us discern their general ability in spelling as well as constructing sentences. Often however, this would not entirely line up with the coordination of their input. To put it simply, the letter they want to indicate might be 'X' but their input would be at 'R'.
However, we could extrapolate that in the case of non-verbal autism, such an assertion is hard to make, since the child's general learning ability can be questioned, i.e. they letter they wanted to indicate is indeed 'R', even though that's not the correct letter.

Challenges

Based on our personal observations and informal interviews with the facilitator and parent we could identify certain painpoints of this method.
As a process, using RPM via letterboards to teach is very instinctive and intimate. Different individuals could have varying approaches to this, for example the facilitator we observed had developed a sing-song prompting style. Interestingly enough the parent did not use this method to nudge the child. The onus is on the facilitator to guide the child during their learning phase. However, the method of evaluating a child's perfomace through a session is rather rudimentary - pen and paper.

Observation

Evaluating the child's performance manually is complicated. Keeping a check on the child's interactions (eye movements, verbal cues and interactions) along with correcting the actual inputs (checking the spelling, grammar or structure) is a difficult to manage by single individual simultaneously.

Data

Apart from manual, self-reported markings and annotations there is no convenient way of tracking a child's itneractions. This is essential for revising the facilitation of prompts by understanding how the child makes errors. Some attempts using cameras have been made to mitigate this, however these can often be obtrusive and disrupt the sessions entirely.

Participation

Involving a parent into the RPM session could be considered a means to distribute the load on the facilitator. However, we learnt that physically co-locating a parent in the same space could distract the child (if they are not the facilitators themselves). This makes it complicated to provide parents and other aids some form of reflection with regard to the child's goals.

Apart from these challenges other issues that may arrise are specific to the child's inherent or learnt skills upto that point.

The Design

Early Prototype

The initial idea was to combine two modalities, touch and sight, to monitor the child's interactions with the letterboard. This was motivated by bridging the discripancy in the cognitive and physical expression of the child. Often autistic children know which letter to select, which is evident from their eyes. However, their motor movements are not able to sync the movements.

We decided against this approach due to the hardware overhead that comes with a eye tracker. Beyond the technical complications that arise with embedding a wireless eye tracker on a letterboard, there are design constraints that are breached. The smart letterboard needs to be similar to a standard letterboard - in form as well as function. Eye tracking hardware would appear as distracting peripheral, that disrupts the child's primary task - selecting the correct letter.

Final UI Prototype

1. Facilitator inputs words into the list using the web interface
2. Click on the word created in the list view to begin testing
3. The selected word will show up on the screen and wait feedback
4. The child answers the question by touching the interactive letterboard
5. The touched letters will be sent to the web interface
6. Correct letters will show “Green” and incorrect will show “red”
7. In real-time, the touched letters will be shown in “orange"
8. When the word is finished being spelled out, another word is clicked
9. Repeat for all the words in the list

Adding Words
The facilitator will be able to add words into the list. When each of the words are clicked, the session will start in which the autistic child has to spell the word by selecting letters.

Data Metrics View
As data is being collected and recorded, these simple data representations will allow the facilitators to monitor the child's performance.

Realt-time Data Visualisation As soon as one of the word is pressed to initiate the session, the corresponding word will be displayed in a larger font. As the letters are being touched to spell the word, the corresponding box will be filled with green (correct) or red (incorrect)along with the letters will be shown as well.
The array of boxes are scrollable, enabling the clinicians or the caregivers to browse through the aggregated data.

Visual Metaphor
there will be a metaphoric design of the letterboard (hardware) to allow clinicians to easily see which letters are being touched currently. The last letter touched will be displayed in orange.

The Development

Architecture

Our main objective was to integrate an existing piece of hardware with a sleek interface for the facilitators, teachers and parents of autistic children. A well designed user interface and sophisticated hardware artefact needed to communicate in order to work in unison. This requires a quick and efficient backend that is reliable and easy to configure.
For the purpose of our work we decided to develop our backend server on Flask - a lightweight Python based framework. This backend server is based on the REST API architecture and allows both GET and POST requests on specific handles. These requests are used by the interface and the responsive letterboard respectively to communicate with the server. In order to support quick real-time transactions between what a child touches on the board and what is displayed - the backend maintains a simple tuple based data structure that keeps a record of < timestamp, character >.

The frontend JS uses regular interval AJAX query to continuously “ping” the backend with GET requests and receive the < timestamp, character >, that is accordingly updated on the user-facing interface. The JS is also robust enough to check the incoming characters from the letterboard with the characters that should be spelt based on the word the facilitator inputs. This happens with the help of using two different pointers on two different arrays - one for the word entered by the teacher and one for the incoming characters from the letterboard. This dual cursor correction allows us to measure the difference between selected and actual character, which in turn helps us calculate the accuracy of the child’s response in real time.
By design this server can be easily hosted locally on the personal interface by the users. However, this is also light enough to push onto a hosting app and use a simple cloud data structure like SQLite.

Hardware

Our responsive letterboard has three major components:

• MPR121 Capacitive Touch Sensor Breakout
• Feather HUZZAH ESP 8266 Wi Fi module with built-in battery charging and JST jack
• Lithium Polymer Battery

The capacitive touch sensor used here has twelve channel which means it can handle twelve individual touch pads. Since there are twenty-six letters in our board, we have considered specific grid architecture that enables us to use only eleven channels of the sensor. Adding a Wi-Fi module into circuit allows sending collected data from capacitive touch sensor to the facilitator’s laptop. Since an ESP8622 Wi-Fi microcontroller is embedded in Feather HUZZAH, the microcontroller can be programmed using Arduino IDE. To make the board easily portable, a 3.7 V Lithium polymer battery is used to manage power.

The hardware part of this interactive letterboard consists 3 specific layers and an isolator layer. The first layer from the front is a wooden frame with which the user interacts. It appears as an alphabet stencil, much like the classic letterboard.
The second substrate is a grid of aluminum foils attached to the intersection of conductive wires in rows and columns.
Third substrate is combination of five rows and 6 columns. The touch interaction on two intersecting touch channels and provides a tuple that is interpreted as an alphabet.
There is an isolator sheet between the conductive wires in the rows and columns. The holes in the isolator sheet directs the touch to a specific intersection and blocks interference in capacitance from other channels.

Schematic Design, CAM file for printing the board

Printed Circuit Board

Future

Improvements

The current form of the responsive letterboard is simply a proof of concept to showcase the potential of a simple instrument.
Some directions of improvement are:

• The design of the letterboard itself can be refined to have completely hollow letters, so that it imitates a standard letterboard closely. Especially because children often like the tactile feedback of inserting their finger or pencil through the letters.
• Storing the performance data of a child across multiple sessions. Holding onto such information can help reveal trends, recurrences and outliers in a child's interaction behaviour.
• Exploring the posibility of acquiring the facilitators interactions as inputs, for example their prompts. Understanding this can provide a richer representation of a learning session.