RCUK Digital Economy Programme: The main partners in this Digital Economy Project are Dundee University’s Computing, Aberdeen University’s Department of Computing Science and Capability Scotland. We will work closely with Dr Nicola Grove, the director of OpenStoryTellers and Communication Matters (ISAAC UK)
Introduction
Being able to tell stories about ourselves is a central part of the human experience and of social interaction. Most people do this naturally, for example while chatting with family members over the dinner table. But telling stories about oneself can be a real struggle for people with complex communication needs (CCN); they find it very difficult to create and articulate such stories. People with CCN (i.e. individuals with severe physical and communication impairments and possibly varying degrees of intellectual disability, e.g. due to cerebral palsy) rely on computer-generated synthetic speech. Speech generating devices are currently limited to short, pre-stored utterances or tedious preparation of text files which are output, word for word, via a speech synthesiser. Restrictions in speed and vocabulary can be a frustrating experience and are an impediment to spontaneous social conversation.
This project is a follow on to the feasibility study “How was School Today…?” where we wanted to see if we can help children with CCN create stories about what they did in a day by developing a computer tool which produces a draft story based on knowledge of the user’s planned daily activities (e.g. from a diary) and automatically-acquired sensor data; and also an editing and narration tool which lets the user edit the story into something which is his/hers and not just a computer output.
Please contact Annalu Waller (email: a.waller@dundee.ac.uk or see other contact details below on the people section of this site) for more information.
People
Prof Annalu Waller
Principal Investigator
Professor of Human Communication Technologies
Contact: a.waller@dundee.ac.uk
Phone: +44 (0)1382 388223
Prof Ehud Reiter
Co-Investigator
Professor of Computing Science
Rolf Black
Researcher Co-Investigator
Rehabilitation Engineer
Dr Ross Turner
Research Fellow
Dr Nicola Grove
Honorary Research Fellow
Contact: info@openstorytellers.org.uk
Web: http://www.openstorytellers.org.uk/
Dr Nava Tintarev
Research Fellow
Daniel Herron
Research Student
Dr Joe Reddington
Research Assistant
Meg Mitchell
Research Student
Stephen Knox
Research Student
Jay Bennet
Research Student
Publications
2012 Book Chapters
“Personal storytelling for children who use augmentative and alternative communication”; A. Waller and R. Black; in “Using Storytelling to Support Children and Adults with Special Needs: Transforming lives through telling tales“, Nicola Grove (ed); 1st Ed.; Routledge, Oxford, UK; ISBN: 978-0-415-68775-1 (Order your copy here: Amazon.co.uk).
2012 Journal Papers
“Supporting personal narrative for children with complex communication needs”; R. Black, A. Waller, R. Turner and E. Reiter; ACM Transactions on Computer-Human Interaction (TOCHI) 19(2) (2012), ACM, pp.15. DOI: 10.1145/2240156.2240163. (Download the paper here: Rolf’s Computing page).
2012 Conference Proceedings
“Automatic Utterance Generation for Personal Narrative: System development and feasibility experiences”, R. Black, A. Waller, E. Reiter, N. Tintarev; Communication Matters National Symposium (2011), Leicester, UK, 23-25 September. (Abstract, pdf)
“Automatic Utterance Generation for Personal Narrative – System Development and Feasibility Experiences”; R. Black, A. Waller, E. Reiter, N. Tintarev; 15th Biennial Conference of the International Society for Augmentative and Alternative Communication (2012). Pittsburgh, USA, 28 July – 4 August. (Abstract, pdf).
2011 Conference Proceedings
“Connected Computer Aided Communication”, R. Black, D. Sloan, A. Waller; Digital Engagement 2011, the second meeting for the Digital Economy community in Newcastle, UK, 15-17 November (2011). (Local copy, External copy, online proceedings)
“How was School today…?” A Prototype System that uses a Mobile Phone to Support Personal Narrative for Children with Complex Communication Needs, R. Black, A. Waller, E. Reiter, N. Tintarev, J. Reddington; Demo Session at the Second Workshop on Speech and Language Processing for Assistive Technologies (SLPAT) (2011), Edinburgh, Scotland, UK, 30 July. (External copy, online proceedings)
“A Mobile Phone Based Personal Narrative System”, R. Black, A. Waller, N. Tintarev, E. Reiter, J. Reddington; The 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) in Dundee, UK, 24-26 October (2011). (Local copy, External copy, online proceedings)
“How was school today…? – Using a mobile phone to support data collection for automatic narrative generation”, R. Black, A. Waller, E. Reiter, N. Tintarev; Communication Matters National Symposium (2011), Leicester, UK, 25-27 September. (Abstract, pdf)
“”Hands Busy, Eyes Busy”: Generating Stories from Sensor Data for Automotive applications”, N. Tintarev, J. Reddington, E. Reiter, A. Waller, R. Black; The 3rd International Workshop on Multimodal Interfaces for Automotive Applications (MIAA) in conjunction with IUI 2011, the International Conference on Intelligent User Interfaces in Palo Alto, CA, 13 February (2011). (Local copy, External copy, online proceedings).
2011 Academic Theses
“A case study of aided story telling: Comparing the Step-by-Step™ with the How Was School Today Prototype”, J. Menjivar Dominguez, K. Yläneva, Degree of Master of Science in Speech and Language Pathology (2011), Uppsala University, Disciplinary Domain of Medicine and Pharmacy, Faculty of Medicine, Department of Neuroscience, Logopedi. (Download the thesis here).
2011 Other Public Outputs
Nicola Grove, Martine Smith, Annalu Waller, Rolf Black. (2011). Talking About Experience – A practical workshop on using AAC equipment to support Personal Narrative. AAC Study Day in collaboration with Communication Matters: London (14/06/11).
2010 Conference Proceedings
“A Personal Narrative Support Tool for School Staff and Parents of Children with Complex Communication Needs”, R. Black, E. Reiter, A. Waller; RAatE Conference & Exhibition (2010). Warwick, UK, 29 November. (Abstract, pdf)
“A New Tool to Support Interactive Narrative Experience for Children with Communication Disorders”; R. Black, A. Waller, E. Reiter, R. Turner; 14th Biennial Conference of the International Society for Augmentative and Alternative Communication (2010). Barcelona, Spain, 24-29 July. (Abstract, Slides, pdf)
“Using NLG and Sensors to Support Personal Narrative for Children with Complex Communication Needs”, R. Black, J. Reddington, E. Reiter, N. Tintarev and A. Waller; First Workshop on Speech and Language Processing for Assistive Technologies, SLPAT 2010. (2010). Los Angeles, USA, 5 June. (External copy, online proceedings).
2010 Other Public Outputs
“Finalist: Community Project of the Year”, R. Black, A. Waller, E. Reiter, R. Turner, BCS UK IT Industry Awards (2010)
“Outstanding ICT Learning Initiative of the Year”, TES Schools Awards (2010)
“How was School today…?” – A Prototype System that Uses Environmental Sensors and NLG to Support Personal Narrative for Children with Complex Communication Needs. System Demonstration at SLPAT 2010; R. Black, J. Reddington, E. Reiter, N. Tintarev and A. Waller. (2010).
2009 Conference Proceedings
“How was School Today…?, Evaluating the Personal-Narrative-Telling Prototype: Preliminary Results”, R. Black, A. Waller, E. Reiter, R. Turner; Communication Matters National Symposium (2009), Leicester, UK, 20-22 September. (Abstract, pdf)
“Using NLG to Help Language-Impaired Users Tell Stories and Participate in Social Dialogues”, E. Reiter, R. Turner, N. Alm, R. Black, M. Dempster, A. Waller; ENLG2009, 12th European Workshop on Natural Language Generation (2009). Athens, Greece, 30 & 31 March. (Paper, pdf).
2009 Other Public Outputs
EPSRC Final Report for “How was school today…?” – Supporting narrative for non-speaking children, a feasibility study. PDF document: Dundee, Aberdeen
A. Waller, R. Black and N. Grove. (2009). Enabling children with complex communication needs to tell stories – Study Days in collaboration with Communication Matters and Great Ormond Street Hospital: London (27/01/09), Manchester (28/01/09) and Edinburgh (03/02/09).
2009 Press
- BBC News: National coverage on Radio 2 and 3, Television: BBC 1, Reporting Scotland New software helps children speak (BBC News, Health: Video report shown throughout the day including the 10 o’clock news.), Software ‘gives children a voice’ (BBC News, News Frontpage)
https://www.youtube.com/watch?v=5-cnRx29SKA
- STV news at Six – Central edition (6pm):
https://www.youtube.com/watch?v=iazZmD7mOFw
- The Times: in print on Friday, 5 June 2009 and online: TimesOnline: How was school today? Now disabled pupils can tell the story
- The Guardian: guardian.co.uk: Device offers hope to children with communication problems
- The Herald: in print on Friday, 5 June 2009, and online: Web Issue: Scientists’ new device gives gift of speech to children unable to talk
- The Press and Journal: System developed to help children communicate
2008 Conference Proceedings
“Supporting Personal Narrative for School Children with Communication Disorders – the Development of a Prototype”, R. Black, A. Waller, E. Reiter, R. Turner; RAatE Conference & Exhibition (2008). Coventry, UK, 1 December. (Abstract, pdf)
“Tell me about your day: creating novel access to personal narrative”, R. Black, A. Waller, E. Reiter, R. Turner, A. Braekkan, M. Duffy and J. Shannon; Communication Matters National Symposium (2008), pp.28. Leicester, UK, 21-23 September.
“Tell me your story: A storytelling workshop”, R. Black, A. Waller, E. Reiter, R. Turner, A. Braekkan, M. Duffy and J. Shannon; Communication Matters National Symposium (2008), pp.19. Leicester, UK, 21-23 September
“Across the lifespan: Parents and families of people who use AAC: A new research agenda”, B. Hemsley, J. Goldbart, A. Waller and K. Launonen; 10th Biennial ISAAC Research Symposium (2008). Montreal, Canada, 8-9 August.
Timeline
April 2019: Share My Day
Jay Bennet is working on his Honours Project “Share My Day – How was School today, 10 year on“. The Android App uses RFID tags, voice recordings and photographs in a new update of the original “How was School today…?” software.
November 2012: Education Technology
Read more about the use of tablet technology in AAC and our projects, such as “ECHOES2” and “How was School today…?”. You can read online or download a copy here.
July 2012: On the News
| The Dundee AAC Group and the Straight Talking Group have made it into the news again. STV News at Six reports about our upcoming trip to ISAAC in Pittsburgh. Alongside a 1-hour presentation on “How was School today…?”, the groups will present a number of research projects and personal stories around AAC. Find out more on our Facebook event page or directly at the conference website: http://isaac2012.org/.Here a link to the complete STV News at Six from Wednesday, 25th July 2012. We are at about 8’40” into the programme. |
June 2012: HwSt goes Mobile
| Daniel Herron graduated this month with his Honours Project “How was School Today?” Goes Mobile. The Android App allows the recording of voice messages and photographs to support children with complex communicaiton needs to talk about their personal experiences. He developped the app in close collaboration with a special school to ensure that it meets the requirements of real school life. For a quick overview please take a look at his poster from the Honours project exhibition: Daniel Herron’s Project Poster (PDF).Find more information on our projects on our research group site: Dundee AAC Group. |
February 2011: ISAAC 2012 in Pittsburgh
| We have been accepted for ISAAC 2012 in Pittsburgh: Please take a look at our abstract: Automatic Utterance Generation for Personal Narrative – System Development and Feasibility Experiences.We will be presenting about several projects of our research group, please check our website for more information: Dundee AAC Group or like us on Facebook for the latest updates: Dundee AAC Group, How was School today…? |
November 2011: In the Press
| Take a look at the Guardian online: Technology that gives children a voice. |
November 2011: Dundee Science Festival
| Spreading the word: During the Dundee Science Festival we presented our work in the Overgate, Dundee, speaking to more than 500 people… |
September 2011: Communication Matters National Symposium
| We will present the latest results of the project.You can download the abstract here: Rolf Black, Annalu Waller, Ehud Reiter and Nava Tintarev: How was School today…? – Using a mobile phone to support data collection for automatic narrative generation |
August 2011: We’re going to ASSETS 2011
| We will be presenting at ASSETS 2011 in Dundee, 24-26 October 2011: “A Mobile Phone Based Personal Narrative System”, Rolf Black, Annalu Waller, Nava Tintarev, Ehud Reiter and Joseph Reddington. Will we see you there? |
April 2011: Microsoft Trendbook Zukunft Bildung (Future Education)
| How IT changes our Life – IT Trends for the Education of Tomorrow. Our latest research project “How was School today…?” has made it into the Microsoft (Germany) Trendbook “Future Education”. Download the PDF here. |
February 2011: “Talking About Experience: Personal Narratives Workshop”
| Talking About Experience – A practical workshop on using AAC equipment to support Personal Narrative. Following on from the Story Telling Study Day in 2009, this workshop will offer an opportunity for teachers, speech and language therapists, parents, family members and carers who work with children and adults who use AAC and/or learning disabilities to engage in the practicalities of using mid- and high-tech AAC devices to support the development and sharing of personal narratives. This event will include both traditional platform presentations and practical workshop sessions. |
February 2011: Call for papers now open: SLPAT at ACL EMNLP 2011
| Second Workshop on Speech and Language Processing for Assistive Technologies (SLPAT). ACL EMNLP 2011 Workshop, 30 July 2011, Edinburgh, Scotland. This 1-day workshop will bring together researchers from all areas of speech and language technology with a common interest in making everyday life more accessible for people with physical, cognitive, sensory, emotional or developmental disabilities. This workshop will build on the first such workshop (co-located with NAACL HLT 2010); it will provide an opportunity for individuals from both research communities, and the individuals with whom they are working, to assist to share research findings, and to discuss present and future challenges and the potential for collaboration and progress. |
January 2011: Collaboration with Uppsala University in Sweden
| We are working very closely with the Speech Pathology Departement (Logopedi, Department of Neuroscience) at the University of Uppsala, Sweden, and this year two of their Master students have written their thesis on our project. Please take a look at their thesis at this link. |
September 2010: BCS UK IT Industry Awards 2010
| We have been nominated as finalist in the BCS UK IT Industry Awards 2010 in the category “Community Project of the Year”: Please take a look at the list of finalists. (Press release) |
August 2010: RAatE 2010
| We are going to present at RAatE (Recent Advances in Assistive Technology & Engineering). Please take a look at the conference website: RAatE 2010. |
June 2010: TES Awards 2010
| Fantastic news: Our collaborators won the TES award for: OUTSTANDING ICT LEARNING INITIATIVE OF THE YEAR! Congratulations! Please take a look at the TES website and a follow-on article in the TES Scotland Magazine. |
April 2010: SLPAT 2010
| We are going to present on the First Workshop on Speech and Language Processing for Assistive Technologies (SLPAT) in Los Angeles this June. Take a look at the NAACL HLT conference website: NAACL HLT 2010 and the workshop website: SLPAT 2010. You can download our abstract here here.We will also deomonstrate our prototype on the day. You can find out more about this in our demonstration proposal. |
March 2010: ISAAC 2010
| We have been accepted at ISAAC this summer in Barcelona. Take a look at ISAAC’s conference website: ISAAC 2010 and download our abstract and slides in Publications. |
Januar 2010: We have officially started part 2
| We have officially started our project. We are one of three projects that have been funded as part of the Digital Economy Programme.More official info at the EPSRC website: “The Dundee part” and “the Aberdeen part”. |
November 2009: Article in the University publication “The Bridge”
| Launched in 2008 the Bridge features more commissioned articles, original writing and focuses on the achievements of University alumni, staff and students. Read online or download the October issue with an article about our project on page 14. |
October 2009: “Research in the Wild” – Successful bid for funding!
| We managed to secure funding for another 18 months! Our project proposal was accepted for the Digital Economy Call “Research in the Wild”. |
October 2009: Impact! World
| Explore the impact of the Digital Economy Programme research at www.impactworld.org.uk. (Find us in the archived projects.)We also feature in the latest “Impact” newsletter: View the email on the web here or download the EPSRC pdf version) |
September 2009: Communication Matters National Symposium
| We have been accepted with our talk about the latest results of the project.You can download the abstract here: Rolf Black, Annalu Waller, Ehud Reiter and Ross Turner: “How was School Today…?”, Evaluating the Personal-Narrative-Telling Prototype: Preliminary Results |
August 2009: EPSRC – Annual Report 2008-09
| Our project is featured in the EPSRC annual report. Quote: “The ‘Digital Economy’ encapsulates research and development that delivers novel design or use of technologies to help transform the lives of individuals [… and] may lead to many life-changing breakthroughs from projects such as: […] – A computer tool incorporating GPS and barcode scanners to help children who cannot speak create a story about their day at school.” (download an extract of the report here. |
July 2009: Choice
| Capability’s magazine Choice features an article about the project at their school. Quote from one of the participants: “It made me feel really good about myself and I would definitely like to use it again.”(download an extract from the issue here or the whole issue (pdf, 18MB) directly from Capability (Choice website) |
June 2009: Press Release
A revolutionary software system which enables children with communication difficulties to take control of conversation has been unveiled at Capability’s Corsford School. Please download the press release here: Pioneering Scottish Technology Empowers Disabled Children (PDF) or take a look at the colaborators’ pages: University of Dundee, Press Area, Capability Scotland, Press Area or University of Aberdeen, Media ReleasesSome of the press coverage, including radio and television:
For more information please contact Prof Ehud Reiter (technical questions) and Prof Annalu Waller (assistive technology and disability issues). |
May 2009: Follow-up Project
| As part of their NLG Platform Grant at the Department of Computing Science, University of Aberdeen, we will be looking into a more long term use of our personal narrative telling prototype |
March 2009: 12th European Workshop on Natural Language Generation
| We have been accepted with our paper on the project. |
November 2008: AAC Study Day
| As part of this project there will be an AAC STUDY DAY in collaboration with Communication Matters and Great Ormond Street Hospital for Children in early 2009 titled:”Guess What Happened Today!” Enabling Children with Complex Communication Needs to Tell StoriesIt is going to be a valuable study day on the role of story telling within language development, and how story telling can be encouraged and facilitated.Presenters will be: Nicola Grove, Annalu Waller and Rolf Black.The study day will be held on three days in different parts of the UK which should allow everyone interested to participate:London, 27 Janurary 2009 Manchester, 28 January 2009 Edinburgh, 3 February 2009 |
June 2008: Communication Matters National Symposium
| We have been accepted with our talk about the project. Additionally there will be a workshop on personal narrative.You can download the abstracts here: Annalu Waller, Rolf Black, Ehud Reiter and Ross Turner: Tell me your story – a story telling workshop and Rolf Black, Annalu Waller, Ehud Reiter and Ross Turner: Tell me about your day – Creating novel access to personal narrative |
FAQ
What is How Was School Today trying to do?
- Our goal is to help children who cannot speak (or who have very limited speech) tell simple stories about their day at school, to parents, carers, and teachers. In this project we are focusing on children with cerebral palsy.
Why are we doing this?
- Children with severe language and communication limitations find it very difficult to have conversations with people, instead they are often limited to passively responding to questions (“yes”, “no”, etc), or communicating simple needs (such as “I am hungry”). This limits their communication and self-confidence, and in the long term (as adults) can lead to social isolation.
- Our long-term vision is to develop a suite of tools, which can be used by children and adults of varying levels of disability, which help people engage in social conversation and story telling, and which also help teachers and therapists teach these skills. If successful, we believe this will lead to major quality of life improvements for people with major communication disabilities, and also for their parents, family members, and friends.
How does “How was School Today…?” work?
- Sensors track the location of the children in their school, and also who they interact with and (to a limited extent) what they do. The computer software distills from this data a simple story about the child’s day, using artificial intelligence and natural language generation techniques. A user interface then lets the child to edit the story (for example taking out things he or she doesn’t want to talk about, and adding comments like “This was fun”), and then to tell it a parent or other adult.
What is the status of “How was School today…?”
- Some media reports suggest that the system is complete and ready to be used throughout Scotland or the UK. This is not true, unfortunately. We need to spend at least another two years on research and development before this can happen. We need to make the system more robust, reduce the amount of technical support required, give the children more control over the stories (including the type of language as well as the content), adapt our interfaces so they can be more easily used by children with varying levels of impairment, etc, etc. We also need to test the system on more children, for longer periods of time, to assess long-term impact.
Who is involved in “How was School today…?”
- How Was School Today is a collaboration between the University of Dundee, the University of Aberdeen, Capability Scotland, and Communication Matters. It has been primarily funded by the EPSRC/RCUK Digital Economy programme.
Links
Our Voice
Our software now speaks using the unique Heather Scottish voice by CereProc. The voice has quickly become very popular on VOCAs (Voice Output Communication Aids) in Scotland.
Our Logo
For our logo I used two icons from the Nuvola icon theme for KDE 3.x by David Vignoni (http://icon-king.com/?p=15). I found them at http://commons.wikimedia.org/wiki/Nuvola.
The idea is that normally a school diary is used to communicate to the people at home what happened during a school day. And in our project we will design a prototype that allows the child to tell their own story using the new device.