New immersive captioning tech
The package of training and software has been developed by experts in drama and performance at the University of Nottingham, in collaboration with Red Earth Theatre, a pioneering company based in the East Midlands.
The experts have worked with digital researchers in the University’s Mixed Reality Lab to develop the new projection mapping software. It can be used with traditional theatre projection systems to make captioning part of the artistic design of the show using any part of the set, props and even actors themselves.
The new tech also allows much faster generation of captions via an easy import of text from a Word document – where scripts often reside – rather than someone having to hand type sections of script in multiple files.
The first production to showcase this creative way to display the words and sounds of a show is the epic Arctic adventure story, Soonchild by Russell Hoban. The book has been adapted for the stage by Red Earth Theatre and is currently on tour in the UK. The technology will be made available open source to other theatre companies with training and the loan of hardware included once the tour is finished.
A special performance of Soonchild will be staged at Nottingham Trent University’s Waverley Studio theatre on Thursday 17th October at 6pm to celebrate 21 years of Red Earth Theatre and showcase the new technology.
Head of Drama and Creative Writing at the University of Nottingham, Professor Jo Robinson, said: “We were really conscious that for deaf audiences performances are not generally immersive, as you need to keep switching focus between the action that’s happening on stage and the captions at the side or bottom of the stage. So our question was, can we put the captions into the action and make them an integral part of the production on stage?”
“Our answer was to enlist the skills of researchers in our Mixed Reality Lab to develop new software to allow theatre designers to do just that. It was essential to involve deaf people in the project as well and we are thrilled that we have been able to bring everyone’s skills and ideas to life in this ground-breaking new package of captioning technology.”
Dr Paul Tennent from the University’s Mixed Reality Lab explained the technical process: “A key innovation we have made is to simplify the process of projection mapping. Almost every set designer makes a scale model (either digital or physical) of the set before building the real thing. We make use of these models to create a 3D digital model of the set which we project right onto the real set. We then align the physical space with its digital twin. The process of alignment is simply a matter of clicking on specific locations on the real physical set, then the system can use this knowledge to align the digital twin in 3D space, so the projection is perfectly mapped regardless of the position of the projector or set pieces in a fraction of the time taken with traditional projection mapping software.
“We are able to manipulate the digital version, for example by adding captions, graphics, or video to it, and those objects appear on the projection, and subsequently on the real set, in the correct place and accounting for occlusion and projector position. Then we simply hide the digital twin, leaving only the captions/graphics displayed on the set.”
Dr Pierre-Alexis Mével from the University’s Department of Modern Languages has drawn on expertise in screen translation to develop guidance for legibility and positioning of creative captioning in live performances. He has also produced a short video featuring creative captions to be used by the company and the venues to which they tour in order to build awareness of what captions can do.
Red Earth Theatre has a pioneering track record in making integrated and accessible theatre using sign language and projected captioning as part of the action. Co-Artistic Director, Amanda Wilde, said: “This research project with the University of Nottingham has encouraged us to evaluate our current access provision and to be more creative with how access is integrated into our future touring shows. As a small-scale touring company committed to providing access for D/deaf audiences, it has afforded us the space with our research colleagues, to be more ambitious and inventive in developing our access offer and to push the boundaries of captioning in theatre. Because the work is research-based we are allowing ourselves to challenge current captioning conventions and test whether our ideas work with audiences. The research element of the project gives us the opportunity to both excel and fail in equal measure, a factor most important to testing new ideas and new approaches to access.”
The project has been funded by the AHRC and EPSRC as part of their ‘next-generation’ immersive experiences funding stream. This provides opportunities to multi-disciplinary research teams to promote and develop immersive experiences in the performance industries.
New immersive captioning tech