In January 2021, the EASIER consortium embarked on a journey of enhancing communication between spoken and sign language users. In this pursuit, during the 36 months of the project, the EASIER project partners have meticulously and collaboratively developed and enhanced a diverse range of assets, reflecting the partners’ shared commitment to improving accessibility and inclusivity. These assets include:
- Mobile application and the corresponding desktop platform that serve as the central point for bridging linguistic gaps between spoken and sign languages, providing a user-centered interface and the architecture for seamless translation across several spoken and sign languages.
- Multilingual translation models that support high-quality translation in low-resource language settings. The project introduced and improved three fundamental translation directions: Sign-to-Spoken, Spoken-to-Sign, and Spoken-to-Spoken. These models formed the backbone of EASIER’s translation capabilities, ensuring a comprehensive approach to language accessibility.
- Advanced signing avatar technology that provides superior visualization of sign languages. The project’s signing avatar, “Paula,” has undergone significant enhancements during the project’s lifespan. Paula now possesses advanced abilities to convey emotions. Additionally, language-specific strategies for mouthing have been integrated, enhancing the avatar’s expressiveness. Notably, Paula can process elements of the EASIER notation, resulting in improved prosody and clearer affect within signed sentence animations.
- Cutting-edge technology for affect extraction, encompassing voice, text, and video inputs. The incorporation of these innovative features enabled the recognition and portrayal of emotions, enhancing the depth of communication experiences: affect extraction from voice analyzes vocal characteristics to detect emotions; affect extraction from text processes written words to discern underlying sentiments; affect extraction from video captures facial expressions.
- Sign language recognition models that provide a bottom-up phonetic representation. EASIER employed cutting-edge 2D to 3D lifting technology, enhancing the recognition models. This advancement ensured a more accurate portrayal of gestures and expressions, contributing to clearer and more precise communication.
- Multi-sign language lexical resource with an interlingual index as the backbone – a machine-readable and semantics-based lexicon. The project added multiple European sign languages to the Open Multilingual Wordnet with the aim to establish connections between signs with shared meanings across languages.
- Web applications that allow post-editing of machine translation output, including NERstar which includes automated processes during import, editing, and export to enforce limits and optimize subtitles and glosses; AZDV which provides corrections as graphical diagrams interpretable by software, enabling avatar animations as replacements for erroneous portions of machine translation output; and SignTube, an editing tool for alignment between the subtitle file, time code, and sign language video.
- Best practices and harmonized guidelines for generation and annotation of new sign language resources and training resources. EASIER has emphasized best practices for data generation, storage, and reuse and provided guidelines that play a pivotal role in ensuring the integrity and accessibility of sign language resources, fostering consistency and quality. Additionally, EASIER actively participated in several standardization development organizations, contributing to the establishment of industry standards in the field of accessible communication.
Besides, EASIER has greatly expanded the network of stakeholders to understand and align with user needs and preferences. One of the key aspects of the project was the end user involvement, achieved through active engagement and gathering end user feedback from sign language communities in France, Germany, Greece, the Netherlands, Switzerland, and the UK. This work was coordinated by the European Union of the Deaf (EUD) who led the team responsible for providing the definition of evaluation metrics and the evaluation protocol, assessing user requirements, needs, and gaps, recruitment strategies of end user groups, as well as the performance of project evaluation studies. All together, EASIER engaged 152 deaf and hearing individuals across seven language communities in evaluation studies. Not only did this valuable community feedback significantly influenced the technology development process by shaping the design of the mobile application, avatar, and translation models, it also increased awareness and insights into these advanced technologies within deaf communities through their direct engagement.
Lessons learned
One of the biggest lessons learned throughout the EASIER project pertains to the quality of translation between spoken and sign languages: although the translation quality has not reached the satisfactory levels yet, the EASIER consortium members recognize the key factors and reasons behind it and understand what is needed to move beyond the current state-of-the-art. EASIER and its sister project SignON have paved the way for further developments and advancement in the field of sign language machine translation.
Furthermore, through participation in the project, many EASIER project partners, including the EUD gained important insights into how to best communicate complex topics surrounding sign language technologies to deaf communities in an accessible way, particularly how to frame evaluation of the work in progress. Throughout the project, it became clear that deaf media and communication professionals should have a central role in outreach activities to ensure smooth and efficient communication and dissemination to target end users. It also became evident that deaf communities are quite keen to engage with topics surrounding sign language technology when they are made accessible, and that they are eager to engage in open and honest discussions on the subject.
Outreach
EASIER has also achieved success on the outreach level. The project was highly committed to communicating about its work and disseminating its results to advance the research but also sensitize the hearing population about deafness and deaf culture and raise awareness among deaf communities about the state of the art technologies aimed at reducing communication barriers.
The project developed a bespoke visual identity that clearly conveyed the EASIER’s team optimism, creativity, and commitment to inclusiveness, and a multilingual project website characterized by light and responsive design, featuring a keyboard navigation bar and a dedicated customizable accessibility plug-in providing various accessibility tools. The EASIER website was selected as one of the three finalists of the 2021 .eu Web Awards competition, an annual online competition launched in 2014 by the European Registry for Internet Domains (EURid) to acknowledge the top .eu websites in various categories. EASIER was nominated within the Laurels category established for educational institutions, charitable organizations, and ongoing pan-European projects. Besides, EASIER ran three communication campaigns, released 56 news articles; produced numerous videos; published over 40 scientific papers; and organized 10 events, engaging large groups of stakeholders.
The community gathered around the project was multinational and uniquely interdisciplinary and comprised researchers, developers, interpreters, and the general public, including deaf and hearing citizens from across Europe.
Sustainability and exploitation of project results
Recognizing that innovation is an iterative process, EASIER has placed strong emphasis on the strategic reuse of its outputs and continuous improvement. The tools, models, and resources have been meticulously designed to be adaptable and extensible. By encouraging the reuse of project outputs, the EASIER consortium has fostered a collaborative environment where researchers and developers can leverage the results and achievements of EASIER as building blocks for their future projects. This cyclical process of reuse and improvement propelled sustained progress, creating a ripple effect of innovation within the accessible communication landscape. Through these collaborative efforts, EASIER has ensured that its contributions will serve as enduring resources, fostering a legacy of inclusivity and accessibility.
Further information on the work and results of the EASIER project has been provided in public deliverables, submitted to the European Commission.