Vicomtech advances in the automatic generation of intra- and inter-lingua subtitles for live content

Within the European HBB4ALL project, Vicomtech-IK4 implements a pilot for the automatic subtitling of content in Spanish and translated into English.

Subtitling is the preferred and most economical method of making multimedia content accessible to people with hearing disabilities and exploitable in international markets where languages ​​other than the original are spoken.

Automatic subtitling that generates subtitles from the input audio without intervention is an increasingly popular solution. And the reality is that today there is no technology to generate automatic subtitles in real time but they continue to be generated manually.

The Vicomtech-IK4 technology center has been working for years on the development of technology for automatic subtitling based on automatic speech recognition technology and within the European project HBB4ALL it has implemented a pilot for the automatic subtitling of content in English broadcast in real time (intra-lingual subtitles). In addition, this pilot incorporates real-time automatic translation technology, which makes it possible for these subtitles to also be consumed in Spanish as they have been translated in real time from English (interlingual subtitles).

In addition to offering alternative content to the content that the user has chosen, this technology adapts to the characteristics of the device that the user is using. This is achieved thanks to the adaptive streaming technology that allows audio, video and subtitle streams to be adapted independently to the characteristics of the end device, such as the resolution of the device or the quality of the connection.

This pilot is the result of a European project in which Vicomtech-IK4 participates to integrate accessibility technology to multimedia content produced under the HBBTV standard, technology for interactive television that aims to merge digital television content with web content.

Specifically, Vicomtech-IK4 has integrated technology for generating intralingua (in the original language of the content) and interlingua (translated) subtitles, generating a video stream in real time that can be consumed on any device, including televisions connected to the Internet. All this in order to add new experiences and environments to viewers.

The European project called HBB4ALL is led by the UAB and is made up of 12 partners who are experts in accessibility and multi-device environments, including universities, television channels, production companies, research centers and SMEs. Among the partners, in addition to Vicomtech-IK4, we can highlight the German technology center, IRT, Screen Subtitling Systems or the four television production companies RBB, RTP, TVC and SwisTXT, among others.

The intra-lingual subtitle generation technology comes as a result of the European SAVAS project led by Vicomtech-IK4, which aimed to develop automatic subtitling technology for 7 European languages. In the same way that the automatic subtitle translation technology comes from another European project also led by Vicomtech-IK4, the SUMAT project.

It should be noted that the high need for automatic subtitling is a reality, since there is a European audiovisual law of 2010, which affects all Member States, by virtue of which there is an obligation to increase subtitled content progressively until reaching 90%. Currently, this percentage is not met, and the objective of the law was to achieve it in 2013, but the reality is that until now there is no technology on the market that helps with automatic and real-time subtitling. Although failure to comply with this law is not penalized today, accessibility to content is a pending issue and there will come a day when all content has to be accessible and if it is not, sanctions will begin. Therefore, technology is essential to help create subtitles automatically.

Legal framework

In recent years, audiovisual accessibility legislation is becoming more restrictive. As a consequence of Article 7 of the European Audiovisual Communication Services Directive, Member States have been taking the necessary measures to ensure that the services of audiovisual providers under their jurisdiction are gradually accessible to people with hearing or visual disabilities.

In accordance with the European directive, in 2010 the Cortes definitively approved a new General Law of Audiovisual Communication at the state level, which imposed content accessibility obligations on all television operators for the first time. Specifically, the established accessibility measures affected the field of subtitling, audio description and sign language.

Despite the existence of this law, today the vast majority of television channels fail to comply with these obligations in terms of the percentage of content that must be broadcast with accessible information (more than 90% in subtitling).

For pre-recorded or semi-live broadcast content, subtitling is performed and launched manually.

On the other hand, for live content, some televisions already integrate automatic speech recognition technology, but always using the re-speaking technique, in which a professional re-speaks the content against a recognizer trained for their voice and in a controlled acoustic environment. "This method guarantees good results and a low delay, but it incorporates the difficulty of cognitive fatigue of the professional, who on average only supports 30 consecutive minutes of speech, being solved through substitutions. In addition, it is worth mentioning the high pricing of these professionals," they say from Vicomtech.