Mixed reality: revolution in virtual production environments

The director of photography Juan Luis Cabellos, AEC delves into virtual or mixed reality production environments (understood as integrating virtual reality and real elements), increasingly common in the audiovisual industry.

The electronic “forillos” made with large LED screens where backgrounds filmed in 2D are projected They are nothing new and they have been in use for some time now. The mixed reality, however, it does introduce important new features that represent an improvement in this technique and imply changes in the way of understanding production design.

Técnicamente, consiste en una enorme pantalla LED en la que se proyectan escenarios virtuales creados en 3D y un sistema que ubica la posición de la cámara frente a la pantalla. La clave está en la recolección y gestión de determinada información de la cámara: posición, altura, ángulo de visión y ajustes de la lente entre otros. Con estos datos, el sistema Unreal Engine of Epic Games es capaz modificar el escenario 3D y adaptar la perspectiva de la proyección para que sea coherente desde el punto de vista óptico. El escenario se mueve siguiendo a la cámara, dando así la sensación de que los actores están realmente insertos en esos entornos.

From the point of view of cinematography, there are several advantages that the system provides. First of all, the light is interactive. This is surely one of its main attractions. Everything that does not appear in the painting is a source of light, both on the walls and on the ceiling. This fits with the lighting logic of a space created in 360º. We also have options to work with it: from modulating its intensity, to creating windows of any color to illuminate or using black stripes. All of this is modified regardless of the exposure of the background that appears in the plan.

Another great advantage is the ability to freeze time. There are situations in which, filming outdoors, we know that it will be very complex to maintain the connection light. Sunsets or sunrises are the great challenges. Now you can shoot full-time during the witching hour.

Regaining full control

For cinematographers, a no less important aspect is the recovery of total control of the aesthetics of what is shot: both the foreground and the backgrounds. When working with color schemes, it may happen that the backgrounds are not ready before filming or, simply, that it has been decided to change or modify them in the editing process. This is always a feared situation. Now what you see and illuminate is what you take with you. No later surprises.

On a technical level, the screens LEDs have certain characteristics. At a minimum, they must be able to reproduce the gamut of the DCI standard to have the greatest number of color nuances possible. On the other hand, it is necessary that the distance between pixels be as close as possible to avoid seeing the plot. For example, the LED screens that have been used for the series The Mandalorian of Disney+ have a pixel separation of 2,84mm. Likewise, its high maximum brightness level, between 1000 y 4000 nits, guarantees the possibility of having highlights in the background.

Another key is the resolution with which the screens work, and the fact of offering a definition above 4K garantiza un nivel de detalle satisfactorio en los fondos. Es muy interesante como trabaja el sistema para optimizar el procesamiento de la imagen proyectada. El compromiso consiste en renderizar a máxima calidad solo lo que se ve en el cuadro más un cierto margen de seguridad -esto es lo que se denomina frustrum-. El resto se visualiza a una calidad inferior para aliviar dichos procesos de procesado de imagen. Que el fuera de campo tenga menor resolución no es relevante, porque sigue cumpliendo perfectamente la función de iluminar.

Producciones en realidad mixta

A nivel organizativo, las producciones con realidad mixta requieren que el diseño de producción de adapte a sus necesidades concretas. Los fondos han de estar terminados para el inicio del rodaje de modo que el equipo de VFX ha de incorporarse al principio. La ventaja respecto al sistema de trabajo tradicional es que los efectos digitales se ruedan en cámara lo que hace que el flujo de trabajo de postproducción más eficiente.

It is easy to assume that this new technique of filming in virtual production environments will become more and more common. While it is true that it is not the answer to everything - we are not going to stop filming in natural locations or sets - it is an excellent tool for filming situations that are impossible for a conventional production. Without a doubt, mixed reality expands and increases the narrative possibilities and improves the finish of audiovisual products.