1. Introduction

Over the past few years, terms such as "Ghost-Frame," "Frame-remapping," and "PhantomTrack" have emerged, each reflecting variations of a core innovation: Sub-Framing Technology. This technology enables more seamless integration of virtual environments, addresses technical constraints, and enhances the overall experience for both crew and talent on set. 

2. What is Sub-Framing Technology?

In standard LED volume setups, the LED screen displays content at a specific frame rate, for example, 50 or 60fps. The camera captures each frame in sync with the screen's frame rate. Sub-Framing Technology introduces a more dynamic system, where each frame divides into multiple sub-frames (or "slices") that can display different content sequentially, such as a static color alongside video footage.

The number of sub-frames is hardware-dependent and can range from 2 to currently max. 15 at a 60fps frame rate, which increases at lower frame rates. Using sub-frames creates an environment where multiple visual elements—such as video content and chroma-key layers—are displayed simultaneously. By adjusting the camera's shutter angle and/or timing, you can select which sub-frames you capture, allowing for varied use cases and configurations.

There are two primary setups:

1. Multi-Camera Setup: One camera captures the normal video feed, while another camera, using an offset reference signal, captures a different sub-frame, such as a static color or a different video feed.
2. Single Camera Setup: One camera records multiple sub-frames, producing separate video streams for each, creating various output streams in real-time.

3. Applications of Sub-Framing Technology

Sub-framing opens up a wide range of possibilities in virtual production and broadcasting. While each application can be used independently, they are often combined to address specific technical needs.

Naked Eye View

This feature enhances crew and talent interaction by displaying content such as AR markers or text overlays on the LED screen that are invisible to the camera but visible to the talent. Examples include:

• AR Markers: Helps actors interact with virtual elements by providing visual markers that don't interfere with camera recordings.
• Text Overlays: Displays helpful information such as auto-cue, countdowns, or instructions for on-screen talent, improving performance without needing additional reference monitors. 

Camera Tracking

Traditionally, camera tracking requires additional hardware like special cameras or reflective markers placed around the stage. Sub-framing technology simplifies this by displaying tracking markers directly on the LED wall, invisible to the audience but detectable by tracking systems (e.g., Mo-Sys, Stype, Zeiss), eliminating the need for extra markers, improving accuracy, and simplifying setup.

Chroma Recording

Green screens are often used to film scenes with post-production effects. Sub-framing allows simultaneous capture of virtual content and a chroma-key layer using the same camera, blending the advantages of real-time virtual production (ICVFX) with the flexibility of green screen post-production.

Multi-Source Broadcasting

Sub-framing technology can broadcast multiple versions of content simultaneously. For example, different languages or regional sponsor advertisements can be displayed and captured, tailored to different audiences in real-time.

Multi-Camera

Broadcasting with multiple cameras on a virtual set typically requires precise coordination between the cameras and the media server. Sub-framing allows each camera to capture its unique view simultaneously, eliminating delays or inaccuracies during live switching and enabling seamless transitions between multiple camera angles.

High Frame-Rate Capture

While sub-framing itself isn't focused on slow-motion, the technology has significantly improved LED panels' ability to display and record high frame-rate content, achieving frame rates up to 360fps with moving content and 900Hz with static content, without issues like scan lines or flickering.

Hidden Feed (GhostFrame)

One of the most well-known applications, GhostFrame, displays hidden video content or chroma keys by showing an inverted sub-frame that blends into a white background when viewed by the naked eye. This feature is handy for regulatory compliance, such as hiding specific advertisements from live audiences while displaying them on a broadcast.

The downside is that the inverse will give a slight whitewash to your LED screen and might flicker more noticeably to some people. This feature should only be used on the best-performing LED panels, like the BP2V2 or RB1.9BV2 or -V3, as maximum performance is required to tune the sub-frames in a way that minimizes the flickering and is not noticeable. 

4. Required Equipment

LED Screen

The quality of the LED screen is pivotal to the success of sub-framing technology, and this involves two key components: the LED panels and the LED processor.

LED Panels
The performance of the LED panels is crucial and should not be overlooked. Selecting the appropriate panel driver configuration and ensuring precise PCB (Printed Circuit Board) design is essential. This allows the panels to handle high data rates and maintain accurate timing, which is critical for sub-framing to function smoothly.

LED Processor
The LED processor manages the incoming video signal and controls the LED panels. Typically, the processor is produced by a different manufacturer than the panels, making it essential to choose a compatible combination carefully. The processor's role is vital in ensuring the panels correctly display the sub-framing content.

There are two primary approaches to processing in sub-framing technology:

1. Brompton, Colorlight, and Novastar Solutions: These processors use "Frame-remapping" or "Frame Multiplexing," where sub-frames are generated directly on the processor and then sent to the panels. This requires a higher data load, resulting in the need for more distribution boxes and additional processor capacity.
2. Megapixel's Camera Mode (GhostFrame): In contrast, Megapixel's solution offloads much of the sub-framing workload to the receiver card within the LED panel. Video streams are sent to the panels, but the receiver card directly generates elements like chroma colors, still images, or inverted sub-frames. This reduces data transmission requirements and processor load while maximizing the LED panel's potential, reaching speeds up to 900Hz.

Each approach has its strengths, and the choice between them depends on the specific requirements of the production environment.

Additionally, it is imperative to check the full panel performance. For most sub-framing applications, having as many sub-frames available as possible is essential, as this will provide flexibility in configuring the setup, maximizing the capabilities, and reducing possible flickering.

Camera 

Rolling vs. Global Shutter Cameras
Global shutter cameras are the preferred option for working with LED screens, as they minimize or eliminate issues like scan lines. However, rolling shutter cameras can still be used effectively with LED setups, although they typically require more careful configuration to avoid potential problems.

This is particularly important when using sub-framing technology. Most applications demand a global shutter camera to ensure smooth performance. Still, there are certain use cases where rolling shutter cameras can work, depending on the camera model and setup.

When using sub-framing, the camera's shutter must be adjusted and reduced to capture only the required sub-frames. As more sub-frames are configured, each will be visible for a shorter time on the screen, necessitating a lower shutter angle. Rolling shutter cameras, in particular, have a longer sensor readout time (the time required to capture light from the sensor), meaning their shutter angle might not be able to be reduced further to prevent capturing elements from other sub-frames, often leading to an increased risk of scan lines appearing on the camera feed due to the nature of rolling shutters.

Single-Speed vs. High-Speed/Slow-Motion Cameras

Most cameras are "single-speed" by default, meaning they record and output one video signal at a given frame rate. This is sufficient for many sub-framing applications, such as multi-camera setups or hidden markers, eliminating the need for more expensive high-speed cameras.

However, high-speed cameras, like the Sony HDC3500 or HDC5500, Grass Valley LDX150, and AtomOne SSM500, offer additional capabilities by generating multiple video outputs. These cameras, including the new PhantomTrack solution from RED, produce "phase outputs" that all run at the same frame rate (e.g., 50fps) but are offset by fractions of a frame. Initially developed for slow-motion recording, this feature is ideal for sub-framing applications like multi-source recording, where more than one sub-frame needs to be captured. In such cases, high-speed cameras are essential for achieving the desired results.

Synchronization

As timing is crucial with sub-framing, it is essential to synchronize (genlock) all equipment within your setup. Investing in a good SPG (Sync Pulse Generator) can go a long way. When using multiple cameras, account for the need to offset the sync signal, which can sometimes be done on the camera but should otherwise be accounted for on the SPG. 

Lighting

Select your LED lighting carefully to prevent flickering on camera. The camera's reduced shutter angle, necessary for sub-framing, requires precise lighting setups to ensure consistency, avoid flicker artifacts, and provide enough brightness to light people and objects on stage.

Conclusion
By leveraging Sub-Framing Technology, production teams can achieve greater flexibility, enhance the visual experience, and streamline virtual production workflows. Whether it's camera tracking, multi-source broadcasting, or blending ICVFX with chroma-keying, sub-framing represents a crucial innovation in digital content creation and virtual production.