Spectral JPEG XL image format introduced to efficiently store data even about invisible light

Intel scientists have developed a format for recording images called Spectral JPEG XL, which allows data to be recorded in a wide range of the spectrum beyond the standard set of red, green, and blue. Even areas invisible to the human eye are supported.

Image source: jcgt.org

In science and industry, there is sometimes a need to capture colors that the human eye cannot perceive, such as ultraviolet and infrared light, or the specific wavelengths that plants need for photosynthesis. Some cameras are designed to capture the subtle differences that make paint colors appear the way they should under a given light. Current formats for recording such information require recording 30, 100, or more data points per pixel, resulting in files that swell to several gigabytes in size—too large to store and analyze.

The solution was proposed by Intel scientists Alban Fichet and Christoph Peters, who developed the Spectral JPEG XL format, which can record spectral data while still supporting compression. Traditional digital image files record information about just three colors: red, green, and blue (RGB). This is enough for everyday photographs, but true color capture requires a greater set of details. Spectral images are more accurate because they record saturation not only in RGB, but also in tens or even hundreds of narrow wavelengths. This detailed information covers the visible spectrum, as well as near infrared and ultraviolet regions, allowing for more accurate modeling of how materials interact with light.

Such files store data not only in three RGB channels — there are many more of these channels, and each represents the intensity of light in a specific, very narrow range of wavelengths. The scientific work published by the authors of the project discusses images containing 31 channels, and even provides examples with 81 spectral bands. These channels must capture a wider range of brightness values ​​— standard 8-bit images are no longer sufficient, so 16- and 32-bit floating-point numbers must be used for each channel.

There are many practical applications for this technology. An automaker needs to accurately predict how paint will look under different lighting conditions. Scientists use spectral imaging to identify materials based on their unique light signatures. Renderers need it to accurately model real-world optical effects like dispersion and fluorescence. Astronomers analyze the spectral lines of a gamma-ray burst to identify the materials present in the explosion. The OpenEXR format used to store this data today was not designed with such broad requirements in mind, and existing lossless compression methods like ZIP do not reduce the data volumes significantly.

Spectral JPEG XL uses a method called discrete cosine transform (DCT). In simple terms, it works like this: when you look at the color transitions of a rainbow, you don’t need to record every wavelength to understand what you’re seeing. DCT transforms smooth wave patterns into wave-like components (frequency coefficients), which are added together to recreate the original spectral information. MP3 audio is processed in a similar way: instead of recording every tiny vibration as a separate sound wave, the format records the important frequency components that are perceived by the ear, and discards everything else. In the same way, Spectral JPEG XL records the data that determines how light interacts with materials, and less important details are compressed. The data is then evaluated by dividing the spectral coefficients by the overall luminance, so that less important information is not damaged as much during compression. The resulting data stream is fed into a codec, and instead of inventing a new file type, the standard JPEG XL image format is used, into which specially prepared spectral data is written.

As a result, the authors of the project managed to reduce the size of spectral images by 10-60 times compared to the standard lossless compression of the OpenEXR format — the file sizes became comparable to regular high-quality photographs. At the same time, important OpenEXR functions are preserved, including metadata and support for a wide dynamic range. Some information is lost during the compression process, but the format is designed to discard less noticeable details first — compression artifacts occur in less important areas, and important visual information is preserved.

There are still some limitations. Spectral JPEG XL will be widely used only if software tools are continually developed and improved; initial software implementations may require further development to fully exploit the format’s capabilities. Not everyone will embrace the lossy format; some fields that make particularly fine-grained measurements may need to explore alternative storage methods further. Spectral JPEG XL may initially be useful in scientific imaging and high-quality rendering; but many fields from transportation design to medical imaging continue to generate large amounts of data, and compression technologies may eventually find their way into these fields as well.