Suny Behar
Suny Behar

Suny teaches up-and-coming filmmakers at UCLA's Graduate School of Theater, Film and Television as well as serving as a consultant to companies like Canon, HBO and Google.

Understanding Canon Cinema RAW

January 31, 2014

This article was originally published on October 11, 2013 and has been updated to include current product information.

As digital cinema camera technologies continue to evolve, one pattern has clearly emerged as the new flavor of our time: recording RAW camera sensor data. With the emergence of powerful post-production workstations, it makes theoretical sense that one would want to record all the unprocessed data the camera sensor has to offer and deliver it to post-production to process that data and create the final images we seek. Yet, in reality, the methodologies of implementing these steps seem as varied as the cameras themselves, suggesting that there are other factors at play that influence this seemingly straightforward concept.

In my daily dealings as a director, cinematographer and educator, I am astounded at the amount of confusion I see amongst producers, studio executives, clients and even my advanced graduate students at UCLA Film School when it comes to the topics of RAW, Uncompressed and Log recordings. These terms seem to be used interchangeably and often in the wrong contexts. The goal of this paper is to help demystify some of these terms by specifically analyzing Canon's implementation of Cinema RAW recording in its flagship camera, the EOS C500. By identifying and explaining all of the tenants that make up this novel approach to RAW recording, the hope is to empower creative storytellers to use this new technology to its fullest potential by capturing and delivering images of stunning depth and quality.

Is RAW really RAW?

According to the Merriam-Webster dictionary, the definition of RAW is: "being in or nearly in the natural state: not processed." Applying this to image capture would mean capturing all of the sensor data information and delivering all of it in it's "natural state" to post-production for processing. If this sounds familiar, it is because this implementation has existed for quite a while now in the photography world. Canon's own Canon RAW file format (.CR2) offers this workflow for photographers wanting to record unprocessed image data and later apply settings such as picture style, color space, color balance, contrast, saturation and tonal values. Applying this to images recorded at 24 fps should be an uncomplicated proposition then – and yet, it is not.

When we talk about recording untouched RAW sensor data, we have to realize that we are talking about an enormous amount of information. In this age of digital acquisition where we no longer have a negative to store in a vault, all of the data that is acquired has to be duplicated several times for backup and safety purposes, resulting in hundreds of terabytes worth of data that has to be archived. A still photographer is given a choice for every still photo that is taken. Do I need to record a RAW still image, knowing that a single photo on an EOS 5D Mark III at full-resolution RAW is about 27 MB per frame (processed at 16-bit per channel mode in Adobe Photoshop, this will become a 126 MB frame) or is it okay to apply compression to the image (JPEG for example) to bring the storage down to a 1/10th of the size given the delivery medium of the final image? This is a decision that is not offered to a cinematographer on a frame-by-frame basis, but rather on a project-by-project basis. How much compression am I willing to accept for the millions of frames I am about to record?

Camera manufacturers, such as DALSA, have gone down the path of offering complete RAW image pathways but have not encountered much success 1 and adoption in our industry2 , and the reality of RAW cinema recording has very few companies offering a pure RAW untouched option. The main reason for this is image size, image pipeline and storage. Producers and show runners are rightfully asking: "Will I really see that big of a visual difference if I record untouched RAW sensor data and multiply my storage costs tenfold?" and the honest answer is: “Not necessarily.”

Through techniques such as compression, gamma mapping and bit rate reduction, we can achieve images that are almost visually and imperceptibly different from their untouched counterparts. RED Digital Cinema pioneered compressing RAW recordings3 from 3:1 all the way to 18:1 using their REDCODE RAW Codec4 and all of the new digital cinema cameras from SONY (F65, F55, F5) only offer compressed RAW recordings in their RAW recording options at compression ratios of 3.6:1 (F65RAW-SQ5, F55RAW6, F5RAW7) or 6:1 (F65RAW-Lite8).

Because our concept and definition of RAW recording is now open to include compression, the question that begs to be answered is: "What else can we do to the RAW recording to ensure that we deliver the highest possible quality from the sensor in the most efficient and economical way?"

 


1 http://www.teledynedalsa.com/corp/news/169

2 http://www.btlnews.com/crafts/camera/the-lessons-of-dalsa-digital-cinema

3 http://www.google.com/patents/US8358357

4 http://www.red.com/tools/recording-time

5 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F65.shtml?PID=I:35mm_F65:F65#/f65t1_10

6 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F55.shtml?PID=I:35mm_F55:F55#/f55t3_2

7 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F5.shtml?PID=I:35mm_PMWF5:PMWF5#/f5t3_2

8 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F65.shtml?PID=I:35mm_F65:F65#/f65t1_10


 

Canon Cinema RAW

Canon has made a very late entry into the competitive field that is digital cinema. Being a latecomer has the disadvantage of having to play catch up with the leaders in the field, but there are also the advantages of learning from others and not being tied to any legacy standards.

The general philosophy behind Canon Cinema RAW seems to be centered around this concept: instead of capturing the native untouched sensor data at high bit depths and then compressing that data for recorded delivery, why not make sensor gain adjustments for user adjusted variables like ISO and White Balance at high bit depths and then deliver that data at a manageable bit depth, but in an uncompressed form. This approach is unique to Canon and worthy of some further analysis. Figure 1 shows all the characteristics associated with recording Canon Cinema RAW.

Format

Much like the famed Cineon format9 pioneered by KODAK10 to scan film negatives11, the Canon Cinema RAW format is captured in individual files where every frame is a separate file. These files are assembled in their recording order in what is called a “file stack.” So when you look at a shot captured in Canon Cinema RAW, you will find a folder full of individual .RMF files. You will notice quite a few similarities between the Cineon file format and how it handles film negative and the Canon Cinema RAW format and how it handles RAW data. They are both 10-bit file formats and both are encoded with Log data (more on that later). One of the differences is that Canon Cinema RAW, being a newer format, also allows the embedding of both audio and rich metadata in each individual file.

When discovering that the recorded .RMF files were "only" 10-bit, I initially wondered if this was enough bit depth to maintain high quality files. After all, while I understand that film was scanned at 10-bit in Cineon format for years without issue, aren't most of the other digital cinema cameras recording 16-bit RAW files? The answer is: sort of. The cameras that record 16-bit RAW files do so with varying amounts of compression, not uncompressed like Canon Cinema RAW. The Arri Alexa does record uncompressed RAW, but it does so in 12-bit Log format, not in RAW uncompressed 16-bit Linear12. We can quickly see by looking at the chart below how these image sizes compare.

 

Camera Codec Raster
Size
Compression Frame
Size
SONY
F65

F65RAW-LITE

8K

6 : 1

6.2 MB/
Frame13

SONY
F65

F65RAW-SQ

8K

3.6 : 1

10.4 MB/
Frame14

SONY
F5/F55

F5RAW/
F55RAW

4K

3.6 : 1

5 MB/
Frame15

RED
EPIC

REDCODE
RAW

5K

3 : 1

6 MB/
Frame16

RED
EPIC

REDCODE
RAW

5K

5 : 1

3.6 MB/
Frame17

Arri
ALEXA

ARRIRAW
16:9

2.8K

N/A

7 MB/
Frame18

Arri
ALEXA

ARRIRAW
4:3

2.8K

N/A

9.3 MB/
Frame19

Canon
C500

Canon Cinema
RAW

4K

N/A

11 MB/
Frame

 

From the table, we can observe the effects of recording an uncompressed RAW signal versus a compressed one. As it currently stands, the Canon EOS C500 records the most amounts of RAW data per frame of any of the current digital cinema cameras. The Arri ALEXA would record more data because it records in 12-bit Log, but due to its maximum raster size that’s limited to 2.8K instead of 4K, it maintains a file size of 7 MB/Frame in 16:9 mode. What this shows us is that you have to be careful for what you wish for. If we were to ask Canon to record at higher bit depths than 10-bit uncompressed in 4K, this would result in exponentially larger files that ironically would then have us asking them if there was any way they could compress them for us. The walk away here is that while the file container might be 10-bit, like Kodak's Cineon, the uncompressed nature of the four 2K streams packs a tremendous amount of sensor data per frame into Canon Cinema RAW.

Embedded Values

This is probably the most controversial yet innovative feature of Canon Cinema RAW.

If the definition of RAW is that of being strictly untouched sensor data, then embedding values seems to contradict this, but in fairness, so does applying lossy compression to the RAW signal. It is important to remember that at no point in the signal path is the sensor data debayered in-camera. As explained in a subsequent section below, the four parallel 2K data read-outs from the sensor are multiplexed and the resultant signal is recorded as an .RMF file stack where it still needs to be processed and debayered for viewing and use in post-production. Canon's viewpoint is that "a RAW signal is one where no process has been applied to the signals coming out of the image sensor that impair their dynamic range, their optoelectronic transfer function or their color space. The original sensor outputs can be completely recovered as linear light representations in the color grading process. " While debating the merits of what qualifies as RAW and what doesn't could be entertaining and nourishing to the soul, I will leave that discussion to people much smarter than I am. Instead, I am more interested in analyzing what is embedded, why Canon found it important to embed this information and how as a professional cinematographer, I can take advantage of this novel process to enhance the images I create.


 


9 http://www.cineon.com/file_format.php

10 http://motion.kodak.com/motion/uploadedFiles/US_plugins_acrobat_en_motion_support_dlad_cineonfileformat4.5.pdf

11 http://motion.kodak.com/motion/uploadedFiles/H-387_Digital_LAD.pdf

12 http://www.arri.com/camera/digital_cameras/workflow/working_with_arriraw

13 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F65.shtml?PID=I:35mm_F65:F65#/f65t1_10

14 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F65.shtml?PID=I:35mm_F65:F65#/f65t1_10

15 http://pro.sony.com/bbsc/ssr/show-highend/resource.solutions.bbsccms-assets-show-highend-F5.shtml?PID=I:35mm_PMWF5:PMWF5#/f5t3_1

16 http://www.red.com/tools/recording-time

17 http://www.red.com/tools/recording-time

18 http://www.arri.com/camera/digital_cameras/workflow/working_with_arriraw/recording.html

19 http://www.arri.com/camera/digital_cameras/workflow/working_with_arriraw/recording.html


 

Embedded Values: Canon Log

Figure 2 shows us that one of the key elements to the Canon Cinema RAW process is the gamma mapping or tonal mapping from a high bit depth linear image to a 10-bit Canon Log encoded image.

All digital cameras capture light in a linear fashion. When you double the light that hits the sensor, you get a doubling of the values recorded by the sensor. However, this is not how the human visual system functions. It also happens not to be how most visual display systems, such as televisions and projectors, are coded. When a light source is doubled, we tend to perceive a small increment in light output. As such, we are said to have a more logarithmic response to light rather than linear. Figure 3 shows Canon Log versus a Normal1 gamma curve. While these curves represent camera gamma curve picture styles, they are a good illustration of what a linear response to light would look like compared to a logarithmic response.

Because digital sensors do not capture light as we perceive it, they tend to describe more highlight information than shadow information, giving an unequal weighting of discreet steps to brighter areas of the frame versus their darker counterparts. By applying a gamma transform or a tonal map to the linear sensor read-out, you are able to repackage the data in a more meaningful and efficient way and certainly closer to how the human visual system will perceive it. The question is then: "Why is this applied to the RAW signal and not left to the post-production realm?"

In order for a sensor to achieve really good dynamic range representation, the data that it perceives must be recorded at very high bit depths. As we noted earlier in the format section, trying to record 4K data at very high bit depths and in a completely uncompressed way would yield enormously large files that most productions would not want to deal with, but here is the catch – they don't have to. By applying this gamma transform, you are able to efficiently store the usable information provided by the sensor in a more economical container without any visually perceptible loss of data. That is the genesis behind the Canon Log implementation: take all the high bit depth linear sensor data from Canon's Super 35mm image sensor and remap it into a more efficient and economical 10-bit Log container without losing any highlight or shadow information in the process is the genesis behind the Canon Log implementation.

Figure 4 gives us a representation of this nonlinear transformation of the linear sensor data. When film is scanned into a Cineon file, it too is mapped into a 10-bit container using a logarithmic curve for this exact same reason. It is interesting to note that the Canon Log curve is steeper than the Kodak Cineon curve. This explains why Cineon footage appears flatter and of lower contrast than Canon Cinema RAW footage transformed through Canon Log. This result is by specific design. Canon engineers designed Canon Log to apply a transformation from linear to log that will preserve the entire dynamic range (shadows to highlights) recorded by its sensor – no more, no less. The Canon Log transform was not designed to yield a specifically low contrast image. The low contrast nature of the Canon Log embedded files comes from the result of remapping the entire available dynamic range of the source linear sensor data into logarithmic form. This is one reason Canon applies the moniker of RAW to its recorded files. The original sensor outputs can be completely recovered as linear light representations in the color grading process.

Embedded Values: ISO + White Balance

When setting up to light a scene, two of the very basic tenants a cinematographer considers are: what is my base ISO? And what is my base color temperature for the scene?

Without a base ISO, it would be impossible to know how much or how little light one would need to illuminate a scene for a desired f-stop and, concurrently, without a base color temperature, one would not know whether their lighting instruments should be calibrated towards Tungsten or Daylight or even what gels would be required. Given these fundamental tenants, the question becomes: "What can be done to optimize the image quality once these user variables have been selected?" The answer provided by the Canon engineers has been to apply gain and other signal extraction strategies right at the sensor level.

Figure 5 demonstrates the power of making adjustments at the sensor level. What we notice is that this strategy allows the mid-grey reference point to stay relatively constant from ISO 320 to ISO 80,000. This is very uncharacteristic of modern digital cinema cameras because most cameras that offer a RAW recording capability do not make allowances for the user selected ISO. Rather, they offer the ability to change the ISO setting in post-production. The reality of this offering is that every CMOS sensor, no matter the manufacturer, has a nominal ISO and a nominal color temperature at which it produces the most favorable results at the highest signal to noise ratio. What this means is that if a sensor is tuned for 800 ISO and it is rated at 3,200 ISO, the image can be adjusted in post, but it will suffer from diminished shadow detail response. Likewise, if that same sensor is exposed for 320 ISO, the image will suffer from diminished highlight protection. Without adjustments made to the sensor, the amount of stops above and below middle grey will vary depending on the ISO the sensor is exposed at, forcing possible compromises to the highlight or shadow portions of the image.

Figure 6 illustrates the idea that if a cinematographer can count on having a consistent amount of latitude above and below middle grey across multiple ISOs, then this person can now push the camera by a stop or two or, conversely, pull the camera by a stop or two without changing the dynamics of how the light is distributed in the scene.

The main counterargument I hear for not wanting to make sensor adjustments to match your settings is: "What if I change my mind? Or what if in post, the scene changes and has to head in another direction?"

These are valid concerns and these situations do occur, but my answer would be twofold. In a vast majority of the cases, the final image will tend to look as intended by the creatives who created the image. Would you then not want your image optimized for how it will look most of the time? And secondly, should some accident or event cause the footage to be taken in a different direction, it is important to remember that we are still talking about recording a RAW 11MB frame. While the image is tuned for a particular ISO and White Point, there is still plenty of data in that file to take it in other directions, if called for.

Something that is also easily overlooked is the fact that the Canon EOS C500 actually has an ISO setting for 20,000 (expandable to 80,000!). In most RAW cameras, if one were to push the ISO setting in post to 20,000 ISO, all you would get is digital noise. The signal to noise ratio would be so poor that it would render the entire image unusable. The fact that the EOS C500 can record an image at 20,000 ISO with a signal to noise ratio of 49dB is rather spectacular, and directly attributable to the sensor adjustment strategy Canon employs.

In early digital video cameras, the characteristics of the noise (video noise) and the appearance of fixed pattern noise (fixed striations lurking amidst the random noise) made it so that all noise was undesirable and avoided at all cost. The EOS C500, with the help of its specially designed Super 35mm image sensor, seems to render fixed pattern noise at such low levels as to make it invisible to the eye. This allows the noise pattern at higher ISOs to feel very organic and filmic, and makes shooting at high ISOs not only possible, but also quite possibly desirable.

Let's look at a practical example.

Figure 7 shows two models posing for a shot. The ISO was set at 3,200 and the scene was lit for 3,200 ISO and normal exposure.

Figure 8 shows a split view of 3,200 ISO exposed normally on the top half and 850 ISO exposed for 3,200 ISO and retimed to look normal on the bottom half. The point of this exercise is to set up a scene with an illumination level set for 3,200 ISO and to see what the difference would be if we set the camera to 3,200 ISO (and thus allowed for sensor level adjustments) or if we simply left the camera at a nominal 850 ISO and adjusted the image in post.

Figure 9 shows a 300% magnification of the area between the two models. What becomes evident is that the noise floor is significantly raised on the lower half of the image (850 ISO and 2 stops) versus the upper half of the image (native 3,200 ISO), which is something that is consistent with what we learned in Figure 7. We also notice an elevated level of blue and red channel noise chatter in the black level.

Similarly on Figure 10, we can see that the video noise introduced in the underexposed skin tones creates the perception of a loss of resolution compared to the much cleaner gain adjusted 3,200 ISO equivalent.

It is important to remember that these sensor gain adjustments happen very early on in the image pathway and at very high bit depths so that when combined with other signal extraction strategies, they are able to produce significant and measurable increases in image clarity and signal to noise performances.

Signal Path

Most digital cinema cameras that offer a RAW recording option do so by offering either a proprietary port or pin connector assembly to a RAW recording device or by compressing the RAW recordings on-board the camera. Here, again, Canon has chosen to walk a very different path.

Figure 11 shows the signal pathway of the 4K RGB 4:4:4:4 Bayer RAW data. The sensor was specifically designed to readout the 4K signal as four parallel 2K data formats. These 2K formats include a red channel, a blue channel, and two separate green channels labeled “Gr” and “Gb.” These four signals are then multiplexed in accordance with the SMPTE 3G SDI serial interface standard ST 425-1-2011. Because this particular SMPTE protocol allows for four distinct channels (red, green, blue, alpha), Canon was able to substitute the alpha channel as a carrier for its second green channel, creating a 4:4:4:4 signal path that is R,Gb,B,Gr. You might wonder why you need to know this. The answer is rather simple: this singular decision opens up a whole world of options and opportunities when it comes to recording RAW signals.

By packaging the signal according to a codified SMPTE specification based on the use of standard 3G SDI interfaces, Canon has allowed cameras like the EOS C500 to transmit uncompressed 4K RAW video signals over a single 3G SDI serial interface at all progressive rates up to 30p, and at standard progressive rates up to 60p using two 3G SDI interfaces. The first implication of this is rather obvious: a wide range of possible recorders.

There are a growing number of SDI-capable recorders on the market and many of them are already certified to record Canon Cinema RAW, as seen in Figure 12. The real advantages of having many competitors in this field are not simply price and size, but also functionality. Because third party recorders are designed to handle signals from more than just one camera manufacturer, they tend to each have unique identifying features that set them apart. The CODEX Onboard S can record RAW data out of the camera into its Virtual File System, which allows you to generate all types of dailies and deliverables from within the system itself. Convergent Design offers extremely lightweight and cost effective recorders that also double as on-board monitors, while the AJA Ki Pro Quad can record compressed 4K ProRes4444 files while passing on the 4K RAW signal through a thunderbolt connection. All of these variances and options offer EOS C500 shooters an extended pool of workflow possibilities that can be tailored and customized to suit any requirements. Given the ubiquity of 3G SDI, it is reasonable to expect that the list of compatible SDI recorders will only grow longer from here.

There is a secondary implication to being able to transmit RAW data over a 3G SDI serial interface and that is the signal transmission itself.

As television makes the slow but seemingly inevitable transition from HD to Ultra HD or 4K, there is an increasing need and desire to future-proof media content recorded today. This process is fairly simple for scripted dramas or comedies with a built in post-production workflow that can handle the RAW file conversion and editorial process, but what about high profile live events such as the Olympics or World Cup Soccer?

Figure 13 demonstrates a hypothetical workflow one could use to take advantage of the ability to record RAW over 3G SDI. For a high profile live event, one could send both the HD Output (ready for Rec709 HD Broadcast) as well as the Canon RAW signals (Canon Log) and feed them through an Optical Fiber Encoder. The signals would be sent up to a mile away where they could be decoupled and the HD Output could be beamed live via satellite in High Definition in real time, while the 4K RAW data gets recorded for mastering and redistribution purposes. This workflow removes all the burden of media management and offload from the shoulders of each camera operator and places it in a centralized and dedicated controlled environment. This is just one of many permutations possible. Because the signal can run through 3G SDI switchers as well as distribution amplifiers, it would be hard to imagine a scenario that this type of flexibility couldn't accommodate.

Conclusion

Digital cinema cameras seem to be improving by leaps and bounds at a pace that is sometimes difficult to keep up with. With the introduction of RAW recording, we have witnessed a huge step forward in the quest to dethrone celluloid as the high-end acquisition format of choice. Canon's novel approach to RAW recording reminds us that the ultimate goal of a cinema camera should be to yield an image of high quality, wide dynamic range and excellent contrast and color reproduction. From the inspiring results I have seen from Canon Cinema RAW so far, it seems like we might be in the early innings of what defines a RAW recording solution. With some clear benefits in low-light recording sensitivity, expanded and consistent dynamic range and modular and flexible signal routing, Canon has certainly pushed this conversation forward in a meaningful way. As RAW recording evolves and matures, we should expect manufacturers to challenge the status quo and offer varying and competing solutions and processes towards the goal of allowing us to record incredibly vivid imagery in the most efficient manner possible.

I know I'll be keeping an open mind.

The CDLC contributors are compensated spokespersons and actual users of the Canon products that they promote.

http://learn.usa.canon.com

© 2014 Canon U.S.A., Inc. All Rights Reserved. Reproduction in whole or in part without permission is prohibited.