Transcription
It's a well-known fact that camera sensors come in all sorts of different resolutions. When capturing video, though, the image produced by a camera sensor is often scaled to one of a few common resolutions like 1080p or 4K. But how is this scaling accomplished?
This might seem like an uninteresting question because scaling digital images to different resolutions is easy to accomplish in modern photo and video editing software. But the process of creating a lower resolution image from a higher resolution sensor becomes a lot more interesting when we consider the limitations imposed by a rolling shutter. The limited readout speed of modern sensors leads camera manufacturers to employ some more creative methods of reading out sensors at different resolutions. And the different techniques that are employed can have a significant impact on the final image's quality.
To demonstrate this, let's look at a side-by-side comparison of how two different cameras produce 4K and 1080p images. So this is a test pattern that I'm displaying on my monitor and then filming to hopefully reveal how the two cameras resolve fine detail. For the first comparison, we have both cameras shooting at 4K and we're zooming in to 400% to see what kinds of differences are present. And it looks like both cameras are quite detailed at 4K and both of them are just about able to resolve the gaps between the individual pixels of my monitor. If I had to pick a favorite, I'd say that camera A's image probably looks better. Both images have some weird moire patterns visible, but they seem to be a bit more severe in the case of camera B.
However, when we switch both cameras to record at 1080p instead, we can see a very different relationship between the two. As one would expect, there's a drop in quality going from 4K to 1080p. But camera B manages to hold on to a lot of the important details. The text is still legible and we can even see a hint of the monitor's pixel grid. But camera A is a very different story. The image is much blurrier to the point where the text is becoming difficult to read and it might be my imagination, but the moire appears a bit worse as well.
So what's going on here? Well, to answer that, we need to recognize the fact that neither of these cameras have sensors which are exactly 4K or exactly 1080p. There's some in-camera scaling going on to achieve these different resolutions. But the cameras are using different methods to achieve that scaling. So in this video, we're going to talk about the different ways of reading out camera sensors and the effects they have on the final image. Stay tuned.
Okay, so let's start by addressing why different scaling methods are sometimes necessary and why in-camera scaling sometimes differs from scaling performed in post-production. When we change the resolution of an image in most editing software, the program is using a scaling algorithm to either add or remove pixels without impacting the perceived quality of the image. These algorithms come in many different shapes and sizes, all of them with their own benefits and drawbacks. Some do a great job of keeping the images look the same, while others attempt to enhance the image's quality at the cost of introducing some small artifacts into the finished product. I'm not going to spend too long going over the different algorithms that are available because, like I said, these scaling algorithms aren't always the ones that are used inside of cameras.
The main problem with using a scaling algorithm like this is that it requires reading out the value of every single pixel on a camera sensor, performing some processing on those pixels, and then scaling the image to the final resolution. This is a problem because the more pixels the camera has to read from the sensor, the longer it will take to produce a frame. Modern cameras usually don't have the hardware to read out every one of their pixels simultaneously. Instead, they only include the hardware necessary to read out a single row of pixels and they work their way from the top of the sensor to the bottom over the course of a few milliseconds. Sensors which operate this way are said to have a rolling shutter, a term you've probably heard before to explain the odd distortions that can occur around fast-moving objects. And it is true. Because sensors with rolling shutters take some amount of time to scan from the top of the sensor to the bottom, the bottom of their images represent a slightly different slice of time than the top of their images. And this discrepancy can lead to all sorts of nasty distortions when either the camera or the objects in the frame are moving quickly.
But not all rolling shutters are created equal. The time it takes for a sensor to read from the top to the bottom can vary drastically with some able to complete a frame in less than 3 milliseconds and some taking more than 10 times that amount to finish scanning. The slower a camera's readout speed is, the more noticeable the rolling shutter distortions will be. Now, a lot of the time, a sensor's readout speed is determined by choices made during the design process with some sensors giving greater priority to fast readout speed than others. But even once the sensor's design is set in stone, there are still some ways to improve the readout speed. Namely, if the camera can reduce the number of lines it needs to read out, it can complete a frame in less time. So for example, if we want to produce a 4K image using an 8K sensor, we could read out the whole sensor and scale the result down to 4K. Or we could simply skip reading out every other row. Doing this would produce a 4K images worth of pixels while doubling the sensor's readout speed. Thus, some rolling shutter distortions will be reduced. And as an added bonus, we may even be able to read out the sensor at a higher frame rate, which is always nice.
There is, however, a significant downside to this line-skipping approach. While skipping rows does improve readout speed, it comes at a significant cost to image quality. Take a look at this comparison of an image read out at 8K and downscaled to 4K to this other image that was read out at 4K by skipping lines. The line-skipped image exhibits strange and ugly-looking artifacts when trying to resolve fine details, where the oversampled image looks more natural. This phenomenon is called aliasing and understanding it will be critical to understanding why cameras read out their sensors in different ways.
So if you'll permit me, let's take a detour into the world of signal processing. Now, you might be wondering what signal processing has to do with processing still images from a camera sensor, but it's actually very relevant. While the field of signal processing often deals with signals like audio waves or electromagnetic waves, it's actually not that difficult to think of an image as a signal as well. Just like a sound wave is comprised of changes in pressure across time, an image is comprised of changes in luminance across space. Now, sure, the values of an image signal vary across two dimensions instead of one and we often have to store multiple images together to produce color, but none of this changes the underlying mathematics. The same techniques and concepts used to analyze one-dimensional waves can also be applied to two-dimensional images. But for the sake of making things easier to visualize, let's start by talking about aliasing in the context of a one-dimensional signal.
So let's say we have an audio signal traveling through the air and we'd like to capture that signal so we can process and transmit it elsewhere. There are two main ways we can do this. We could use a purely analog approach where a microphone converts the changes in pressure it receives into changes in electrical voltage that it transmits to a speaker. But if we want to use computers to aid in processing the signal, we have to convert the signal into a digital form. Instead of varying continuously over time, the signal's intensity needs to be broken up into discrete samples that can be stored sequentially in the computer's memory. So we have an analog-to-digital converter sample the intensity of the signal a certain number of times every second. And then, once we're ready to reproduce the original signal, we interpolate between those captured values as smoothly as we can. But if you look at the example I've been showing on screen, you might notice that the reconstructed signal is not exactly the same as the original signal that was captured. Sure, we've managed to reproduce the broad strokes of the signal's shape, but a lot of the finer details are missing. And to understand why this happens, we have to take a closer look at the sampling rate which was used to digitize the original signal.
In the prior example, I chose a sampling rate arbitrarily. But if we want to accurately reproduce a signal, we have to give the choice of sampling rate a bit more thought. If we use too many samples, then the storage space or bandwidth required to save or transmit the digitized signal will become impractical. But if we use too few, we'll be missing out on critical details which we might want to capture. So how do we know how many samples are needed to reproduce a given signal? Well, let's look at another example. Here on the left, I have a simple sine wave with a frequency of 10 cycles per unit time. And on the right, I'll show a reproduction of the signal after converting it to digital and back again. Let's say I arbitrarily chose to use six samples per unit time to attempt to capture the signal. You'll notice that the reproduced signal does not match the original signal, but it doesn't look like random noise either. In fact, the reproduced signal also looks like a sine wave, but one with a lower frequency than the original. And this phenomenon is what we call aliasing because the reproduced wave is taking on the identity of another wave which wasn't present in the original signal. And it's important to note here that aliasing like this cannot be corrected after the signal is captured. And the reason is that there are actually many different original waves which would all produce the same output wave at the sampling frequency. There's simply no way to tell which wave this is supposed to be based on the samples that were captured. Remember this because it'll be important later.
So then, if the sample rate I'm showing here is too low to accurately capture the signal, how high does it need to be to produce an accurate output? Well, let's watch what happens to the output signal as I increase the sampling rate. You can see that as the sample rate increases, the frequency of the output wave increases along with it until stop. Right there. When the sampling frequency is exactly double the frequency of the input wave, we can see that the output wave matches the input perfectly. And as we increase the sampling frequency past this point, the output stops changing and we maintain an accurate reproduction of the signal. This principle that the sampling rate should be double the frequency we're attempting to capture is called the Nyquist-Shannon sampling theorem. It states that in order to determine the appropriate sampling frequency for a given application, we must first determine the highest frequency that we wish to be able to record accurately. Then, our sampling rate should be double that.
As an example of this theorem in action, let's take a look at the sampling rates which are actually used to record audio. The two most common standards are to use either 44.1 kHz or 48 kHz as the sampling rate of the recording device. So, where do those numbers come from? Well, we probably don't care about capturing sounds that are outside the range of human hearing since they won't contribute anything to the listening experience. And if you look up the highest frequency that a typical human can hear, we find that sure enough, it's around 20 kHz. So, according to the Nyquist theorem, it makes sense that the sound will be recorded at a sampling rate around 40 kHz. Now, you might notice the frequencies used in the real world are slightly higher than 40 kHz, but let's set that aside for now. We'll get back to it.
Now, let's take what we've learned about sampling rates in one-dimensional signals and try applying it to a two-dimensional image. In this context, signals take the form of variations in the intensity of light across the frame. As an example of a signal that might be recorded in an image, let's take a look at this square wave which alternates between pure black and pure white with a frequency that corresponds to the number of cycles per image width. In the context of an image, low frequencies refer to gradual changes in luminance, while high frequencies refer to the fine details which change rapidly from pixel to pixel. Now, just like we saw before, if the sampling rate is too low, then higher frequency signals will be replaced by lower frequency signals that don't accurately reflect what we're trying to capture. Now, in the case of an image, the sampling frequency is just the number of pixels along each axis. Each pixel captures a distinct value representing what the signal looks like at exactly that position in frame. So, if the image's resolution is too low to accurately represent a given signal, the output will resemble a lower frequency signal that wasn't actually present. Suffice to say this is a problem. The moire patterns produced by aliasing look wrong to our eyes because we know what these things are supposed to look like in real life, and we know this pattern shouldn't be present. Aliasing is an artifact, one that we should do our best to avoid whenever possible.
The only reason it doesn't happen all the time is that real cameras have an inbuilt defense against it. You see, up to this point we've been conceptualizing a pixel as representing the value of a signal at a single infinitely small point. But the pixels present on a real sensor don't measure a single point. Rather, they record the total intensity of light over a small area. The effect of this is similar to what would happen if we average the value of a signal over some area. Going back to our earlier example with an alias sine wave, we can see that averaging the value recorded over a small region reduces the amplitude of the alias signal, but non-alias signals aren't diminished as much. In a complex signal with many overlapping frequencies, this has the effect of reducing the impact of aliasing as the sampled region increases in size. If we imagine dividing a camera sensor up into tiny regions, one for each pixel, then we can examine how much of its region each pixel records. This is called the fill factor. If a pixel only covered 10% of its region, it would have a fill factor of 10%, and aliasing would be relatively common. But if a pixel covered 100% of its region, the fill factor would be 100%, and aliasing would be minimal. With advancements in sensor technology, pixel fill factor has improved considerably. Modern sensors use microlenses to focus incoming light down onto a photodiode, resulting in a very high fill factor. It's hard to find numbers on exactly how high the fill factor is on specific sensors, but based on what I've read, it seems like most cameras these days have a fill factor of at least 90%. Since the fill factor of modern sensors is so good, aliasing is a relatively minor problem which only pops up in particularly challenging situations.
One kind of image which doesn't benefit from this averaging effect is anything generated by a computer. In computer graphics, by default, each pixel samples a single point. So, aliasing is a big problem. This is why computer-generated images have to employ advanced anti-aliasing algorithms to try and reduce the presence of jaggies and moire patterns. But again, real cameras benefit from a form of built-in anti-aliasing which doesn't require any special computational effort. With that in mind, let's stop and think for a moment about whether we might have seen this kind of aliasing in an image before. To find an example of aliasing, we're looking for some object that has very fine details, most likely with a uniform pattern to them that's likely to be just outside of what a typical camera can resolve. Oh, I know. Clothing. Certain kinds of clothing or other fabrics will often lead to aliasing that we can actually see clearly. In reality, the camera will be looking at a very small pattern of light and dark spots. But since the pattern's frequency is beyond what the camera can resolve, aliasing will occur. And when it does, we'll see a distinct low-frequency pattern which shouldn't be there, but is nonetheless present in the captured image.
Now, with an understanding of aliasing under our belt, let's finally return to the original topic of sensor readout modes. At the start of the video, I said that using line skipping to improve the readout speed of a sensor comes at a cost to image quality. Aliasing is a major part of that cost. A camera's built-in anti-aliasing depends on sampling almost all of the light that comes in from the lens, leaving minimal gaps between pixels that don't get recorded. Skipping lines though results in large gaps left behind by the rows that were skipped over. This reduces the effective fill factor of the sensor, and an increase in aliasing is the result. Take a look at this. I have this vector graphic that I've rendered at 32K resolution to minimize aliasing, and I've written some code which will simulate reading this image out at different resolutions with different sensor scan modes. For example, let's say that we're working with an 8K sensor that has a fill factor of 90%. If I have my code sample the image at 8K and then use a scaling algorithm to bring the image down to 4K, we see a pretty clean image. There's a little bit of aliasing in the really challenging patterns, but it's not too bad. However, if I instead read out the sensor 8K and then skip every other row and column, I still get a 4K image, but with significantly more visible aliasing. And if this isn't bad enough, skipping lines also results in an increase in noise across the image. If you'd like to learn more about the factors which affect an image's noise level, I made a whole video about that which you can watch if you'd like. But the short version is is that while reading out fewer pixels results in less read noise from the sensor, skipping some of those lines means capturing less of the source image, which results in a greater amount of shot noise across the image as a whole.
So, to summarize what we've learned so far, reading out a sensor with a higher resolution and then scaling the result down to a low resolution leads to great image quality at the cost of a slower rolling shutter that can distort the image. Line skipping on the other hand offers increased readout speed at the cost of increased aliasing and noise. So, with these two options in mind, it seems like using a high-resolution sensor to produce a low-resolution image is a bad idea since we have to accept one of these two drawbacks. If we imagine replacing an 8K sensor with a 4K one, we could produce an image with similar image quality to the downscaled image, but similar readout speed to the line skipped image. But changing the physical resolution of the sensor would require using a different camera entirely, so let's say we can't do that. Another option would be to crop in on the sensor and only read out the centermost pixels to produce a 4K image. But this would be equivalent to just using a smaller sensor with 4K resolution. And if we paid for a large sensor, then ideally we'd like to enjoy all of the optical benefits that come along with it.
There is one other option, though. The problem with using a digital image scaling algorithm to bring an 8K image down to 4K is that it requires reading out the full 8K's worth of pixels before processing can occur. The bottlenecks in this case are the analog-to-digital converters, which take some amount of time to perform the conversion for each row. But, what if we could scale down the image before it was digitized? As in, we utilize all of the pixels to capture light, but then we only digitize a quarter as many. Now, working in the analog domain means advanced scaling algorithms are out of the question, but who says a scaling algorithm has to be complex? What if we just took the voltages produced by neighboring pixels and wired them up in sequence, producing an output voltage equivalent to the sum of all four inputs? There's no computation required here. It's all accomplished through simple electrical circuits. And once it's done, we'll only have a quarter as many values that need to be digitized. Thus, the ADCs will be able to work their way through them much faster, and read out speed will improve as though we were reading out a 4K sensor instead of an 8K one. This won't lead to an increase in aliasing or noise, either, since all of the pixels are still being utilized, the fill factor of the sensor is unchanged. So, we should expect image quality to be comparable to what it would be on a native 4K sensor. Combining multiple pixels into one in the analog domain like this is called pixel binning, and it's a technique that's incredibly widespread. Have you noticed that recent smartphones advertise high resolutions like 48 megapixels, but the actual images they produce are much smaller than that? It's a result of pixel binning at work. When utilizing the full sensor area, the phone can operate the sensor in binning mode to produce an image with lower resolution that's easier to process and store. But, if the user wants to digitally zoom in, which they likely will since there's no optical zoom, then the phone can crop in on the sensor, but switch it out of binning mode so that there's minimal loss in resolution. This approach offers the best of both worlds. The user has access to high resolution to enable zooming in, but they aren't burdened by having to read out as many pixels when they aren't zooming.
Binning can even be used to improve low-light performance. Like I talked about in my video from last year, reading out more pixels for the same sensor area means that the amount of read noise introduced by the sensor will increase. For example, with all else being equal, one would expect that reading out an 8K image and then downscaling to 4K would result in twice the amount of read noise compared to reading out at 4K natively. This happens because each pixel that gets digitized will add some amount of noise to the image, and even after downscaling the image, the overall noise level will still be higher. But, by using pixel binning, we minimize the number of pixels to digitize, and the read noise is kept to a minimum. This is another reason why binning is commonly used on smaller sensors that struggle in low light.
There are a few limitations to the pixel binning approach that are worth keeping in mind, though. Firstly, pixel binning usually requires the ratio of the source resolution to the output resolution is a whole number. That is, binning can reduce the resolution of an image by a factor of two or a factor of three or four, but other ratios may require a combination of binning and downscaling. Also, when performing pixel binning, it's necessary to consider the color filter array used on the sensor. Most sensors use a Bayer pattern filter to capture color information, which works fine when the image is read out and demosaic prior to downscaling. But, if the pixels need to be combined before the demosaicing step, then some additional work will have to be performed to ensure color information is preserved. Namely, if adjacent pixels are combined with no regard for the color filter, then information from different color channels will mix together, and demosaicing will not be possible. To solve this, many sensors designed for binning will utilize a quad Bayer arrangement, where the Bayer pattern is applied two by two groups of pixels rather than to individual ones. So, when the pixels are binned together, a proper Bayer pattern emerges, which can then be demosaiced. However, when such a sensor operates without using pixel binning, it becomes necessary to shuffle pixel values around in order to create a workable Bayer pattern. This process can result in a slight reduction in detail, since nearby values tend to bleed together and create a softer image. It's also possible for sensors to work in the opposite way, using a conventional Bayer filter for a one-to-one readout, and then shuffling pixels around when utilizing pixel binning. It all depends on which mode the camera manufacturer decides to prioritize.
So, those are the four main ways a camera sensor can be read out at different resolutions. A one-to-one readout followed by image scaling produces great image quality and is quite flexible, but comes at the cost of readout speed. Cropping in on the sensor improves readout speed and is computationally simple, but reduces the effective sensor size, leading to numerous optical drawbacks. Line skipping utilizes the full width of the sensor, but results in a significant increase in aliasing and noise. And finally, pixel binning has good aliasing and noise characteristics and offers solid readout speed, but it requires a bit of extra hardware to implement and can result in a slight reduction in quality under some circumstances.
So, there you have it. There is no more to talk about. Don't look at the runtime of the video, it's fine. There are no more tangents to go off on. This is a completely satisfactory explanation, right? Um, except didn't I say we were going to talk about why audio is sampled at a higher rate than is actually necessary? Yeah, so about that. It's actually quite common for signals to be sampled at a higher rate than is actually necessary to reproduce the input signal correctly. And understand why, we have to ask ourselves what happens if higher frequencies that we don't care about make their way into our signal. After all, even if human hearing maxes out at 20 kilohertz, that doesn't mean that higher frequencies don't exist. What if there are sound waves at 25, 30, or even 50 kilohertz traveling through the air? What happens when they hit a microphone that's only designed to handle 20? Well, aliasing, that's what. Remember, signals which are too high frequency to be recorded correctly show up in a captured signal disguised as lower frequency signals which weren't actually present. So, sounds which are outside the range of human hearing would end up distorting the recorded sound away from what it would actually sound like if you were standing next to the microphone. Since it's not possible to remove aliasing after a signal has been converted to digital, it's necessary to filter out high frequencies in the analog domain. We can do this with something called a low-pass filter. It's a simple circuit which reduces the amplitude of high frequencies while leaving low frequencies alone. Filters like this have a configurable cutoff threshold that can be set to remove any frequencies which would result in aliasing. The problem, though, is that analog low-pass filters are never going to be perfect. Ideally, we'd want them to completely eliminate frequencies above the cutoff threshold and leave the lower frequencies completely intact. But, in practice, there's always going to be a transition period where the signal is only partially suppressed. So, this means we'll either have to accept some higher frequencies leaking through and affecting the result, or suppressing some of the frequencies that we actually want to be recording. But, what if we didn't have to solve all of this purely in the analog domain? After all, once the signal is digitized, it's not too difficult to apply a digital low-pass filter to remove high frequencies. The digital approach can't eliminate alias frequencies which are disguised as lower frequencies, since there's no way to distinguish them from the true signal, but it can eliminate high frequencies that are captured without aliasing. So, the solution is to use an analog low-pass filter to suppress the high frequencies as much as possible while leaving the frequencies we care about alone. Since the low-pass filter isn't perfect, some of the higher frequencies will slip through. So, we sample the signal at a slightly higher rate than would usually be necessary in order to capture these higher frequencies without aliasing. Then, with the signal properly captured, we can digitally remove the frequencies that are outside the range we care about. The result of this process is a digital representation of the original signal that captures all the frequencies we care about while also eliminating aliasing caused by the presence of higher frequencies. This is why it's standard practice to record sound with a slightly higher sample rate than is actually necessary to produce the full range of human hearing. It gives a bit of a buffer to allow for the imperfections of analog low-pass filters.
These same principles can be applied to images as well. Now, unfortunately, capturing an aliasing-free signal isn't as simple as adding an additional low-pass circuit to the camera sensor. This is because aliasing occurs as soon as a continuous signal is broken up into discrete samples, which happens as soon as the incoming light hits the sensor and is broken up into discrete pixels. Therefore, it's necessary to filter the signal before it even hits the sensor. Thankfully, this can be achieved by adjusting the optics in front of the sensor. You see, in the context of an image, a low-pass filter has the same effect as slightly blurring the image. When an image is blurred, the finest details disappear, while the broad strokes remain. To put it another way, the highest frequencies are removed, while the lower ones are preserved. Since all optical systems cause some amount of blurring in the projected image, all we need to do is ensure that the image is blurred enough to remove any frequencies which the sensor can't capture, but not so much that the image becomes noticeably soft. However, precisely matching the blurring characteristics of the camera lens with the sensor isn't exactly practical. Ideally, we'd like to be able to choose our preferred lenses without having to consider the resolution of the camera sensor. The solution is to add an optical element to the camera, rather than the lens, which performs the appropriate amount of blurring for the sensor. This is called an optical low-pass filter, or OLPF, since it fulfills a similar role to an electrical low-pass filter, but in the optical domain.
Now, OLPFs are subject to the same limitations as electrical low-pass filters. They can't create a perfect cutoff between suppressed and pass-through frequencies. So, it's necessary to perform the same oversampling technique on the camera sensor to digitally remove the frequencies that the OLPF can't. Let's look at an example. Let's say that our goal is to create a clean, detailed 4K image without any visible aliasing. The first step is to add an OLPF between the camera sensor and the lens, which preserves any details that would be visible in a 4K image, but attempts to cut off the higher frequencies. This filter isn't perfect, so some of the frequencies that are just beyond the resolving power of a 4K sensor manage to slip through. Therefore, the camera sensor is built for a slightly higher resolution, say 4.6K, so that it's able to capture all of the frequencies passing through the OLPF without aliasing. The camera captures an image that is slightly soft when viewed at 4.6K, but that's okay, because the goal is only to capture enough detail for 4K resolution. After the image is captured, it gets digitally scaled down to 4K resolution, either in camera or in post-production. Digital scaling algorithms typically involve filtering out any frequencies that won't be properly recorded in the scaled image, so the greater than 4K frequencies are removed, and the result is a nice, clean 4K image without any meaningful aliasing. And oh, by the way, this hypothetical example isn't hypothetical at all. It's exactly the approach used by the ARRI Alexa 35 to create a clean 4K image off of its 4.6K sensor.
So, there you have it. A camera which features an appropriate OLPF and oversamples its images from a slightly higher resolution sensor will produce the best, clearest image possible. The main downside of this approach, though, is cost. OLPFs cost money, and they don't really enhance the image quality outside of the few situations where aliasing is likely to be visible. So, budget-oriented consumer cameras will often omit them to save on cost. Further, reading out the sensor at a higher than necessary resolution without using binning or skipping will result in a slower readout speed and more pronounced rolling shutter artifacts. And like I said in my previous video, a higher resolution sensor will also be at a disadvantage when it comes to read noise. The tradeoffs in readout speed and noise performance can be mitigated with better sensor design, but this also increases cost, just like the OLPF. It's for this reason that we often see a divide between professional cinema cameras that aim for the maximum possible image quality, regardless of cost, and consumer cameras, which try to produce the best value possible. The former typically do include OLPFs and are capable of reading out at higher resolution while maintaining solid readout speed and noise performance, while the latter will often prioritize one or two of these aspects and make tradeoffs in other areas. It's with these consumer and prosumer cameras that we often see alternate readout techniques like line skipping and pixel binning employed.
With that in mind, let's summarize what we've learned so far about the different ways cameras can read out their sensors and the pros and cons of each. We'll be analyzing each readout mode in four different categories. Aliasing and detail, readout speed, noise performance, and field of view.
Oversampling plus OLPF. If the camera includes an OLPF in front of the sensor and reads out all of its pixels before digitally scaling the result to 4K, image quality will be maximized and will typically exceed the quality of a native 4K sensor. Oversampling on its own is almost as good, but with some aliasing artifacts present. Rolling shutter in this case is quite bad, since the camera is reading out so many pixels, and noise performance will also be slightly worse than a 4K sensor, assuming the per-pixel read noise is the same in both cases. However, this approach does utilize the full width of the sensor, so it gets a pass in the final category.
Pixel binning. If the camera combines charges from neighboring pixels in the analog domain before reading out their values, readout speed can be improved to more or less match the performance of a native 4K sensor. And noise performance is a similar story, since the charges are combined before read noise becomes a factor. Field of view is also the same as a native 4K sensor. Image quality will often be quite similar to the native readout, though it may be slightly worse depending on the color filter array and the specific techniques used to bin pixels. The only real asterisk to pixel binning is that it requires a sensor which is designed to operate in binning mode, which not all of them are.
Line skipping. This is an alternative to pixel binning that can be used if the sensor doesn't support combining charges prior to digitization. Line skipping has similar readout speed to a native or binned readout and maintains the camera's field of view, but it suffers quite a bit when it comes to noise and image quality. Line skipped images suffer from aliasing far worse than any other readout mode, and since the camera isn't utilizing all of its surface area, noise performance also suffers.
Cropping. Reading out a smaller region of the sensor can improve readout speed without requiring binning or line skipping, so it performs well in terms of image quality and readout speed, but noise performance suffers since the full sensor isn't being utilized. And of course, cropping reduces the field of view, which often comes with optical disadvantages.
Now, with all that under our belt, let's go back to the example I showed at the start of the video. Two cameras, one which performs better at 4K and one which performs better at 1080p. Okay, so uh I'm going to go ahead and go off script for a bit here, and we're just going to go ahead and walk through some of these comparisons that I've shot together and uh see if we can use this uh test footage to try and uh infer what readout modes are being used by the uh the two cameras that were used to shoot this, and I know the specs of their sensors and everything, but even so, a lot of times camera manufacturers don't actually specify what readout modes are being used, so test footage like this can be very helpful to try and, you know, deduce what the camera is doing internally. So, I've got camera A and camera B. One's going to be on the left, one's going to be on the right. I'm not going to swap them between, so they're always going to be, you know, one on the left, one on the right. It's going to be the same ones. Okay, so our first comparison here is going to be both cameras shooting at 4K resolution at 24 frames a second, and then I have the footage cropped in to 400% so we can see the fine details. So, here we go. We've got camera A on the left, camera B on the right, and then I've I've sort of normalized the exposure between them. Uh they're both filming the same pattern on the same monitor. I've matched the field of view as closely as I can. I've, you know, matched the focus as closely as I can, so hopefully the only variable here is going to be the recording resolution that's specified as well as the frame rate. So, if we compare these, on camera A side, you can see we've got a very detailed image. And again, this is this is way blown up. Like, even this this is blown up to 400%, so we're we're getting something way, way zoomed in. But even so, we can see quite a bit of detail here. We can actually see the individual the individual pixels of the monitor, uh which is which is very impressive. So, you can actually sort of sort of see the pixel grid, uh these sort of faint blobs here and here and here and here. And, you know, the text is very sharp. It's it's clearly legible. You can see all the individual pixels making up the text. Uh this image is being shown on the monitor at one-to-one scale, so one pixel in the image is exactly one pixel on the monitor. So, yeah, so you can see, you know, we've got perfect perfectly sharp pixels here. You know, we can't quite make out, you know, subpixel level detail, but you know, again, considering we're blown up to, you know, more than 400% here, this is this is quite good quality. If we go and we compare that to camera B, we can see that camera B looks less detailed because it's it's just it's softer, you know? We can s- definitely see these vertical stripes, um but they don't seem to be, you know, as defined. It's a bit harder sometimes to make out the pixels, you know, and the horizontal stripes are not as clearly defined. Just the pixel grid is not as clear as it is here. And, you know, it it looks almost like an aliased pattern, like it's it's sort of just barely not able to resolve that grid. And, you know, this is actually a great a great test case then, because, you know, we've got this grid that this camera is just barely able to resolve, and this one is just barely not able to, and that sort of gives us the um the difference in in detail that we're getting here. And, you know, the text is still definitely legible. It's still quite clear, but it's it's it is softer, you know, it's definitely softer than than what we have on camera A. And if we compare aliasing between these two images, so you'll note that the aliasing both of them have it, but it's quite different. So, on camera A, you can just about make out there's these like diagonal stripes here. So, there's like a darker one and then a lighter one and then a darker one and then a lighter one and then a darker one and a lighter one and there's a bit of like a color shift as well. And if I go and put the image into motion, you can see that that sort of shifting as the the camera shakes slightly. Um so, there is some aliasing on camera A, but it's it's not it's not too bad, you know, this isn't super distracting, I would say. And again again, we're we're zoomed way in here. And then you know, on camera B, I think that this this pixel grid is an alias pattern. And you know, we can see that as well, you know, when we when it shakes, you can sort of see like the the horizontal uh and vertical lines sort of shaking along with the camera. Um and that again, you know, indicates that this is uh sort of a moire pattern. And you know, if you sort of squint your eyes a little bit, you know, it almost disappears, but but it's definitely there. So, yeah, so this this tells us that, you know, whatever is going on between these two cameras, camera A has has a better image. It has a more detailed image than camera B when doing the full 4K readout. But we can we can learn quite a bit if we go on to another test here.
So, this is 4K at 120p. So, first thing you'll notice very clearly is we have a substantial field of view difference between the two. So, again, these are both scaled to 400%. I did my best to match the field of view with the lens while recording, but I haven't you know, I haven't changed the scaling. Both of these are the same scale in the editor. And so, you know, if we compare you just look at camera B, it's field of view, we can see that camera B is a little bit tighter. It's a little bit tighter when going to 120p. And you know, that indicates cropping, of course, cuz it's it's cropping in just ever so slightly. And on camera A, we're getting a very substantial crop. We're getting a much bigger difference in field of view. Everything just gets much bigger. But you'll note, even though camera A is cropping in quite a bit, we still have a lot of detail, right? And so, the fact that camera A is able to produce a a 4K image at this field of view and then, you know, cropping in by that amount and it still has a very detailed image and it's still shooting 4K, you know, that tells you that camera A's sensor is probably greater than 4K resolution, you know, it's hard to say exactly exactly how much greater, but you know, the fact that it's able to crop in this much and it still still has a nice detailed 4K image is is quite telling that it's that it's probably going to have more than 4K resolution. You know, whereas camera B, you know, it's cropping in, but it's cropping in less. And that's what sort of an indication that for one thing, it could be that it it just can't crop in anymore before it's not shooting 4K anymore or it could be an indication that it doesn't need to crop in more than just a couple percent in order to get that 120p readout. And so, what we're actually finding out here most notably in this test is that we're finding out there's a significant difference in in readout speed between the two cameras because if you ask the camera to record at a higher frame rate, it has to improve its readout speed such that it can record 120p. Like I believe for 120p, it has to be like it's something like below 8 milliseconds or whatever um to do a a 120p readout cuz it has to read the sensor in less than 1/20th of a second. And if if it's not able to do that at its normal readout mode, then it has to improve readout speed somewhat. And we can see both of these cameras are cropping, which will improve readout speed by the proportional amount, you know, but there's not the quality isn't changing a ton. You know, we're not seeing a huge difference in aliasing or in, you know, detail. So, it looks like they're just cropping. It looks like you know, they're not switching into binning mode, they're not switching into line skipping or anything like that. They're just cropping in on the image. And camera B is cropping quite a bit less than camera A is, which maybe indicates that camera B doesn't need to crop its image as much to achieve a 120p readout speed where camera A has to crop a lot more. So, that would imply that camera B has a faster readout speed than camera A.
So, we move on next, we're going to talk about 1080p. So, these are both cameras set to record 1080p internally. So, this is not, you know, 4K on a 1080p timeline. This is both cameras shooting 1080p internally. And wow, what a difference. Um And so, you know, you remember looking at 4K, they were close to each other, but you know, camera A was more detailed. And then you we go to 1080p, and wow, you know, camera A is a lot less detailed than camera B. Now now, they're both worse than 4K. You know, if we just look at camera B, you know, the 1080p is softer. It's less detailed than the 4K, which is to be expected, you know, cuz we have a quarter as many pixels. In the case of the of camera B, it's it's a pretty reasonable difference, you know, this this is sort of what I would expect to see dropping from 4K to 1080, you know, zoomed in this much. Whereas, you know, camera A, we've got an enormous difference in quality here, just a huge one. We've gone
from being able to make out the individual pixels to barely being able to read the text at all. Um, but note that the the field of view has gone back to being more or less the same between the two. So, we don't have that massive difference in field of view anymore, um, but we do have a huge difference in quality. And so, this this sort of implies that in order to create a 1080p image from whatever their native resolution is, uh, these cameras are having to employ different readout modes to get there.
Now, in the case of camera B, well, the aliasing isn't really significantly worse than the 4K. You know, they both have some aliasing in there, you know, these, you know, funky patterns, but it's not significantly worse in the case of camera B. So, that sort of tells us there's not line skipping going on here. And, you know, we can't really tell much about the readout speed, but we know that there's no line skipping and the quality is quite similar. So, that could mean that it, for example, has a quad Bayer sensor and and that, you know, it's able to switch into binning mode without much drop in quality, but, you know, and that could also potentially explain why it's softer, why it's softer 4K. If it had a a quad Bayer sensor, then one would expect the image in binning mode to be quite good, but then it to be a little bit softer when at full resolution. So, this could mean that it's a quad Bayer sensor switching in and out of binning mode. That's a looking pretty plausible.
It could also be oversampling, you know, because again, we're not seeing a huge difference in aliasing and, you know, without really knowing about the readout speed, we can't really say whether it's binning or oversampling. If if it was oversampling, we would expect the readout speed in 1080 to be the same as in 4K, more or less. And if it's binning, then we would expect to see a readout speed improvement. So, we'll look at that in a second.
But, you know, in the case of camera A over here, we've got that huge difference in quality. So, we know there's no oversampling cuz, you know, no oversampled image is going to look this bad. And, you know, we can also say pretty, you know, it's it's not line skipping either because, you know, again, we've got aliasing in both cases, you know, these sort of funky uh diagonal patterns. And, you know, those are still present here. You know, and they look about the same size and shape. We've still got those diagonal aliasing patterns, but there's not really a whole lot other aliasing going on. And this sort of implies that we're doing a binning operation, but rather than doing a a quad Bayer binning, that perhaps we're doing, you know, rearranged binning that's having to rearrange the pixels to make them into 2x2 grids and then and then it's having to debayer that. And we know that when you have a non-quad Bayer sensor and you expect it to bin down, that you do get a softer image. You know, but this is a very soft image. So, perhaps one could wonder whether instead of doing a 2x2 binning without the quad Bayer filter, it's perhaps instead doing a 3x3 binning or even a 4x4 binning. You know, to get something that's this bad where everything it's just sort of bleeding together, you know. So, that's that's telling us quite a lot here.
So, you know, at this point we're thinking camera B is probably either binning a quad Bayer sensor or it's oversampling and camera A, it's looking like it's most likely binning from a greater than 4K sensor all the way down to 1080, which is giving giving this this blurry result. And if if camera B is binning, then the quality difference could be explained by, you know, camera B has a quad Bayer filter, so it works better in binning mode. It also could potentially have a lower resolution, so it's not having to bin as much. You know, this one could have a Bayer filter, so the binning results are worse. It could also have a higher resolution, which would be supported by the the 120p test.
So, now we're going to go to 1080p at 120p. And now, you know, this is very interesting because now, if you remember in the 4K 120p test camera A had a very different field of view. It was cropping in quite a bit, whereas camera B was cropping in only a little bit. And in the 1080p 120 test, now it looks like camera A isn't cropping or it's not cropping nearly as much because it has a a much more similar field of view to camera B now. So this implies that it's using it's using yet yet again a different readout mode. And we can also see that in the case of camera A, you know, the image in 24p versus 120p is not that different and any differences that are there could be explained by, you know, just more compression, you know, just from less bit rate relative to each frame. You know, so it looks like in the case of camera A that it's able to read out at 120 frames per second in this mode. So whatever readout mode it's using here it's able to do it at 120p where whatever readout mode it was using to start with at 4K, it's not able to do at 120 without cropping. So this that would again support the bending conclusion because if in this case camera A is bending quite substantially, if it's bending, you know, 2 by 2 or 3 by 3 or something like that then then that would explain the increased readout speed. So now it can actually hit 120 where it couldn't before.
And if we look at camera B, oh my gosh, what is going on here? So this is 24 and this is 120. So we've got a very clear difference in quality here. Now it's not a huge difference in field of view so it doesn't suggest cropping but what this looks like is if we look at this, you know, well hang on. Now look look at how big these dots are. You know, look at relatively how large, you know, sort of the frequency of the the dot pattern is at 24. And then now at 120 it looks like the frequency is a lot lower of those, you know, oscillations, right? We've also got like just really really strange details. Like the text is almost illegible at this point. Like you can I mean down here it's a bit more readable, but up here it's like, you know, look at that. Like it's just missing. Some of it's just missing. And what that suggests to me is that we're seeing line skipping here because we're seeing a significant increase in aliasing. That's the the high frequency pattern becoming a low frequency one. So that suggests aliasing right off the bat and then also, you know, the fact that we're we're making the text so much harder to read and so that would suggest that camera camera B, I mean, in 1080p 120 has now for some reason gone to line skipping rather than binning, which is very interesting. And, you know, you'll note that now there's not really a field of view difference where you remember in 4K camera B did crop in just a bit to get to 120p readout speed whereas whereas here it's going to line skipping but it's not cropping.
And then when we go to uh 1080p at 120 versus 240. Now these are both both camera A or no, sorry, both camera B in this case because camera A doesn't even have a 240 frame per second mode at any quality whereas camera B does have a 240 frame per second option at 1080p. So this is 120 on the left here. Oh no, yeah, so this is one, sorry, I I mislabeled these. So these are both camera B. So this is camera B 120p on the left here and camera B 240p and look this looks like line skipping just like before. This looks like pixel binning. So that's that's really weird cuz it looks like in camera B we've got a whole bunch of different readout modes. We've got whatever it's doing at full 4K and we've got a version of that but cropped in and then we've got some method of down sampling to 1080 that it uses up to, you know, up to 60p or whatever and then at 120p it has another mode for line skipping down 1080 and then it has yet another mode for binning down to 1080. So it it has three different ways of getting to 1080p from whatever it's native resolution is. Whereas in the case of camera A, you know, we've got we've got a really detailed readout here at 4K and we've got, you know, looks to be a crop in, you know, either to 4K or nearly 4K and then we've got this different readout mode to get down to 1080 from whatever it's native resolution is that it it maintains regardless of of the frame rate at 1080p.
So now comes the time to reveal what cameras these are based on the information that we've learned. So uh camera A is the Sony a6700, which uses the same sensor as the Sony FX30 and ZV-E10 II. So that is a 6K Super 35 sensor. So camera A's native resolution is roughly 6,000 pixels. So it's greater than 4K resolution, which is what we suspected based on the crop in. So in its, you know, in its native 4K recording mode at 24p, it's reading out a 6K sensor and it's producing a super duper detailed 4K image. And the fact that it is so detailed, you know, suggests that it's over sampling. That's what it's doing. It's over sampling. It's got a 6K sensor, it's reading out all 6K's worth of pixels and then it's down scaling it digitally to to 4K and that's why we've got this super great detailed 4K image at 4K 24. And then, you know, when we go to 4K 120, it has to crop in quite substantially and it's still at 4K, which it can do. You know, this is roughly a 1.5 times crop. It might be like 1.55 times and it still has 4K resolution, which makes sense. You know, you can crop into 6K about 1.5 times and you still have 4K. So, you know, that's good. So it it can still crop in quite substantially but this also tells us that its readout speed is not fast enough for 4K 120 and that's true. If we look up the results for the a6700's sensor, its readout speed is faster enough for 4K 60 but not 4K 120.
And then in 1080p, in 1080p, clearly the readout speed is faster, like cuz it can do 1080p up to 120 where it couldn't do 4K up to 120 with a full readout. So it's clearly and, you know, based on the detail, it's clearly not reading out full 6K anymore, right? But the aliasing is not that much worse. So what I think is going on here is I think that it's binning. I think it's binning 3 by 3. So it's taking 6K, it's binning it 3 by 3 and it's binning it down to 2K and then that is slightly resampled to 1920 by 1080. And because it's not using a uh because it's not using a color filter array that's designed to be binned that way, you know, colors are sort of bleeding together cuz they have to be rearranged before they're binned and then and then de-mosaic. So uh that's why that's why the 1080p on the a6700 and the FX30 is so soft. You know, that comes with the benefits and drawbacks because the benefit is when it's using this binning approach, now the readout speed is fast enough for 120p. So that also means the rolling shutter won't be as bad. So the rolling shutter won't be as bad at 1080p, but the actual image quality is going to be noticeably worse. So that that is what's going on with camera A, it's the a6700. It over samples 6K down to 4K. Um and then for 120 it has to crop in and then for 1080p it has to bin 3 by 3 to get to 4K. Well, to get to 1080p, I mean.
And then so now we're going on to camera B. Camera B is of course the a7S III. Um and it has a 4.2K sensor. It's full frame. So for starters uh camera B does not crop in very much when going to 120 because it can't. Because if it crops in any more than this then it doesn't have enough pixels for 4K. It's only a 4.2K sensor. So it can only crop in about 10% before it is less than 3840 by 2160 resolution. It So it is over sampling in the 4K 24. It is over sampling 4.2K down to 3.8K, which is, you know, not nothing, but it's it's not a very significant over sample. Um it's it's it's more or less a native readout. It's just over sampling slightly and then if you're in uh the 120 mode or if you have the active stabilization on, then it just goes to a one-to-one readout of 3840 by 2160. And, you know, again, this tells us uh we would expect the a7S III's readout speed to be quite good because it can do it can do 120 with only a 10% crop and that again is true. Uh the a7S III has I think roughly in the neighborhood of 8 milliseconds readout. Um it's just barely not fast enough for 120 at the full readout. So it does crop in a tiny bit, but that's still quite good. It's it's much better than the readout speed on the a6700, which again is supported by the testing.
And so when we go to 1080p, you'll note that the a7S III, it's 1080p is so good because it's almost certainly over sampling still. It's it's reading out the full 4.2K and then it's down sampling that to 1080p. And what gets really strange though is when we get into the higher frame rates. Uh 1080p 120. And this is where honestly I get a little bit lost. Cuz I don't know why on earth it would start line skipping at 120 and then bending at 240. It doesn't make sense because we know that the a7s III can do a 120p readout with only a 10% crop, right? So, if if I were the engineer designing this and I would say, "Okay, well, what readout mode should it use in 1080p 120?" I would say, "Well, you know, just crop in 10% to 3840 by 2160 and then down sample." But for some reason they instead decided it decided to more or less keep the field of view the same but use this awful pixel line skipping to get to 120. Which is a very strange choice, I think, that they would prioritize, you know, not having to crop the sensor at all for this difference in quality. So, that that's very strange to me and the only the only reason I can possibly I can possibly think for it is that there could be, you know, something with the fact that it's actually bending from 48 megapixels. This is something I've talked about in previous videos. When I say that the a7s III has a 4.2K sensor, that's not strictly true. Instead of a 12 megapixel sensor, it actually has a 48 megapixel sensor that it is always in bending mode. So, it's a quad Bayer 48 megapixel sensor that is always in bending mode. So, the a7s III functionally has a 12 megapixel sensor, but technically physically it's a 48 megapixel one that's bending. And it's and it's always bending, right? Um that's that's this is really strange. I don't know why they would do this.
So, I'm I'm a little bit out of loss here. The 240p is also very confusing because, you know, I understand that it would need to switch bending switch readout modes to get to 240. You know, but okay, if they're line skipping here then surely they could take this and then crop 10% to get to get to 240, right? Cuz if it's line skipping 4K to 1080 then that's going to double the readout speed. So, if you take the the 10% crop to get to 120p and then you line skip to get to 240p, surely you'd be able to do 240p with essentially this readout mode but with a 10% crop. But this is clearly bending. This is not the same readout mode as we have over here, clearly. Which is baffling to me because um the sensor is already bending. It's already bending 48 megapixels down to 12. Whereas this result indicates that it's bending 48 megapixels down to 12 and then bending 12 down to three. Um which is very I didn't I wouldn't have thought that was possible. And maybe it maybe it also has like a like a hybrid like quad Bayer, but then also but only halfway like or like a double like it bends once and then bends again. I don't know how this is possible. I I really I really would love to see have a conversation with Sony engineers about what the heck is going on here because I I I really don't know.
Um, I know I've been a little bit rambly here, but just to wrap things up, we can see here that the different readout modes can have a very significant impact on image quality, right? It's it's a matter of whether do you get this image or do you get this image, you know? And and that's nothing to do with, you know, the dynamic range or the or the compression or anything like that. It's the same compression in both cases. It's it's this versus this and the difference is the readout mode and the native sensor resolution. So, understanding the readout modes that your camera uses is very important because it lets you make decisions about how you're going to set up your settings, you know? On the a6700, it's better to shoot 4K and then downscale at 1080 in post, you know, versus shooting 1080 in camera. There's a big difference between those two. Whereas on the a7s III, there's not really a big difference because the 1080p is so good. So, you know, it it it it is very important to understand what your camera is doing and unfortunately camera manufacturers love to not exactly tell you what the camera is doing. They love to be vague and just say, "Oh, yeah, there's like some resampling going on." It's like, "Okay, but what kind of resampling?" So, camera manufacturers love to not tell you. Um, I don't think any of the information that we've uncovered about these two cameras are actually confirmed by the manufacturer. These are just, you know, reasonable inferences based on the test results, but I don't think Sony has actually told us what the readout mode is. So, it could be that I'm wrong. It could be that I'm totally wrong about what's going on here. But this is just the best result the best guess that I have based on the test results. So, yeah. Anyway, I hope you all enjoyed this video. I hope you found it informative. And yeah, my name is Camera Conker and I'm signing off.