Intro

Pixel binning is the process of combining the data in a group of pixels into a single pixel, such as a 2×2 or a 3×3 block. Doing so can increase the effective sensitivity or reduce the noise present in the resultant pixel. The trade off is of course reduced resolution in the final images.

To illustrate this consider the case of of the 2×2 block being combined to increase sensitivity. If the camera exposes an image for a quarter of the time that it normally would, each pixel is underexposed by 2EV. However when the pixel values are added up, the resulting single pixel will have the correct exposure. Therefore preforming this binning has created a single image with a resolution a quarter of the original, but shot at an ISO 2 stops more sensitive.

Motivation

Pixel binning to average values (for reduced appearance of noise) is uninteresting. This averaging is performed by the majority of image editing software when an image is scaled down, which is done anyway to any image when prepared for the web.

Pixel binning for addition to simulate higher ISO values however is more interesting, this potentially unlocks a little extra performance in low light. The shorter exposures could be key in freezing motion in dimly lit areas such as at concerts or some sports.

But would this pixel binning really be any better than underexposing the image then correcting exposure in a raw processor? Does it really gain anything?

Test images

Searching doesn’t readily produce any software that will perform pixel binning, but fortunately Python facilitated the creation of a script that would take in a 16bit grey-scale image and bin the pixels by addition in just 9 lines.

Pixel Binning test images.

The image above shows the results. Three test images were captured of the same scene at ISO6400 at 0,-2 and -3EV. The underexposed images were processed with the python script to bin the pixels up to the correct exposure (2×2 for -2EV, 3×3 for -3EV), simulating an ISO of 26K and 52K. The same images were also treated in adobe camera raw by adjusting exposure value, then scaling the image to match the binned image. For comparison the 0EV image has also been scaled to match. The images above are 100% crops from the images.

Analysis

It’s quite easy to see that there is effectively no difference between the images that have been binned, compared to the images manipulated in Photoshop. And neither manages to match the level of shadow detail present in the image correctly exposed in the first place. A moment considering the maths behind the two processes shows the reason for the similar appearance.

Consider the case of 2×2 pixel binning. Label the 4 pixels A-D, and the binned pixel X. The additive binning method provides the straightforward equation:

X=A+B+C+D

Whereas the photoshop method multiplies each value by 4 (+2EV), then averages them:

X=(4A+4B+4C+4D)/4

Which clearly is identical. So does additive binning really provide any advantage over binning to average? Well in this case no, they are the same thing according to the equations above when done in post. The problem with the additive method is that the noise is also summed. Once the light incident upon a pixel falls below the noise floor, the data is pretty much lost and adding another three pixels at the noise floor just gives you more noise, rather than aiding recovery of shadows.

Can it still work?

So while you can do pixel binning in post if you want, it doesn’t particularly gain much. So is it still worth doing? Yes, when done during capture.

The key issue is overcoming the noise floor, and once the noise is present, no amount of post work with remove it. Read noise is one of the problems, which is the noise generated when the pixels are read from the sensor. If pixel binning is done on sensor, i.e the sets of pixels are combined before being read, then the read noise will be greatly reduced (one portion of noise per final pixel instead of 4/9/etc). This means that additive binning really can provide some benefit in this case by reducing that source of noise. This is performed in some point and shoot cameras where high ISO images come out with reduced resolution, but also on high end cameras such as the Phase One medium format backs with the Sensor+ setting.

Our location makes us fortunate enough to have been able to spend New Years Eve 2011/12 at the fantastic fireworks show in Central London, attended by hundreds of thousands of others and watched around the world by many more.

Of course a 360 panorama was produced, with an extra twist. A video of the fireworks display is embedded into the panorama., allowing you to see the full 10 minute display inside the scene.

Click the image below to see the video in the panorama. Navigate with the mouse.

A selection of stills are also available in the flickr set below.

The Tablets of the Missing is a 160m long memorial in the American Military Cemetery in Cambridge, UK. The wall contains over 5000 engraved names of dead or missing service personnel from WW2.

Click the image above to explore the full sized image with your mouse.

The engraving along the top reads:

The Americans whose names here appear were part of the price that free men for a second time in this century have been forced to pay to defend human liberty and rights – all who shall hereafter live in freedom will be here reminded that to these men and their comrades we owe a debt to be paid with grateful remembrance of their sacrifice and with the high resolve that the cause for which they died shall live eternally.

This image is originally from about 18 months ago, but the stitching was never satisfactory. Since then software has come a long way and a much better result has been produced. It is however still far from perfect with stitching errors aplenty. Ideally it needs to be shot again, but I am no longer in the vicinity of Cambridge, but will definitely have another go if ever revisiting.

When originally produced, the image was published on gigapan where it received a very humbling response as a few people tagged their fallen relatives. Quite moving.

On 28th/29th Oct, the Relentless Freeze Festival visited London’s Battersea Power station. With it came music, a retail park and of course freestyle competitions for ski and snowboard on a huge temporary snow jump.

Click the image above to revisit the festival. Experience the atmosphere at the top music acts and see the competitors fly off the ramp. For a slower pace explore the retail tent to the offerings or see the art in the graffiti area.

As a bonus, here’s a collection of stills from the event

The Occupy London campagn has been camped out by St Pauls Cathedral and in Finsbury Square since mid October. Their prescence is striking to anyone passing through the areas and despite much support from the Cathedral, did force a temporary closure. On Sunday 23rd October we popped down to see what all the fuss is about and to shoot a virtual tour of the camps in both areas.

Click the image below to visit the scenes and explore the area for yourself.

Introduction

I think that it would be reasonable to say that gigapanography is a fairly well established area of photography. Gigapixel images formed of thousands of images can and are produced regularly with readily available hardware and software. These images have also entered the public eye, with the phenomenal amount of detail visible in them attracting the casual browser as well as marketers.

What we have yet to see quite so much of is for this high resolution image stitching to expand into additional dimension of time. Imagine being able to zoom into, explore and manipulate a streaming video in the same way and to the same scale that you can these existing gigapixel images.

Obstacles

2D images can be troublesome to stitch if adjacent images are not matched for white balance and exposure, though software has become quite good at correcting for these issues. At the overlap between images stitching errors can also arise, especially if a misaligned set-up introduces parallax errors or there is movement within the scene between the capture of images.

When expanding from stitching still images to video streams brings a a few additional problems. The issues identified above with 2D images have to be corrected, matched or at least smoothed not only between adjacent images, but also between sequential frames. At normal video frame rates, manual input the likes of which can be readily done during the post processing of gigapixel images becomes impractical for even a few seconds of footage.

After the concept of expanding stitching from still to moving images came to mind, research into the idea came up with few results. This was made all the more difficult by the ambiguity of the term ”stitching video” which can also refer to the concatenation of distinct clips. I therefore decided to see what i might be able to do myself.

Proof of concept

Using a pair of P+S cameras with similar video capability, I shot a few seconds of SD footage with the cameras adjacent and with their fields of view overlapping. Each frame from the video files were saved to a .jpg file for processing. Processing was done using a simple batch script to guide Panorama Tools through stitching each pair of frames. The script uses only the first pair of frames for alignment (therefore assuming constant relative positioning of the cameras) and all future frames are aligned based on that pair, though blended individually. This prevents artifacts that may arise due to changing detected alignment between frames. However no attempt is made to make blending consistent between frames.

The results of this script were recombined into the above video file and show some success, although some of the earlier identified issues are clearly visible. The blending at the seam is constantly changing, and while the model successfully moves from one side of the seam to the other, the blending on her is also inconsistent, perhaps accentuated by a small asynchronicity between the streams. My coding skills are however currently insufficient to go much beyond this at present. So what are the more thorough capable of?

Advanced

In between all the links related to combining video clips back to back, I was able to find only one item on video stitching as we are discussing it here. This was post on the Autopano blog back in 2008, where the server version is used to present a much more successful demonstration by stitching 3 simultaneous streams with very few artifacts. I have on occasion even seen this technology used to create immersive, 360 degree video where the viewer can change their view point. I have however yet to see it used to create and present high resolution video by e.g. combining multiple full HD video streams to create something the likes of which we hinted to at the start of the post. Perhaps this is a limitation not of the technology used to create such streams, but that used to view them. While many computers today are capable of 1080p playback, I image very few would be able to playback much higher resolutions, especially while manipulating in real time.

I will keep a loose but keen eye on this field.