photo sharing and upload picture albums photo forums search pictures popular photos photography help login
Samir Kharusi | all galleries >> Galleries >> How M31 photographs at a Dark Location > Measuring Skyfog from Camera JPEG
previous | next
16-DEC-2004 Samir Kharusi

Measuring Skyfog from Camera JPEG

HOW TO MEASURE YOUR OWN SKYFOG: Set your digital camera to ISO 800, aperture to f4 and just place it on the ground looking at zenith. Vary exposure length till you get the peak at mid-histogram (on the back-of-camera histogram!!!). Use either the translation scale given below, easiest, or the following calibration formula to get an estimate of your skyfog, likely accurate to within 0.1 Mag:

Mag/sq arc-sec = 13.93+2.5*log10(seconds to mid histogram at ISO 800 and f4)

High altitude, extraordinarily dark sites on exceptional nights have been reported to be at Mag 22/sq arc-sec (28 minutes to mid-histogram!), but more accessible pristine dark sites can also take up to 16 minutes to reach mid-histogram. In my experience, I would classify any site that takes longer than 12 minutes as very dark and pristine for amateur astroimaging. So you can immediately see how much brighter your skyfog is compared to a pristine dark location. It's that ratio of exposure times to mid-histogram that is important.

TRANSLATION SCALE: You can use the translation scale below to convert "Minutes Exposure to mid-histogram at ISO 800 and f4" to Magnitude per square arc-second and to Visual Limiting Magnitude:

Mins-----------------28---17.7---11----7.0----4.4---2.8---1.8---1.1---0.7---0.45---0.28
Mag/sq arc-sec----22---21.5----21---20.5---20---19.5---19---18.5--18----17.5-----17
VLMag--------------6.6---6.4----6.1---5.8----5.5---5.1----4.8---4.4---4.0-----3.5----3.1
Bortle-----------------1---2----3-----4-------5-------6-------7-------8-------9---------------

TO SPEED UP THE MEASURING: Nobody likes to waste precious dark time waiting 16 minutes for a camera to measure skyfog. You may of course increase ISO and open your lens to a faster focal ratio. Each doubling of ISO from 800 implies a doubling of your time-to-mid-histogram before entering the Translation Scale. Each f-stop wider than f4 also implies a doubling. Let us say we find it takes half a minute to reach mid-histogram at f1.4 and ISO 3200. We need the time to be for f4 and ISO 800 to use the Translation Scale. ISO 3200 is 4x faster than 800, and f1.4 is 8x faster than f4 (difference of 3 f-stops). So we multiply that half minute by a factor 4*8=32x and use 16min on the Translation Scale. It's obvious therefore that with a fast lens we can measure the skyfog very quickly, no matter how dark the site is.

WHICH HISTOGRAM TO USE: The above, out-of-the-camera Jpeg has been reduced in size only, to save on bandwidth. Should be suitable for measurements, histogram plots, whatever. Following queries, I downloaded this image, opened it in Photoshop and displayed its histogram. The result is shown on the last slide in this gallery. I took a sample of the background sky from the above Jpeg frame (keeping away from the vignetted corners and also from the galaxy itself) and got Luminosity readings around 33% of full-white. That's for a 5-minute exposure at f4 and ISO 1250. To translate this roughly (ignoring gamma since we need just a ballpark figure) to my standard reference of "Mid-histogram (50%) at f4 and ISO 800", we need to multiply 5 minutes by 50%/33% and ISO1250/ISO800, yielding 12-minutes or Mag 21+ /sq arc-sec, a satisfactorily dark sky in that specific direction of M31! By following this simple example you can, in fact, deduce your skyfog level from any frame taken at any ISO, focal ratio and exposure length, provided the exposure was long enough for the skyfog to be well detatched from the origin in the histogram of its Jpeg file. Do NOT use the 16-bit Linear-converted file for these estimates, use nonLinear conversion to 8-bit Jpeg (as you do for normal, daytime photography).

STARTING FROM A BUNCH OF SUBS TAKEN DURING AN IMAGING RUN: OK, you did not plan on measuring your skyfog, but many weeks later decided to find out how much skyfog you had anyway. Take any typical one of the Raw subs that you used for stacking. Convert it from Raw to Jpeg using "Normal" conversion, not "Linear"!!! Display its histogram in any image editor and see where the skyfog peak falls, let us call this PCT, eg if the peak is at 23% along the X-axis then we will have PCT=23. You already know the ISO you used and the FR (focal ratio of your OTA, e.g. if it was f6.7 then FR=6.7). Finally the length of the subexposure was EXP in minutes(!), e.g. if the sub was 100sec, then EXP=1.7. To get the equivalent time (Mins) required for the translation scales displayed above, simply use:
MINS = EXP * (ISO / 800)*(50 / PCT)*(4 / FR)*(4 / FR)
Use the resultant MINS to enter the translation scales displayed above and you will have a pretty good estimate of your skyfog at a site you were at weeks or even years ago :-)

HOW DARK CAN A REALLY, REALLY DARK SITE GET?: While amateurs usually refer to Visual Limiting Magnitude, professional astronomers prefer referring to the skyfog through a V filter compared to a star (e.g. the skyfog at the darkest, remotest, high altitude sites have been reported to be about as bright as a Magnitude 22 star placed in each square arc-second). One can make an accurate translation scale from "Exposure to Mid-Histogram at ISO 800 and f4" to "Mag/sq arc-sec". I measured the skyfog at a high altitude, reasonably remote site, tens of km from the nearest village, at 16 minutes to mid-histogram and calibrated it against a Unihedron Sky Quality Meter (21.4 Mag/sq arc-sec). In theory these scales should be reproducible and consistent. However, the Unihedron SQM has a very wide acceptance angle (80+ degree cone) that is probably far too large for a light polluted location, since there are always annoying, localised light sources somewhere, likely leading to readings that are brighter than the true zenith is. The DSLR is a more flexible instrument since we can set its acceptance angle, by simply changing the lens, to whatever is relevant for our particular imaging circumstances.

VISUAL LIMITING MAGNITUDES: For convenience we may talk of Visual Limiting Magnitudes. Note, for convenience only. The VLimMag translation scale shown above is not absolute. It is observer dependent and very subjective, but I found it to be sort of OK at a heavily light-polluted site when I glance up quickly. Staring longer enables me to see deeper, by, say, up to half a Magnitude. It is NOT a linear function of the other two scales, because the sensitivity of human vision seems to plateau out for dimmer stars. To make things worse, there have been reports that some rare, very experienced and exceptional observers can see deeper by as much as two Magnitudes. However, the above scale is based on the extensive research work of B.E.Schaefer and later put into a web-based calculator by K.Fisher. You can check out the numbers for your own site and decide whether it is to be shifted to the left or to the right, depending on your visual acuity, sky knowledge, etc. It's not important exactly where you wish it to end up. Very likely your visual assessment will be within a half VLMag of the above translator scale. But for ease of phrasing, whenever I refer to a number for VLMag on my webpages I am referring to a number obtained using the above translation scale.

The Bortle translation above is my guesstimate since I am unable to reconcile Mr Bortle's Mag/sq arc-sec with my VLimMag. Basically just MEASURE your skyfog and work with Mag/sq arc-sec.

WHY BOTHER?: Why is it useful to know how much brighter your skyfog is compared to a pristine dark site? Budding astrophotographers tend to assume that they have massive light pollution at their home sites because their images do not have a black background. They may or may not. Better find out up front. Actually, as seen from the above example, an image that has been adequately exposed should in fact NOT have a black background. The SNR (Signal to Noise Ratio) in your final, stacked and processed images has a direct relationship to how much skyfog you have. Let us say that you are at a site for which the skyfog takes 1-minute to reach mid-histogram and you have exposed enough frames to end up with an integration time of one hour. You then repeat the exercise at a dark site for which the skyfog takes 15-minutes to reach mid-histogram. At this darker site you will require 1/15th the total integration time as at the first site to achieve the same SNR. Ie 4-minutes integration time will yield a final, processed image with the same SNR as your earlier one-hour stack.

INTEGRATION TIMES: For similar SNR from two different sites, the total integration times used in the stacks of frames need to be in direct proportion to how much more skyfog you have at the more light polluted site. Six times more skyfog (eg semi-rural suburbia), six times longer integration time. So to reproduce those beautiful DSO images that took 3-hours integration at a dark site will require 18-hours integration at the light polluted site; for the same SNR. Basically, extend integration times to the limit of your stamina, at any site.

EXPOSURE TIME PER FRAME: The thing to remember is that with proper post processing we can remove most sources of camera noise, but not all. With modern, low-noise DSLRs used at ambient temperatures cooler than, say, 25deg C, the use of darks, flats and bias frames removes most noise very effectively. But not the Read-Noise of the camera and the Poisson Statistical Noise in the skyfog. Longer and ever longer integration times will reduce the skyfog statistical noise by a factor proportional to the square root of the integration time. But when we stack frames, each with a short exposure, we are also stacking the camera's Read Noise in each frame! We need to separate this Read Noise from the skyfog statistics. Fortunately there is a way to do this. Expose each frame so that the skyfog peak is well removed from the origin of that back-of-camera histogram. An often-mentioned rule is to expose so that the skyfog peak is at a quarter to a third of the horizontal axis of the back-of-camera-histogram. Tracking is usually the main limitation as to how long each exposure can be. So how short can we make each exposure and still feel reasonably secure that we capture as much faint detail in our DSO as we can ever hope for? My own rule-of-thumb: use the highest ISO your camera offers, but not one that you have to invoke some special function. ISO 1600 seems an excellent choice for current (2006) Canon DSLRs. It actually seems to have lower Read Noise than lower ISOs. Expose each frame long enough so that you can see a distinct gap between the TRAILING edge of that skyfog mountain and the origin on the back-of-camera-histogram. In your post processing, after calibrating with darks, flats and bias frames, you will very likely use the trailing edge of that skyfog mountain as your "Black Point" anyway, so you are not likely to sacrifice any faint data on your DSO. Can you do even better by exposing each frame longer? Theory says that the improvement in final SNR diminishes asymptotically from this stage onwards. So, yes, expose each frame longer if you can, even up to mid-histogram, but you are eating into the dynamic range of your camera and risking mistracking, cosmic rays, planes, satellite tracks, whatever. You may of course recover dynamic range by simply switching to a lower ISO. How much is a "distinct" gap? The skyfog mountain is simply displaying statistical variations within it (OK, also vignetting), so if we examine it on an expanded vertical scale (eg we use a logarithmic luminance axis) the trailing edge will extend even closer to the origin than we see on the back-of-camera-histogram. So try to keep a healthy gap, but I expect that any clear gap (10%?) should be quite ample. You will find that even at a dark site, the use of fast optics, eg f4, and ISO 1600 means you need individual exposures no longer than, say, a couple of minutes to do excellent DSO imaging. Of course I trust you also realise that shorter individual exposures means a larger stack to achieve the same integration time. Most DSOs need at least 2 to 3 hours integration time even at a dark site to come out nicely, so shoot a whole night on just one or a maximum of two DSOs. And you will still wish to have shot more frames.

FROM MY OWN EXPERIENCE: A pristine dark site would take 12+ minutes at f4 and ISO 800 to mid-histogram (or VLMag 6+ on the above scale), my own home base fluctuates between VLMag 3 and 4 at zenith (much worse 30 degrees or more off zenith and rapidly degrading to VLMag 1 when just 20deg above the horizon). The more transparent your sky is, lack of dust, the less degradation you get as you go away from zenith and the darker the site feels. One clear winter night/morning at 4am I even measured my home light pollution at VLMag 4.4 at zenith. If only that happened often... I would suggest that if the sky at your homebase is VLMag 4.5 or better at zenith then it`s probably better just to use more frames in your stacking (and filters) than heave all that astro equipment to a darker site. People who have a VLMag 4.5 sky often report that they can "just about" make out the Sagittarius Milky Way. So, IMHO, if you can make out any part of the Milky Way at all at your home base, then you are likely better off stacking loads and loads of frames at home, rather than lugging all that equipment into the boonies.

TO BE NOTED: The translation to Mag/sq arc-sec above is an actual calibration at a dark site. The translation to VLimMag is a subjective estimate and will always remain observer-dependent. All references to mid-histogram in the above discussion refer to the BACK-OF-CAMERA HISTOGRAM (!!!), or the histogram of the Raw frame converted using nonLinear, as used for normal daytime photography. A histogram of the same Raw frame that has been converted in Linear fashion, as used for astrophotography, will be much darker and very likely dependent on camera brand or model. The reason is that there's a gamma curve that is incorporated when you convert using nonLinear or when the camera does the conversion internally straight onto Jpeg. To maintain consistency between camera brands I restricted the above discussion to back-of-camera histogram (equivalent to a Jpeg capture) and a fixed ISO, otherwise data will not be portable between different camera brands. Consequently the above discussion is also not portable to astroCCDs either. You will first have to calibrate it against a normal digicam at ISO 800.

THE GOOD NEWS: Modern DSLRs have low enough thermal noise at reasonable ambient temperatures (25deg C and cooler) that with proper procedures (darks, flats and bias frames used in calibration) they are capable of beautiful astro images of very dim DSOs, and even narrowband imaging using modified DSLRs. They also have low Read-Noise compared to many astroCCDs. This enables their use in the desirable, so-called skyfog-limited mode with quite short exposures using relatively fast optics; a couple of minutes per frame with f4 optics, even at very dark sites. Much shorter exposures may be used at light polluted sites and still remain in the skyfog-limited regime. Many people with excellent tracking prefer to shoot longer exposures per frame, perhaps even at a lower ISO. What really matters is that the skyfog statistics in each frame are not interfered with by your camera's Read-Noise and that you have ample integration time. I discuss the question on minimal sub-exposures more extensively here: http://www.samirkharusi.net/sub-exposures.html

THE BAD NEWS: Integration time! There is no obvious way around having to achieve proportionately longer integration times at light polluted sites compared to a dark site. Shorter exposures per frame simply imply having to take even more frames. Let us work out some numbers: Assume that our mount can deliver satisfactory tracking for 6-minute exposures at our desired focal length. At a very dark site we can often achieve a satisfactory (not great, but satisfactory) image of a faint fuzzy with about an hour integration time. A "great" image will typically require 3 to 5x as much integration time, even at a dark site. On the other hand, star clusters such as globulars require much less integration time. Depending on the focal ratio of our OTA we may find that at ISO 1600 there is a nice gap between our skyfog mountain and the origin, on the back-of-camera histogram using 3-minute exposures. So we decide to use 20x3minute exposures to deliver the one-hour integration time. Back at our home base in semi-rural suburbia we find that the skyfog is 6x brighter than at the dark site. This is not atypical and we need to shoot 6-hours integration time to achieve the same final SNR. We will also likely find that at ISO 1600 there is already a nice gap between the origin and the skyfog mountain with exposures as short as one-minute. So we have a choice, shoot 360x1min frames at ISO 1600 or use longer exposures at a lower ISO? We wish to achieve 6-hours integration time in all cases. Perhaps we decide to settle on 60x6min frames at ISO 400. Either route is valid and the resultant SNR after stacking should be similar, 20x3min at ISO 1600 at the dark site compared to either 360x1min at ISO 1600 or 60x6min at ISO 400 at the light polluted home base. There is no getting around having to increase the integration time enormously. To keep the total number of frames reasonable it is often wise to use a lower ISO at a light polluted site. In all cases you also wish to have a clear gap between the trailing edge of the skyfog mountain and the origin, so that Read-Noise does not mess up your skyfog photon statistics. The greater the number of frames in your planned stack, the clearer the gap has to be and the more dependent you become to the absolute linearity in the response of your camera. I often try to use exposure length and ISO that keeps the total number of frames in a stack under a hundred.

Canon EOS 1Ds
300s f/4.0 at 600.0mm full exif

other sizes: small medium original auto
comment | share
Guest 23-Jul-2013 00:49
Samir
What a terrific piece of information on a topical issue. Thanks a lot for this. One question, does the camera need to be focused or would it be better to have it out of focus? I was thinking that a few bright stars or satellites in the field might afect the histogram. Therefore, having the camera out of focus would average out the light over an area. After all you can not focus on sky fog so does it matter? Regards Doug
Rob 22-Feb-2013 06:04
I was wondering if you could provide sources to justify the equations you present for calculating sky-glow
Much Obliged
Tom 21-Dec-2010 21:47
This is alle very interesting and I want to try your advises on my astrophotography practise, but I have a few questions:
I understand the skyfog-limited mode of working and I understand the advantages of relative short exposures as long as the exposure time is longer than the minimum time that you explained. But what if I am able to expose my frames much longer than the mimimum. What advantage can I expect from that?
Let me explain with this example: I did 8min exposures with f/3.5 optics on ISO1600 on an EOS 20D under fair circumstances (SQM 20.2). No guiding problems, no sattellites. I did 10 of these frames. Up to now I never did more than 3 min with this setup. Ofcourse there is a lot of sky fog in the 8min RAW data, but it turns out that the mean (top of the histogram) is 22000 on a 16-bit scale (65500 max). The max in bright stars is 58000 so there is no clipping in my RAW's. I managed to make a good image using dark frame subtraction and proper postprocessing.
From your article I believe the sky fog limited (=minimum) exposure time in this situation is much lower, maybe as low as 0.5-1min.
So can I actually expect better results when going as high as 8 min? Will 10x8min give better results than 80x1min in this situation? If so, what is the limit? I read something about losing dynamic range because all my pixel values will be between 20000 and 65000 instead of maybe 2000-65000 when doing shorter exposures.
I saw you wrote about the effect of quantum noise when the histogram is too far left. Does that not plead for longer exposures.
It seems to me there must be a sweet spot with optimal exposure time when you ignore problems with sattelites, clouds, mount problem, autoguiding problems. Do you have any ideas about this?

Thanks,
Tom Zalmstra
The Netherlands
Marc Dubbeldam 02-Feb-2010 12:24
Turns out that I am missing something... As Samir kindly pointed out in an e-mail sent to me (see his response attached below), I made the mistake of equating the ISO setting on the camera to the sensitivity of the chip, which implies that the chip would somehow (magic?) be able to register more photons at higher ISO settings. This, of course, is complete nonsense and I should have known better - silly me!

Anyway, so I offer my humble apologies to Samir for trying to prove him wrong, and here's to a better understanding of the workings of digital cameras:

----------
When you shoot astro deep sky, forget everything you ever learnt about ISO in daytime use. Let us say that a 3 minute exposure captures 1,000 photons onto a pixel and these generate 500 electrons from that pixel. The number of electrons generated have nothing much to do with ISO. They will remain essentially 500 regardless what ISO we set. The ISO simply determines whether the camera should display them as 500 units (of some kind, like a pixel luminance/intensity reading going from 1 to 65536 in the 16-bit image file after conversion from Raw) at ISO 200, or as 5000 units at ISO 2000. But in astro post processing it does not really matter whether we start off with the 500 or the 5000. What matters is the relative noise of the skyfog. Obviously, numerically the skfog noise in the ISO 200 converted image will be 10 times smaller than in the ISO 2000 converted image. But the proportions in SNR are identical.

So, what ISO should we choose? Keep away from special-function ISOs like 3200+ or 100-. But in current model DSLRs ISOs between 200 and 1600 behave in predictable ways. So choose an ISO that spreads out your skyfog histogram nicely without cramping the top end (leading to loss of dynamic range). Why spread the skyfog? Well, so that you minimise quantisation errors. Obviously in the example above the quantisation error in going from 500 to 501 is proportionally much larger than 5000 to 5001. So less quantisation lumping in the A to D converter, smoother tonality.

The Read Noise in Canon DSLRs is lowest at ISO 800 and 1600, but it increases by less than a factor 2x when we go down to ISO 200. So within that ISO range, 200 to 1600, we are free to choose whatever plonks the skyfog at around 20% to 30% on the histogram, nice spread, plenty of head room.

To confirm that sensitivity has nothing to do with ISO, please realise that the effective ISO of most astroCCD cameras is less than 160!

I believe that you make the erronious assumption that setting ISO to 200 from 2000 lowers the number of photo-electrons released by a factor of 10. Nope. Same number of electrons, only the display changes, an amplification factor, for both signal and noise. Hence same SNR. Hope I am answering something that you actually queried... (You certainly have - Marc)
----------


Kind Regards,


Marc Dubbeldam,
Durham, UK
Marc Dubbeldam 02-Feb-2010 09:14
Dear Samir,

Thank you for this excellent article; being able to objectively measure the sky brightness using a dSLR is really helpful. I am planning to use this technique to evaluate some potential sites within easy reach from my home; I am keen to find out how they compare!

However, I have an issue with some of your statements in the final section (THE BAD NEWS: Integration time) where you assert that "either route is valid and the resultant SNR after stacking should be similar, 20x3min at ISO 1600 at the dark site compared to either 360x1min at ISO 1600 or 60x6min at ISO 400 at the light polluted home base".

Assuming a "perfect" camera (i.e. no dark current and no read noise - a reasonably valid assumption as the dominant source of noise is the photon noise), the Signal-to-Noise Ratio (SNR) in each pixel equates to:

SNR = TargetSignal / sqrt (TargetSignal + SkyglowSignal).

More often than not, the TargetSignal negligible compared to the SkyglowSignal (particularly in the faint regions of the DSO where the SNR matters most), and the equation can thus be simplified to:

SNR = TargetSignal / sqrt (SkyglowSignal).

This proves the first part of the statement to be correct: 20x3min at ISO 1600 at the dark site is equivalent to 360x1min at ISO 1600 at the light polluted home base, as the total TargetSignal (after stacking) increases by a factor 6 (360 min compared to 60 min exposure time) and the SkyglowSignal increases by a factor 36 (6 times the exposure time x 6 times the brightness of the skyglow) and hence the noise by the square root of 36 = 6. So both the signal and the noise increase by a factor 6, and hence their ratio (the SNR) remains the same.

However, this is clearly not equivalent to 60x6min at ISO 400 at the light polluted home base; reducing the camera's sensitivity by a factor 4 reduces both the Target-Signal and SkyglowSignal by a factor 4, but as the equation shows, the noise only decreases as the square root of the SkyglowSignal, i.e. by a factor of only 2. This means that the signal-to-noise ratio decreases by a factor 2!

Whilst I accept your statement that it may not be desirable to stack 360 individual frames (never tried to be honest!), I don't believe that it is true to suggest that the resultant SNR will be equivalent. Or am I missing something?

Any further contributions to the discussion would be appreciated.

Kind Regards,


Marc Dubbeldam,
Durham, UK