Strange streak discussion: 2004 Dec 7 APOD

[hazeii3]

I see... if I understand you correctly what you are trying to do is to measure the "strightness" of the pattern. By doing the LSF to that template you want to measure variations from it.

Now wouldn't this analysis be insensitive to what side of the streak is changing shape? I mean it would equally pick up a symmetric narrowing of the streak and an asymmetric narrowing of it, or even a change of idrection without a noarrowing...

If my interpretation is correct, could you come up with a template that picks up what side of the streak changed shape and in what direction. We would like to know if the streak is ondulating, but its width is constant, or if its width is actually changing periodically.

Also have you run an FFT of your signal? Is there a spike somwhere, or is it the brain picking up a pattern that is not really there?
Cheers

Exactly so, it's a sensitive test of the straightness of the trail. Here's an image that overlays the result onto the original trail, and hopefully makes things clearer (note this is an unenhanced diff, which is what the LS fit alogrithm operated on).


(click image for larger version)

Now, if you scale that image vertical by x5 or so and flip it vertically, you'll see the line is precisely the same as the red trace on the graph I posted earlier.

(if the trail is hard to see in that image, see this enhanced version which covers the same area and has the same rotation angle - note the general similiarity of the graph line to the compressed version on the left)

The algorithm would be, as you say, insensitive to the variations in the width of the streak - as long as the variation in width and the streak cross-section are reasonably symmetrical (the streak cross-section is indeed quite symmetric, but we can't assume that about the streak). Detecting one edge versus the other is maybe possible, but it's likely to suffer badly from noise.

It may however be possible to get a reasonable fix on the streak width by finding the optimum setting for the streak width all the way along the trail (as stated before, the graph used a fixed 22 pixels). Adding that shouldn't affect the line choosen by the LS fit, and might give us width info too.

Ref. the FFT, I'll try that later; note though that the main reason I did this was to improve our understanding of the real shape of the trail; I didn't expect to see the 'humps and bumps'.

[Algolei]

...artefacts....

There's nothing wrong with that spelling. It's an accepted form of the word "artifacts."

[Ernst Lippe]

Following up on the discussion of the straightness (or otherwise) of the trail, here's a least-squares fit to it.


(click image for a larger version)

This was created by first rotating the trail so it was approximately horizontal (I used the raw data for this image), then finding the best correlation (i.e. minimum least-squares error) between an intensity template approximating the trail and the trail itself.

That's a nice analysis, especially the suggestion that there is
some periodicity would be very interesting for the proponents
of the bug theory, because it might help identify which type of insect
caused this picture.

However I have one problem with your analysis:
Why did you rotate the image first? It only complicates the analysis,
because your rotation algorithm could introduce some new artefacts,
e.g. it is very likely that it would introduce periodic effects.
When you do a regression on the raw unrotated data it would be
much more convincing.
Another suggestion: Could you try to add a quadratic term to your
regression? If the streak was indeed caused by a falling object,
the path should be parabola (i.e. a second order polynomial).
Now if you find that the quadratic term is not significant (which
I would expect) we can conclude that there is no evidence that
the path is significantly different from a straight line.
It might also be possible that you'll find a significant
positive coefficient, which would be difficult to explain
by people who believe that the trail was made by a falling
object.

[Ernst Lippe]

Also, still lookig for an answer to the following question:

I've scanned much of this discussion, but haven't seen whether or not there has been a follow up on the condition of the lamp. Is it known what the status is of the lamp? Was the bulb and or lens shattered? Or was it just inoperatable? Was there any damage to the lamp housing?

On another board
http://www.badastronomy.com/phpBB/viewt ... &start=125
an email was posted from Geof Carr (an astronomer at Charles Darwin University):

My boss saw how excited I was about it and owed my a favour, so gave
me the afternoon off. I immediatly rushed down to the wharf hoping to
find evidence of damage and a hot rock resting within a crater! However
there was absolutely nothing at all, no burn marks, or even scratches.
I examined every centimeter of the area, taking care to study all the light
poles with binoculars and examined the railings as well - and found
nothing.

[hazeii3]

That's a nice analysis, especially the suggestion that there is
some periodicity would be very interesting for the proponents
of the bug theory, because it might help identify which type of insect
caused this picture.

However I have one problem with your analysis:
Why did you rotate the image first? It only complicates the analysis,
because your rotation algorithm could introduce some new artefacts,
e.g. it is very likely that it would introduce periodic effects.
When you do a regression on the raw unrotated data it would be
much more convincing.

The reason for rotating it first is it makes the software much simpler - once the trail is about horizontal it's easy to slide the template up and down each column of pixels to determine the best fit at that position. I agree with your point about rotation possibly introducing artifacts, but (a) I wasn't looking for periodicity, I was just trying to determine how close the trail is to a straight line, and (b) I considered there was just as good a chance of introducing artifacts on an unrotated image, because then I'd be sampling along a sloping line and pixel-selection effects ('jaggies') could equally introduce artifacts (at least without filtering, which could in turn cause problems...). For reference, the rotation I used was performed using the 'gimp', with -33.6 degree rotation, cubic interpolation and 'super-sampling', which *seems* to be doing a pretty thorough job.

Another suggestion: Could you try to add a quadratic term to your
regression? If the streak was indeed caused by a falling object,
the path should be parabola (i.e. a second order polynomial).
Now if you find that the quadratic term is not significant (which
I would expect) we can conclude that there is no evidence that
the path is significantly different from a straight line.
It might also be possible that you'll find a significant
positive coefficient, which would be difficult to explain
by people who believe that the trail was made by a falling
object.

Note that (as yet) I haven't tried to fit *any* kind of a line to the graph - what you see (on the red trace anyway) is the raw output of the template vs. each column of pixels (i.e. the pixel position at which the sum of the squares of the differences is minimum).

All good points, particular about the rotation; the way I'm cross-checking this is to generate a simulated streak parallel to the original one, feed it into the same software chain and compare the graph that comes out the other end - I'll post the result later.

[Guest]

AT FIRST GUESS, IT APPEARS TO BE SOME PYROTECHNIC VEHICLE, SIZE UNDETERMINED, THAT LEFT THE THINNEST OF SMOKE TRAILS BEFORE IT DETONATED AT THE FLASH POINT. COMPARATIVE ANALYSIS OF KNOWN PHOTOS OF PYROTECHNICS' SMOKE TRAILS WOULD E IN ORDER, B UT I HAVE A MORE INTERESTING QUESTION-IS THE DARK TRAIL AN ABSOLUTELY STRAIGHT LINE, OR IS IT AN ARC OF INDETERMINATELY LARGE RADIUS?

[victorengel]

Following up on the discussion of the straightness (or otherwise) of the trail, here's a least-squares fit to it.


(click image for a larger version)

As a control test, can you run your same procedure on this picture?
http://the-light.com/Photography/bugsimulation.jpg

Or create one similar to that if you prefer. This is the picture I created to simulate the picture we're discussing. The line was created perfectly straight, so if your chart is valid, it should not have humps like this chart has.

[Guest]

I see... if I understand you correctly what you are trying to do is to measure the "strightness" of the pattern. By doing the LSF to that template you want to measure variations from it.

Now wouldn't this analysis be insensitive to what side of the streak is changing shape? I mean it would equally pick up a symmetric narrowing of the streak and an asymmetric narrowing of it, or even a change of idrection without a noarrowing...

If my interpretation is correct, could you come up with a template that picks up what side of the streak changed shape and in what direction. We would like to know if the streak is ondulating, but its width is constant, or if its width is actually changing periodically.

Also have you run an FFT of your signal? Is there a spike somwhere, or is it the brain picking up a pattern that is not really there?
Cheers

Exactly so, it's a sensitive test of the straightness of the trail. Here's an image that overlays the result onto the original trail, and hopefully makes things clearer (note this is an unenhanced diff, which is what the LS fit alogrithm operated on).


(click image for larger version)

Now, if you scale that image vertical by x5 or so and flip it vertically, you'll see the line is precisely the same as the red trace on the graph I posted earlier.

(if the trail is hard to see in that image, see this enhanced version which covers the same area and has the same rotation angle - note the general similiarity of the graph line to the compressed version on the left)

The algorithm would be, as you say, insensitive to the variations in the width of the streak - as long as the variation in width and the streak cross-section are reasonably symmetrical (the streak cross-section is indeed quite symmetric, but we can't assume that about the streak). Detecting one edge versus the other is maybe possible, but it's likely to suffer badly from noise.

It may however be possible to get a reasonable fix on the streak width by finding the optimum setting for the streak width all the way along the trail (as stated before, the graph used a fixed 22 pixels). Adding that shouldn't affect the line choosen by the LS fit, and might give us width info too.

Ref. the FFT, I'll try that later; note though that the main reason I did this was to improve our understanding of the real shape of the trail; I didn't expect to see the 'humps and bumps'.

NEW GUY: HAS ANYONE MENTIONED WHAT TYPE OF DEVICE OR FILM CAPTURED THE ORIGINAL IMAGE? WHAT WAVELENGTHS WERE CAPTURED, ANY DETECTABLE COLORS IN THE STREAK, HOW FAR APART, TIMEWISE, WERE THE PRECEDING AND FOLLOWING FRAMES? THIS COULD BE A N EXTREMELY FORTUITOUS VIEWING OF A MICROMETEORITE IN IT'S LAST MILLISECOND OF DECAY, TRAVELLING DAMN FAST, MJUST BEFORE IMPACT-BUT WHO KNOWS?

[pshisbey]

What do you think it is?

Read the first 20 (or so) pages of this thread then vote here:

http://www.opinionpower.com/results.cgi?id=377021291

[Acorn]

The "flash" appears to be condensation associated with a shock wave - its positioned correctly for an object traveling trans-sonic. The condition of the light pole is coincidental. Someone already remarked that it did not appear to be involved. The streak may well be a particle trail.

I think you caught a meteor.

[hazeii3]

Good work! As I said before...

Flies and bees have wingbeat frequencies around 200 Hz...[can someone] attempt some sort of analysis on the streak to see if they can discern a regularity to it? Either a regular up and down displacement of the thorax/abdomen, or a preferential shadowing either side of the 'flight path'...[would be] evidence for a periodicity of 200 Hz in the 'dark streak' [and] would tend to disprove most rival hypothesis not involving flying insects, as those hypothesis would lead to predictions of no periodicity."

You have found a regularity with the predicted frequency. but there has been an awful lot of directed searching going on (ie many people looking many different ways for a signal and discarding analyses that don't show it). Now can someone repeat that analysis using a different approach / algorithm to see if the purported signal is real? Can you disprove the null hypothesis that there is no discernable signal in the streak? How significant is the result? Can you extract a signal from a patch of sky without the 'streak'. What about the shadow of the wharf (maybe a possible signal due to waves)? Is there any 'binning' effect in the approach you used? Does the jpeg compression algorithm itself cause an artifact?

Good luck.

All very good points. Note though this wasn't a directed search looking for periodicity (I'd given up on that after the FFT null result), rather I was trying to determine the general shape of the trail by finding some quantitative measure of its deviation from a straight line.

To (attempt to) disprove the null hypothesis, I'm creating an image with a simulated streak that is parallel to the actual one but which is linear and has both equal width, and equal intensity along its length, and repeating exactly the same processing steps on it.

I certainly can't extract a signal from a random patch of sky, because the least squares fit bounces around at random (as is evident at the start and end of the graph). If there is a similiar semi-periodic signal in the sky though, it should show up on the control test I'm working on (as per last paragraph).

The shadow of the wharf shouldn't come into it, as that doesn't fall in the area I'm processing (which is a 200-pixel wide cross-section across the trail).

As for 'binning' effects, the ones I'm aware that could cause problems are the image rotation, compression artifacts, asymmetries in the trail (if one side is brighter than the other, an intensity variation would appear as a side to side motion in the least squares fit), and quantisation issues (the magnitude reduction in the trail is only a few steps in the 256 available), clouds in the background.

Re JPEG artifacts, note JPEG uses an 8x8 DCT so I'd expect periodicity to be related to that (for a sloping line, related to 8/sin(a) and 8/cos(a) for the H and V components which is this case is ~9.6 and 14.5 pixels).

Thanks for the reality check

[hazeii3]

As a control test, can you run your same procedure on this picture?
http://the-light.com/Photography/bugsimulation.jpg

Or create one similar to that if you prefer. This is the picture I created to simulate the picture we're discussing. The line was created perfectly straight, so if your chart is valid, it should not have humps like this chart has.

Ok; I'm working on my own simulation as well, but as all the software is set up I just ran your image though it with the same settings as the earlier graph.

First, here's the path it choose through your image:-

(click image for larger version)

Looks like your simulated trail doesn't extend as far left at the real one does - the algorithm chases a cloud edge instead. Then, part way along it you can see the algorithm decided the nearby cloud line was a better fit than your trail, and hopped over to it for a bit. After that, it tracks straight to the object.

Here's the corresponding graph (compared to my earlier one, I've inverted the trace to make it easier to relate to the image, but note I've kept the same vertical scaling to make it easy to visually compare them).

(click image for larger version)

And to make it easier to compare, here again is the graph for the actual trail.

(click image for larger version)

Summarising how they appear to me, your's does indeed appear straight, starts later (at about x=220) and shows no obvious periodic behaviour. How did you create it, by the way? And thanks for putting in the time to create that image; it's good to have a cross-check!

[FastArtCeeToo]

Good work!

Someone pointed out that you rotated the image before the analysis, but this shouldn't matter if the image was uncompressed... should it?

The slight 'bumpiness' of the streak might be developing turbulence. If the streak was only a couple seconds 'old', the effects of turbulence would be slight, but the bumpiness at the left end of the streak should be a bit more developed.

As to the bug theory: the line of reflections in the water go from edge of water to edge of water. To have a bug fly very straight, with an unusual orientation, a shiny bottom to reflect flash, *and* have its wings in a position such that the ends of the wings correspond almost exactly to the edges of water... well, I think that call for stretching the imagination just a bit too much. It wasn't a bug. And the lightpole/lamp location is just a coincidence. And I don't think its a PhotoShop product.

Can every type of small missile or munition be positively ruled out? (And I don't mean alien death rays or microspace ships or cosmic rays or tears in the fabric of space-time or reindeer or reindeer droppings.)

As nearly unbelievable as it might seem, could it conceivably be a small meteor, caught just before impact in the water? (I don't understand the flat angle, however.)

[Bob Peterson]

The BUG theory would seem to be winner by popular demand. I don't fall in love with my theories. Despite the variables problem, I sure would like to see a similar streak(diagonal) produced with that same camera type. Someone please go to a bee box and shoot away for possible flight streak on a digi.

[hazeii3]

The BUG theory would seem to be winner by popular demand. I don't fall in love with my theories. Despite the variables problem, I sure would like to see a similar streak(diagonal) produced with that same camera type. Someone please go to a bee box and shoot away for possible flight streak on a digi.

With the same exposure (1/20th) second and ideally with a flash as well (at the start or end of the exposure).

I'd have a go, but it's winter here (no flying insects).

[victorengel]

First, here's the path it choose through your image:-

(click image for larger version)

Wonderful!

Looks like your simulated trail doesn't extend as far left at the real one does

Yes. This is because of a limitation in my method (see below for link) which resulted in a trail only blurred 999 pixels.

Here's the corresponding graph (compared to my earlier one, I've inverted the trace to make it easier to relate to the image, but note I've kept the same vertical scaling to make it easy to visually compare them).

(click image for larger version)

I'm pleasantly surprised at how straight it is. I actually wasn't expecting it to be so straight. This is great.

Summarising how they appear to me, your's does indeed appear straight, starts later (at about x=220) and shows no obvious periodic behaviour. How did you create it, by the way? And thanks for putting in the time to create that image; it's good to have a cross-check!

I described my procedure on page 90. Look for the photograph of the bee carrying another insect.

[cg19]

A contrail would not crash into a light post. Could be debris from space. We have more trash up there now than ever. That is the most likely in my view. Other than that, I think the ball lightning is a slight possibility.

[victorengel]

As to the bug theory: the line of reflections in the water go from edge of water to edge of water.

Do you mean from far shore to near shore, as in the reflection of the clouds?

To have a bug fly very straight, with an unusual orientation, a shiny bottom to reflect flash, *and* have its wings in a position such that the ends of the wings correspond almost exactly to the edges of water... well, I think that call for stretching the imagination just a bit too much. It wasn't a bug.

Bugs fly straight all the time, especially bees (except when they are near home or foraging), and also especially over short time periods, like 1/20 second. Even the most wildly zigzagging insect has a zigzag frequency that is longer than the exposure.

We don't know the orientation of the insect, by the way. What looks to be a wing pointing up and one pointing down does not imply the insect was banking like that. In my simulated picture, there were two similar lines. One came from the wing -- the other from the head. Also, even though the flash is brief, it nevertheless spans a period of time. It's possible for the wings of the insect to travel significantly during this time, depending on the flash duraction (we need some more data on this from G3 experimenters or Canon experts). It's even possible that the two streaks are glints off the SAME wing at two different times.

The top "wing" extends significantly above the far edge of the water. So I don't think you can use the coincidence in placement card for that. You didn't mention the alignment with the pole. Others have, though, so let me respond to that by saying it's not in perfect alignment. Additionally, given the size of the insect image and the angle of the tragectory, the odds of intersecting a pole are very high because of the number of poles.

[J Joy]

Thanks to Ernst Lippe for help with my question about the condition of the lamp.

Another idea for possible analysis: Would it be possible to analyze the streak along its cross section for meaningful information. If the streak were from a bug the "density" (to use an old film term) may be different than if the steak were, say, a tubular structure exsisting in the atmosphere. Just a thought. I have no way to follow up on it.

JJ

[Ed in Oregon]

Thanks to Ernst Lippe for help with my question about the condition of the lamp.

Another idea for possible analysis: Would it be possible to analyze the streak along its cross section for meaningful information. If the streak were from a bug the "density" (to use an old film term) may be different than if the steak were, say, a tubular structure exsisting in the atmosphere. Just a thought. I have no way to follow up on it.

JJ

A tubular structure in the atmosphere would have a higher density at the edges than in the center. Just look at a clear plastic tube to see what I mean. This streak seems to have almost equal density across it, or slightly more density in the middle, like a shadow. See some of the difference images back about 9 or 10 pages.

[Guest]

A tubular structure in the atmosphere would have a higher density at the edges than in the center. Just look at a clear plastic tube to see what I mean. This streak seems to have almost equal density across it, or slightly more density in the middle, like a shadow. See some of the difference images back about 9 or 10 pages.

And an equal density is, of course, exactly what you'd expect for an insect body motion-blurred along its long axis (as long as the body was moderately longer than wide). There would be a small loss of density away from the axis because of the narrowing between thorax and abdomen, and an even smaller loss due to the rounding at each end.

[RJN]

I've read about half of the comments and enjoyed the great depth of the discussion, but of course not the flames. For next week I will either write a new APOD or just put a link line at the bottom of an APOD linking to some already existing pages (like Cloudbait's) discussing possible solutions.

I am leaning toward the "bug" solution myself. I do realize that we may never know for sure. Here is a line of thought that I have not seen but I might have missed. How far was the bug from the camera? I think this can be estimated from the angular size of the bug and by assuming a "normal sized" bug like a bee. I think it was quite close! Is this distance consistent with the brightness of reflected flash at the end of the trail?

Is the depth of the darkness consistent with a bug flying by? The angular speed of the bug should be computable. If the darkness is darker than a path created even by a perfectly black bug, then this is a problem.

In general, I would like to thank everyone involved (now and into the future) for participating. Any answer was interesting especially given the prospect that an important celestial event MIGHT have been happening. The collective expertise of the APOD audience is deep and possibly unique. We will likely post another challenge sometime next year.

- RJN

[victorengel]

And an equal density is, of course, exactly what you'd expect for an insect body motion-blurred along its long axis (as long as the body was moderately longer than wide). There would be a small loss of density away from the axis because of the narrowing between thorax and abdomen, and an even smaller loss due to the rounding at each end.

There is not necessarily a narrowing between the thorax and abdomen. Certainly this is true for wasps and bees, but not all insects. Even for wasps and bees, depending upon the perspective, there may be no apparent narrowing since there is significant overlap. Consider a nearly posterior view, for example.

I would also caution that we should not assume that the direction of travel is parallel to the body's orientation. I believe the body is usually inclined moderately. I imagine the angle varies significantly by species.

[victorengel]

I am leaning toward the "bug" solution myself. I do realize that we may never know for sure. Here is a line of thought that I have not seen but I might have missed. How far was the bug from the camera? I think this can be estimated from the angular size of the bug and by assuming a "normal sized" bug like a bee. I think it was quite close! Is this distance consistent with the brightness of reflected flash at the end of the trail?

The normal flash range of the camera, according to dpreview.com is:
• Wide: 0.7 - 5.0 m (2.3 - 16.4 ft)
• Tele: 0.7 - 4.0 m (2.3 - 13.1 ft)

for ISO 100. I can't tell from the EXIF what the ISO used actually was, but I can't believe the information is not there. The EXIF data seems to have detailed flash information, including something Thumbsplus calls Flash bias. It seems like it should be possible to identify the distance to subject quite readily with expert knowledge of the EXIF information and an assumed albedo for various parts of the insect/pollen load. There are a lot of variables, though, and I really think EXIF data expertise is required. The image is not blown (saturated), though, so it should be usable.

Is the depth of the darkness consistent with a bug flying by? The angular speed of the bug should be computable. If the darkness is darker than a path created even by a perfectly black bug, then this is a problem.

I did the calculations for the depth of the darkness, and it seemed to be appropriate. It was just a rough approximation, though. I'd take the assumed length of the insect profile (intersection of the insect's path and the insect in the part of the darkness in question) and divide that by the total length of the trail of darkness. The result is the amount of dodging the insect did to the image.

I approached this a different way, too, and I will redo this more carefully. I took a black blob about the size I presumed to be the size of the insect imaged and placed it on a white background. Then I used Photoshop's motion blur tool to spread the blob over 999 pixels (not enough). I then blended this to a before picture using the multiply blending method and achieved a path that was slightly darker than the study picture. The amount darker might have been close to the amount of additional blurring I needed to do, but I hit the Photoshop maximum.

[M. John Lalonde]

Thanks to "Ed in Oregon" for your great questions.

I am not suggesting that the raindrop travelled along the path of the dark shadow, but rather I was trying to suggest that the raindrop was blocking the light which caused the appearance of a shadow in a straight line radiating from the raindrop, similar to a sky-shadow from an aircraft's contrail. However, in this case the shadow could be a few inches long, but with the appearance of covering a very long distance crossing the landscape.

(Keep in mind that a sky-shadow from a contrail is caused when the water vapour is dense enough to block the light from the sun. The only direction in which the water vapour is dense enough to block the light is when the sun is directly in line with the LENGTH of the contrail, which may be thousands of feet long. There isn't enough vapour density across the width to create any shadow that is visible by the naked eye. This explains some of the features of contrail sky-shadow; 1) the sky-shadow appears as an extension of the contrail; 2) the sky-shadow is always a straight line; and 3) a sky-shadow's life is very brief, as it only lasts as long as the sun is essentially in direct alignment with the length of the contrail.)

Now, I would like to offer a second theory for everyone's consideration:
I would like to suggest that the bright spot in the picture is a halo caused by light hitting the lense of the camera.

Sometimes we see one or more halo in our camera's view finder (reflective lense type such as SLR) when we aim the camera at a bright object such as the sun or sunlight reflecting off water. Depending on the configuration of the lense, you could have one or more overlapping halos. This could explain the apprearance of a circle(s) in the middle of the bright spot in the photo. The source of the light may be sunlight, a reflection off the water, or source of light in the distance.

The apparent appendages radiating from the circle could be a product of the camera's lense construction. These could be associated with a polarized filter on the camera that may not be lined up with the horizon. In which case, the polarized filter would only block reflected light up to 90 degrees from its horizontal position. This could explain partial rays from the bright spot. Do the apparent appendages seem to form another halo of much larger radius that bisects the first smaller halo? Can the photographer tells us more about the camera type, lense configuration, and filters?

The dark line may also be associated with the camera's lense construction. The lense could cause a straight line to appear as emanating from the smaller halo or dividing the radius of the much larger halo.

I am not convinced on the cloaked Klingon Bird-0-Prey hypothesis.