Does this effect have a name? If so, what is it?

[JeanTate]

Consider this thought experiment, conducted in a toy universe.

You select tens of thousands of images of galaxies, and discard those in which two galaxies appear to be interacting or overlapping, and those in which the central galaxy appears to be 'edge-on' or highly inclined. You prepare images of the remaining galaxies so they are framed consistently: centroid in the center of the image, scale such that the galaxy appears to fill ~a quarter of the image, background ('sky') a consistent black, galaxy colors rendered consistently, and so on.

You then ask an awful lot of ordinary people ('classifiers') to look at the galaxies and classify them. You offer them just two choices, 'spiral' and 'elliptical'. There's a tutorial, and some practice images. You make sure the classifiers don't talk to one another, that they are not asked to classify any galaxy more than once, etc, etc, etc.

You have an automated 'classification collection and analysis' system, that no one can interfere with. The system outputs the total number of classifications, the fraction of those which are 'spiral', and the fraction of those which are 'elliptical'.

After your classification project has been going for a while, you check on the results for three galaxies, respectively A, B, and C below.

A -> <- A
B -> <- B
C -> <- C

You find, for each of the three, that the total number of classifications is 10, and that the spiral (elliptical) fractions are 1.0 (0.0), 0.0 (1.0), and 0.5 (0.5), respectively.

No surprises, right?

You keep on collecting classifications, and observe the following, as the number of classifications per galaxy rises:

* for C, the spiral fraction seems to go on a drunken walk; sometimes above 0.5, sometimes below, but tends towards a value that's not far off 0.5
* for A, the spiral fraction remains at 1.0, until - after N classifications - it drops below 1.0; it never returns to 1.0
* for B, the spiral fraction remains at 0.0, until - after M classifications - it rises above 0.0; it never returns to 0.0

After an initial unanimity, spiral (elliptical) fractions will eventually become overwhelming majorities; if a galaxy does not begin with a unanimous classification (assume 'initial' is ten classifications), it will never get one.

Is there a name for this? If so, what is it?

More questions later, but to close with just one: random accidents - classifiers 'clicking the wrong button' - can happen in this toy universe; this is one possible reason why unanimity disappears (and eventually it will affect every galaxy whose initial classification was unanimous). If this were the only cause, and if all classifiers made random mistakes at the same rate (over the very long term), would classifications for 'perfect' galaxies tend towards a fixed value? If so, what?

[geckzilla]

Noise?

[Nitpicker]

What name? I may have misunderstood exactly which "this" you are talking about, but I would be tempted to call them errors. In the case of A and B you might call them one-sided errors. There are always errors when measurement and judgement are involved.

Personally, I think C would end up somewhere between 0.5 and 1.0. If you rounded all the results to just 2 or 3 significant figures, you'd probably hide the errors.

Edit: Noise is also a good word, a la geckzilla's post.

[Chris Peterson]

Noise?

It's a kind of bounded noise. The real data would be presented with a statistical uncertainty. There are clever ways of drawing error bars in cases like this (where the uncertainty in one direction would place the value outside of physical bounds). There are also analysis techniques for using these kinds of uncertainties in calculations, so they propagate correctly.

[JeanTate]

Thanks everyone.

There are clever ways of drawing error bars in cases like this (where the uncertainty in one direction would place the value outside of physical bounds). There are also analysis techniques for using these kinds of uncertainties in calculations, so they propagate correctly.

Where to find details of such clever ways?

Leaving the toy universe and returning to the real one: if I have thousands and thousands of galaxies, and if the number of classifications per galaxy (call it N) ranges from 15, say, to 150, how do I take account of the dependence of the bounded error on N? Especially for those galaxies whose 'spiral fraction's are close to 1.0 (and those whose 'elliptical fraction's are close to 1.0)?

[BDanielMayfield]

Having just watched the Winter Olympics, it occurs that you could handle this whole problem of weeding out erroneous judgments the same way they do in sports were subjective judging of performance is involved; toss out small number of high and low scores. As long as you toss out a slightly higher percentage of judgments than the observed error percentage the clear cases will be corrected back to 100 %, while for the somewhat vague cases this would also eliminate the small numbers of erroneous entries. For cases where you get a 50% result tossing the same numbers of spiral and elliptical responses leaves your result unchanged.

Bruce

[Chris Peterson]

Having just watched the Winter Olympics, it occurs that you could handle this whole problem of weeding out erroneous judgments the same way they do in sports were subjective judging of performance is involved; toss out small number of high and low scores. As long as you toss out a slightly higher percentage of judgments than the observed error percentage the clear cases will be corrected back to 100 %, while for the somewhat vague cases this would also eliminate the small numbers of erroneous entries. For cases where you get a 50% result tossing the same numbers of spiral and elliptical responses leaves your result unchanged.

I think that would only work if your scoring ends up following a Gaussian distribution. Any skewness, or other effects which made the distribution asymmetrical, would introduce a bias. I'm skeptical, given any real controls, that events that are subjectively judged actually end up with symmetric Gaussian scores.

[Ann]

This reminds me slightly of a method demonstrated by a math teacher who teaches at my school. Several years ago he taught in a school in Eslöv. You can find Eslöv on the map at left, close to the center of the map.

My colleague asked his student to write down the distance between Eslöv and Lund. You can find Lund on the map to the lower left of Eslöv. The true distance between Eslöv and Lund is 23 kilometers, so it isn't far, and most of the students lived in Eslöv. Since Lund is a bigger place than Eslöv, all the students would frequently visit Lund. They had a good intuitive understanding of the distance between Eslöv and Lund.

The students had to write down the distance between Eslöv and Lund, based on their own understanding of it. They were not allowed to consult one another, and since this was before the era of Google, they couldn't look it up unless they went searching for a book on geography, which they were not allowed to do. They could only make a best guess, based on their own experience, and write it down individually.

My colleague collected the paper slips with their individual answers. He looked at all the answers and discarded the answers that gave the largest and the shortest distances. He looked at the other answers, added all of them to get a figure, and then divided this figure by the number of individual answers he had received.

Imagine that there had been six students who tried to estimate the distance between Lund and Eslöv. Imagine that their six individual answers had been 10, 18, 24, 28, 30 and 40 kilometers. My colleague would disregard the "10 kilometers" and "40 kilometers" answers, and instead he would add 18 + 24 + 28 + 30 and get 100. He would then divide 100 by 4, since he added up four answers, and he would see that their "average answer" was 25 kilometers. This would be fairly correct, since the true answer is 23 kilometers.

Ann