Moderators: Jenise, Robin Garr, David M. Bueker
Ian Sutton
Spanna in the works
2558
Sun Apr 09, 2006 2:10 pm
Norwich, UK
Steve Slatcher
Wine guru
1047
Sat Aug 19, 2006 11:51 am
Manchester, England
Jenise
FLDG Dishwasher
42664
Tue Mar 21, 2006 2:45 pm
The Pacific Northest Westest
Ian Sutton
Spanna in the works
2558
Sun Apr 09, 2006 2:10 pm
Norwich, UK
James Dietz
Wine guru
1236
Wed Mar 22, 2006 6:45 pm
Orange County, California
Brian Gilp wrote:Let me try a different way to explain my concern with the sample size.
For the sake of illustration lets assume that for the first 12 tastings every wine was the group favorite 2 times. Then for the next straight 7 tastings wine A was the group favorite making it the favorite in 9 of the 19 tastings held. The probability that wine A will be the favorite 7 straight times is 1 in 280,000. While improbable this is not impossible. In fact it is more probable than opening 4 straight flawed bottles of wine (1 in 330,000 assuming 1 flawed bottle for every 2 cases).
Looking at it the other way, wine F would have to be selected the favorite in 3 of the next 4 tastings to bring it to what would be expected. The probability that wine F would be selected 3 straight times is 1 in 216. Selecting it 3 out of the next 4 times makes it even more probable.
If one instead looks at the probability that wine A will not be selected as the favorite again for some time one gets the following. The probability that wine A will not be selected in the next 10 tastings is 1 in 6, in the next 16 tastings is 1 in 50, and in the next 34 tastings is 1 in 500.
With only 20 events recorded, there is still reasonable probability that the distribution will look significantly different once the next 20 events are completedand that the appearance of bias will be significantly reduced.
Ian Sutton wrote:Brian
Assume (as the simple grouping becomes equivalent to) that it's just 1 person choosing the favourite in each of 19 tastings.
Taking Steve's analysis on trust (and my gut feel is that it looks about right), he's saying that the results experienced had less than a 1% chance of happening if the chance of a wine winning was actually always 1 in 6.
Now you say that there is a one in 6 chance of wine A not being chosen in the next 10 events.
That might leave us with (say)
A:9
B:4
C:4
D:5
E:2
F:3
That still looks pretty much to favour wine A
As a bit of fun - to get the group average back to 9 each, then A must go on a run of 35 events without a win (a 0.17% chance of this occurring - i.e. very unlikely indeed).
I think this sort of scenario does often shock people, of just how powerful some results can be, when they seem not too unlikely on the face of it. I recall the company I worked for doing an annual fete, with one stall being 'roll the dice'. There were six dice and if you got 6 sixes you won a car. Seems easy enough, but as you'll be able to check, this is only a 0.002% chance. The stall was hugely popular.
... However, there is one part of this that should stop us drawing too firm conclusions - that Jenise saw the results before letting us know. She reported something seemingly out of the ordinary, but presumably wouldn't have done so if the split had been more even (or if her favourite had won (friendly joshing)).
As an (extreme) example I toss a coin 10 times on 1000 separate occasions. On the 878th occasion I get 10 heads, which individuallly has a probability of less than 0.1% of occuring. That's significant isn't it?! The coin is biased!! Well not if you see the results and then perform the test only on the one that piqued your interest. Getting one instance of 10 heads in 1000 attempts is quite unremarkable (I make it just over a 1 in 3 chance).
regards
Ian
Ian Sutton
Spanna in the works
2558
Sun Apr 09, 2006 2:10 pm
Norwich, UK
Have you been playing with those naughty Sith again Brian...Brian Gilp wrote:Alright Ian I almost have you over to my side.
Brian Gilp wrote: Now consider this. Using your extreme example of coin flipping which I agree with and is the basis for my arguement on variance impacts over small sample sizes, there is absolutely no reason why that could not happen on the first occasion as opposed to the 878th. And if it did happen on the first occasion it should take a significant amount of future occassions before the results start to resemble what would be expected for a random distribution.
Brian Gilp wrote:If as I propose, the original 20 tasting events are just an extreme example such as your 10 heads in a row example, it should not be expected that the results would normalize in only 10 more tastings and that bias would continue to exist for some time. I did not intend to suggest that the results would normalize in the next 34 tastings but that the likelihood of any wine not being choosen over many tastings is greater than one would assume and that it would not be shocking to find a period of tastings in the future where A is not the group favorite.
To repeat something I wrote before just to be clear. I am not attempting to state that there is not a bias in the results. I am stating that due to the fact that these are not controlled experiments and the small sample size that the impact of extreme random events can not be eliminated as the cause.
Jenise
FLDG Dishwasher
42664
Tue Mar 21, 2006 2:45 pm
The Pacific Northest Westest
... However, there is one part of this that should stop us drawing too firm conclusions - that Jenise saw the results before letting us know. She reported something seemingly out of the ordinary, but presumably wouldn't have done so if the split had been more even (or if her favourite had won (friendly joshing)).
Ian Sutton
Spanna in the works
2558
Sun Apr 09, 2006 2:10 pm
Norwich, UK
Jenise wrote:
Well, yes/no. Re the if my favorite had won: I realize you were joshing but I have to nonetheless offer that you're so wrong there.
Ian Sutton
Spanna in the works
2558
Sun Apr 09, 2006 2:10 pm
Norwich, UK
Jenise wrote:This sure has this been interesting.
Brian Gilp wrote:20 samples are not significant for random events. If one is to flip a coin 20 times it is not uncommon to have it come up heads 15 times.
MichaelB
Ultra geek
103
Sat Oct 18, 2008 7:32 pm
Sierra southmost, California
Mark Lipton wrote:Brian Gilp wrote:20 samples are not significant for random events. If one is to flip a coin 20 times it is not uncommon to have it come up heads 15 times.
Actually, that statement is untrue by most definitions of "uncommon." 20 coin flips will result in 15 heads or 15 tails no more than 1.5% of the time, hardly what I'd term a common occurrence.
Users browsing this forum: APNIC Bot, ClaudeBot, Google Adsense [Bot] and 1 guest