Ranking and Preferences

[Jenise]

I'm not sure where I'm heading with this post--maybe I'll work it out by the time I type the last word. But here's what happened: last Friday night was our neighborhood wine tasting. The theme was Pinot Noirs From Around the World wherein we served, blind, candidates from Oregon, California, Washington, South Africa, New Zealand and France labeled A thru F.

As usual, the group of about 55 tasters was polled for their 1st, 2nd and 3rd favorites and, as usual, every wine garnered some first place votes. The lowest ranked wine had around 25 points, and the first place wine got around 85. The first place wine was the A wine, a $15 2007 Dashwood from New Zealand, a pleasant wine for the price but truly nothing special. In second place was wine B, the '08 Au Bon Climat Santa Barbara County described by the winemaker himself as "This wine is just easy to drink...there is nothing pretentious here, just well balanced brightly fruity wine that goes with everything." In other words, it was light and simplistic. It also had that iodine note in the nose that I find often in ABC pinot and do not care for. And as so often happens, my favorites, the 05 Pali 'Shea' from Oregon whose body and complexity were head and shoulders above the rest and the 2006 Louis Latour Valmoissine ($16!!!) from Provence were in last place.

The next morning I got to wondering how a group who would usually vote the biggest, plushest, oakiest wine in first place could also, in the absence of any wine meeting that description as was the case Friday night, pick the thinnest, least interesting wines for the same honor. And as I thought back over their choices for the last few months, in my mind's eye (I have fairly photographic recall) I could see the winners--first bottle on the left. And once I processed that thought, it also seemed familiar that wines E and F are equally often in the caboose.

Let me quickly address how the wines are ordered for pouring: they're not. It's completely random. The wines are covered with black socks and little lettered ID tags are hung around their neck as quickly as possible so that Jim and I can try not to notice which wine went where and no one else can read them as we pour.

So I called my co-organizer and asked if he had voting records, and he did, or at least he had the last 20. He loaded that data onto an Excel spreadsheet: I was right! Wine A has taken first place 9 times out of 20 and last place only once. Wines E and F are the reverse: they've taken fifth or sixth 9 and 11 times, and been in 1st place only once each. Wines B, C and D are spread out more evenly, but it is notable that wines A, B and C have taken first place 85% of the time. Presuming that the majority of tasters work left to right, it's as if they bond with the first wine they taste and have their senses dulled as they work down the line.

Understandable, I guess, humans being what they are and this group being on the majority very inexperienced, but disappointing to think that we can possibly manipulate almost any wine into a first place finish simply by making it the A bottle--or that our tasters aren't able to concentrate past the third glass.

Your observation doesn't surprise me a bit. It's a problem I have when I go on a tasting binge like I did on Saturday.
I think there are two issues - one, you get sensitized to the first wine you taste, and two, you get numb over time.
I imagine when you were doing your sommelier classes, you found the same issues, but maybe they were described differently.

John

Very interesting. I assume there was wide range of experience/knowledge in this group? Here are two interesting experiments I would try: (1) put the same wine in the lineup twice. (2) try this same format with a group of experienced tasters only.

[Jenise]

Very interesting. I assume there was wide range of experience/knowledge in this group? Here are two interesting experiments I would try: (1) put the same wine in the lineup twice. (2) try this same format with a group of experienced tasters only.

Yes a wide range with me the most serious geek in the room, but the majority just come for the social outing. I would guess there are only five people in the room who could name even three of the wines come Saturday morning or less than ten who would ever spend more than $20 on a bottle of wine.

Jenise
A suggestion.
Next tasting, assign half the group to go A to F and the other half to go F to A. That might reduce such bias, but also still be manageable.

Alternatively, see if you can keep it as it is, but keep the records from a handful of tastings and we can get our long-forgotten stats expertise onto it (I know of one other forumite who did stats at Uni - indeed it turns out, the same Uni/course as me!)

FWIW, I'm slightly surprised. I can imagine wine A being memorable, but would also expect wines E and F to also stick in the memory. I'd expect wines C and D to perform worst (purely speculative opinion from me).

Finally, for Pinot, I suspect ordinary people often want the least challenging wines, with soft fruit, no undergrowthy elements and not too savoury. Might be fun to put a good warm year Bojo into such a tasting!

regards

Ian

Interesting. At our local AWS tastings, I've noticed almost the opposite, but the wines are poured by servers, not placed out on a table. It seems the first wine usually ends up with one of the lowest scores. I've attributed this to folks not wanting to give the first wine a score so high that if something better comes along they can't exceed it. Perhaps the best tasting I ever took part in was at MOCOOL a few years ago. The Noland Brothers hosted a Chateau Musar vertical tasting in their hotel room and each wine had it's own glass, so one could compare all the various wines however you wanted to. Regarding Jenise's observation, one way to overcome the problem would be to assign each bottle a different, non-sequential symbol, like a red circle, blue star, green diamond, etc. Then, arrange the bottles in a circle on the table. Each taster starts with a different assigned symbol to taste and 1/3 of the people taste in a clockwise direction, 1/3 in counterclockwise and 1/3 random.

[Jenise]

Howie, it seems I left out some details as you've made some incorrect assumptions. ALL the wines are poured by servers, all six are poured at the same time. It's up to the taster to choose what order to taste in, but most go left to right, ergo A is the first they'll taste. And most taste as they go--only a few of us smell before we taste any.

Whenever I do non-blind tastings, I try to serve wines from right to left in order of growing complexity/age (like most of us, presumably). I'd hate to think there is some structural bias working against that criterion, even if it only applies to newbies.

Very interesting.

A simple idea- have people draw from hat with slips A-F, ask them to start at that table and work around. . Would stagger order,and eliminate crowds around pourer A at beginning of evening.

How fast do people taste? If tasting is spread over several hours, oxidation on wines E & F might be a possible effect

[Jenise]

Would stagger order,and eliminate crowds around pourer A at beginning of evening.

How fast do people taste? If tasting is spread over several hours, oxidation on wines E & F might be a possible effect

Ah, more confusion by my lack of complete detail. This is a sit-down tasting. People bring six glasses and are poured in place. They have about an hour to spend with the wines before casting ballots. No oxidation in effect, here, except where a given wine closes down or drops off quickly.

Ah.
Well, only idea in that case is to change to a non-sequential form of identification(blue, silver, red, purple, green, and orange, or zoo animals, or perhaps street names if this is a neighborhood thing- of course then you get people liking their street more or picking their favorite color) and putting different order at different tables.

Still, while I'm not surprised order can make a difference, I'm shocked at the extreme bias shown by your results

Ah, more confusion by my lack of complete detail. This is a sit-down tasting. People bring six glasses and are poured in place. They have about an hour to spend with the wines before casting ballots. No oxidation in effect, here, except where a given wine closes down or drops off quickly.

This makes no sense to me. I have seen the order effect when I worked a winery tasting room and he poured one at a time. First and Last seemed to be the favorite and I attributed the last to being the only wine with some RS. However, when all wines are available at the same time and people can go back and forth between wines, I see no reason why order should matter. The results you cite are really odd but then again your sample size is really small. While the results seem to point to some impact of order its hard to make any determination on only 20 events.

[Jenise]

Ah, more confusion by my lack of complete detail. This is a sit-down tasting. People bring six glasses and are poured in place. They have about an hour to spend with the wines before casting ballots. No oxidation in effect, here, except where a given wine closes down or drops off quickly.

This makes no sense to me. I have seen the order effect when I worked a winery tasting room and he poured one at a time. First and Last seemed to be the favorite and I attributed the last to being the only wine with some RS. However, when all wines are available at the same time and people can go back and forth between wines, I see no reason why order should matter. The results you cite are really odd but then again your sample size is really small. While the results seem to point to some impact of order its hard to make any determination on only 20 events.

Well, 20 events isn't a lot, you're right. And it wouldn't prove that these results would apply to any group of 60 people except the 60 people in my neighborhood (of which 80%-90% are the same people every time) with their particular mix of experience levels. As to how the order matters, all I can suggest is that apparently the first wine somehow creates a benchmark and the remaining wines are judged as deviations from that--few people who are only casual/social wine drinkers understand why they like what they like or have a reliable palate memory. Perhaps the fact that 75% of the time wines D E and F are in the bottom three even suggests that after the first three, tasters have difficulty telling the wines apart. All I do know is that if I were trying to sell this group a wine, I could pad my chances by getting it into the A slot.

Actually a sample of 20 is a valid threshold for statistically significant results. Not as good as 100 though . . . .

Indeed 20 events isn't a bad sample size at all. It would be important to be sure the assignation of letters was always random and if people consistently went A to F, then even better. Sure there are other variables and possible interactions between wines (e.g. wine 2 might show badly coming straight after wine 1), but there don't seem to be any obvious non-random variables - apart from the order which we'd be investigating.

It would be better to run the test going forwards - there is an issue of devising tests to support the suspicion of what the test data suggests!

I think the answer(s) to this question would be interesting - whether the order of tasting is a significant factor.

regards

Ian

Our once-a-month wine-and-dinner tasting group has been in opertion for twelve years or so and we always pour six wines, blind. Ours is a relatively small group (four couples). I've never noticed any correlation at all between order seved and order ranked. What I have noticed is a correlation between outlier wines and order ranked. For instance, if we were to serve six Pinots and five are from Oregon and one from Burgundy, the Burgundy would have a better chance of being rated first or second. Outlier favoritism is even more likely to be operative if five are Burgundies and one is from Orgeon as some members of the group tend to favor fruit forward wines but then that's another variable.

Anecdotally, it appears to me that the wine that wins a blind tasting is usually one with higher RS. Robert Cunningham has made the same observation.

For what it's worth, when I visited Domaine Fourrier in January, Vicki Fourrier told me that they used to show the Clos Vougeot "Petits" first (because it is their southernmost holding and they like to show the wines south-to-north) but that scores from critics have gone up since they started showing it sixth in the lineup.

Anecdotally, it appears to me that the wine that wins a blind tasting is usually one with higher RS. Robert Cunningham has made the same observation.

I think that would depend very heavily on the group, and would preferentially be true for less experienced tasters. Certainly the wine with highest RS will stick out, but it's not clear that that should bias an experienced taster.

20 samples are not significant for random events. If one is to flip a coin 20 times it is not uncommon to have it come up heads 15 times. Flip a coin 20,000,000 times and the results will be 50/50 unless you have a coin that is biased. The same type of randomness can occur on a roulette wheel, where it often happens that the same number will hit 2 or 3 times within a 20 spin window even though there are 38 different numbers on an American wheel. Along those lines, if instead of wines, you were looking at a roulette wheel with only six numbers, the results reported would be odd but I don’t believe that they would be considered anything more than short term variance.

With the data that exists, I don’t believe it is possible to separate the impact of the random assignment of wine designation from any real bias towards wine A.

Brian, Ryan
20 samples can yield a significant result, but it depends on the specific test and indeed the results.

It's also important to note that this is not 20 individuals performing a ranking of the wines, but 20 events of ~ 50 tasters per session. It's not quite 1000 individual samples, as there are only 20 variants on what wines are shown (this would add complication to the analysis). Nevertheless we are talking about a pretty large sample size (and whilst I may not be able to remember how to analyse such a set up, it is familiar enough to say a working statistician or 3rd year Uni student would find it easy to analyse)

For a formal study, one might seek to assign different order of tasting to different tasters, so that within a tasting, one would have some tasting wine A last, others 5th, 4th and so on. This eliminates some random variation from the wines themselves, but doesn't address the random variation of the tasters. Not feasible here, but I recall multivariate regression would not just look at the order, but also whether an individual tended to go for big reds (say), and indeed cross-variable analysis of whether lovers of big reds didn't like them 1st up!

Random variation is not in itself a big problem (and very much assumed in performing analysis), though the greater the random variation, the less significance you're typically able to extract from the results. Non-random variation is the big enemy here, e.g. if the host always puts the lighter or cheaper wines 1st.

Sorry - all a bit technical (and 20+ years out of practice ), but in summary, yes a test on this scale has every chance of testing the theory.

Jenise
If you do wish to follow this up, contact the stats department of a nearby University. It would make a great practical exercise for the students, to either help devise an efficient method for producing data, or simply to take the results you supply (plus background on the tastings) to challenge them to say how much credence they give to the theory that tasting order affects which wine people prefer.

FWIW at college we did 9 separate stats experiments in 3rd year including
- mustard seeds germinating/growing on soil, cotton wool, etc.
- Whether black people are more likely (than white people) to be convicted of rape (A great example of how too many other factors can make any meaningful analysis impossible).

regards

Ian

Ian I think we are largely in agreement. 20 samples can yield a significant result but I don't think that applies here. I agree that If individual taster data is availabe that may change the ability to draw conclusions. I also agree that future tastings could be modified to try to determine if a bias exists which would also help.

Maybe I did not state it clearly but one of the major assumptions that has to be made for what I wrote previously about the impact of randomness is that the wines would rank in the same order of group preference if presented in a different order. I understand that this is the assumption that is being challenged, however, working from that assumption it becomes an issue of randomness of how the wines were assigned a designation. As it is the group result that is of interest I content that you still only have 20 samples and that it is insufficient to determine if the assumption is valid or not. The group favorite wine being designated as wine A 9 times out of 20 is easily explained by small sample size variance.

To be clear, I am not saying that the results are not caused by some other factor. I am stating that it is impossible at this point to attribute it to any individual factor.

This all has got me thinking.. did the order in which the wines were served affect the results at the famous (or is it infamous?) 1976 Judgment of Paris? Googling hasn't helped me find the order in which the wines were served, so I may have to wait til I get home to check to see if Taber's book has the list so we can begin to validate whether there is a `Stone(d)' effect at work when tasting.

[Jenise]

The group favorite wine being designated as wine A 9 times out of 20 is easily explained by small sample size variance.

Not only was wine A the favorite 9 times out of 20, wine A finished in the top three 17 times out of 20. So I looked back through the records I have at two of the only three times (I was missing one) a wine positioned D, E or F finished in first place. In each case it was a noticeably sweeter, plusher wine very different from the rest. (My own preferences, as you'd guess, are typically diametrically opposite that of the group.)

Dodgy stats deleted for the time being!