Breaking news- one needs to be skeptical of winefair medals

I read this study somewhere else and got in touch with the author about it. He said the duplicates/triplicates were typically presented within a single flight of 30 wines; judges were not told about them; and as far as he knew, not a single judge remarked that a wine tasted like one from before.

The overall results certainly are a blow to the validity of competition results, yet I think two things need to be considered. First, agreement across judges was found to be somewhat better for wines that were disliked. Second, those of us who enter competitions (self disclosure: I don't win much) know that certain people consistently win medals at a rate far greater than chance. You might say there are many possible explanations for this besides the quality of the wine and judges' ability to detect it. Yet, the result stands by itself. There is consistency. Where does it come from?

Dan, Hello.....


With regard to some wineries winning a disproportionate number of medals, the skullduggery is not on the part of the judges, at least in quasi-legitimate competitions where at least the tasting is truly blind. If there is skullduggery that comes in under two conditions:

(a) Those wineries that enter enormous numbers of wines in an equally enormous number of categories - enough so that even the laws of probability say that they're going to win more than their share. I know, for example, of one winery that entered no less than 106 wines in a single competition. Think also of a competition held say in Panama at which 70% of the wines entered are from Central America. Good chance that one of those Central American wineries will come up with the prize for the "best red wine in the world" from that competition.

(b) Even darker skullduggery (what a great word!) comes about in cases where the results go to "the back room" and there are no supervising accountants (CPA's) or attorneys to supervise the calculating of the final scores and results.

Situation (a) is of course one of the reasons that I am opposed to certain competition. Situation ( b) is even more aggravating because at least in my book that is out and out fraud.

Best
Rogov

With regard to some wineries winning a disproportionate number of medals

I confused matters by not distinguishing commercial wine competitions from those for home winemakers. The Hodgson research had to do with commercial wines, but the consistency I referred to was in home winemaking competitions. There's room for skullduggery in home winemaking competitions too, but the stakes (i.e., money) aren't as high so the likelihood is arguably less.

The question is, should one be any more sceptical of medals than of wine critics?

Perhaps scepticism in general is on of the hallmarks of intelligence.

Best regards,
Alex R.

Hi Dan,
I don’t know if this would make you feel any better, but you enter some of the “toughest” amateur wine competitions. NYS Home Wine Competition (I have been entering this competition for the last 15 years, and even though its only open to amateur winemakers who live in NYS, our state has some of the “greatest” amateur winemakers!) Over the years you get to recognize some of the names (especially those who receive recognition for their effort and hard work) and those that have become professional winemakers. The same goes for the National AWS Wine Competition (I know the rules are changing, but this competition was at one time only open to AWS amateur winemaker. Over the years I have observed that there is something “special” about AWS amateur winemakers, we’re all very passionate about our hobby (the distinction between amateur and pro is not so cut and dry), and again there is a large number of AWS amateur winemakers who over the years turned “pro”. I always thought of the NYS Home Wine Competition (which usually takes place at the NYS Fair at the end of August) as a prelude to the AWS National Amateur Competition (which usually took place in the beginning of November).
I view amateur wine competitions not really as a competition, in this sense hoping that the “judges” will evaluate my wines solely on their own merits. I look at in as a way to “test” my wines against so-called more “sophisticated” palates. Getting my wines into the medal round alone (in the preliminary judging of the NYS Home Wine Competition of years past) I thought was a “great” accomplishment. Over the years I value the “feedback” (both negative or positive) more so than the medal or recognition itself. The “feedback” over the years I believe has made me a more consistent better winemaker.

Salute

Victor,

For a few years, the same people that ran the AWS amateur competition ran the NYS Fair commercial and amateur competitions. That is no longer the case.

One of the AWS problems--in the old days--was that some of the amateurs that were also professors in things like microbiology had access to college labs that other amateurs could only day dream over. I considered that to be an unfair advantage.

Incidentally: I'm one of the AWS amateurs who went pro...while I submitted commercial wines to competitions, I never submitted my amateur wines, because of that seemingly unfair advantage of some.

The question is, should one be any more sceptical of medals than of wine critics? Perhaps scepticism in general is on of the hallmarks of intelligence.

Perhaps more appropriate than "sceptical" might be the word "critical". That is to say, juding and evaluating the medals as well as the critics..... I would suggest becoming sceptical only when the medals or the critics in question have earned that dishonor.


Best
Rogov

I don’t know if this would make you feel any better...

Victor, Hi...

What you say makes me feel better only about the types of competitions to which you refer....that is to say judigng of amateur wines. In that I agree with all that you have said.

Three points though

1. If an amateur winemaker invites me to his/her home to taste one or more wines, I will almost invariably slap him or her on the shoulder and say "bravo". And that not because I am being merely flattering but because I am there sharing someone's beloved effort, taking joy from his or her pride in the accomplishment, enjoying good talk and good company and probably fine food as well. I am not there in my role as "critic". As the Freudian will not analyze you when he is in your home neither will the critic judge your wine. If asked for honest feedback I would be indeed honest but the level of that honesty would depend much on how I "read" the person with whom I am sitting and talking.

2. Let's say that the amateur that I visited decides that his/her wines are good enough to put on the market and is now offering them at anywhere from $7.00 - $100+ per bottle, I must put on another hat altogether and then, tasting the wine blind to determine its true qualities and then writing about that wine for good, bad or worse.

3. I have no knowledge whatever of the qualities or abilities of judges who sit on the panels of amateur wine-making competitions; whether those judges are tested in order to establish a base-line among the panels; and how they data from these panels is evaluated. I do believe that one can learn a good deal from such competitions.

Best
Rogov

Let me return to the matter of consistency in amateur competitions. Although my "data" are only anecdotal I'm confident of them. Five or six or more particular winemakers win lots of medals year after year at the NY State Fair, and it's not because they flood the competition with entries. If (strong hypothesis) agreement across judges were indeed random this would be impossible. That is, the judges must be responding to something that sets these wines apart, even if it's access to a good chem lab.

Perhaps this isn't really all that discrepant from Hodgson's findings though. Judges might be in reasonable agreement about the very best and the very worst (Hodgson actually found some support for the latter); it's in between where agreement might be poor. If the categories of very best and very worst each accounted for only a very small percent of the entries (someone with statistical expertise might be able to estimate how wide these categories actually could be), I could imagine the overall result being little consistency across judges.

Some time ago Tom Stevenson compared Champagne performance at the 2 major UK competitions. Here's the article:
http://www.wine-pages.com/guests/tom/IW ... canter.htm

I couldn't resist analysing the data more myself, and here's what I found:
http://www.stirbitch.com/temp/wine_comps.pdf
"Trophy" = 5, down to "No Award" = 0. I've added or subtracted small amounts so you can see all the points.

What a wierd pattern! There is of course a huge cluster of almost 200 champagnes that got no awards in either competition, which I did not plot. Then there is this strange area of "Commended" to "Silver", where few Champagnes get placed, and if one competition does award in this area the other one gives gold. Then a little cluster of agreement on what a gold medal Champagne is. Hardly the sort of thing you would expect.

I think the most remarkable aspect of agreement is in which Champagnes should get an award of any type.