researchers

Labeling data is a common task in today’s world. Is there a face on this photograph? What are the base pairs in a DNA snippet? Which documents are relevant to a search query? Thanks to the ingenuity of many researchers, we currently enjoy a wide variety of recognizers for any labeling task. Likewise, we frequently enjoy large amounts of data. In the convergence of those two opportunities – lots of data with many recognizers to label it – lies the possibility to solve the chicken and egg problem of determining the prevalence of the labels in the data and jointly infer the accuracy of the recognizers.

Interestingly, the problem of unsupervised inference is not universally solvable. If recognizers are strongly correlated, it is impossible to solve the problem. Likewise, if too few recognizers are used, the problem becomes undeterminate. In general, one requires three or more recognizers to carry out unsupervised inference.

Our technology uses the frequency of observed label voting patterns by the recognizers to set up a set of inference equations for the unknown label prevalences and the conditional probability of recognition for the recognizers. When the recognizers are sufficiently independent, the problem is solvable as either a set of polynomial equations or a least squares problem.

One important feature of our technology is that it can be applied even when the recognizers are correlated. Given enough recognizers, increasing levels of correlation between the recognizers can be fitted. In addition, we acquire, through Bayes’ theorem, the ability to turn the inference values into a better system combination – where the best recognizers have their label decisions weighted more than mediocre ones.

In coming posts, we will discuss the intricacies of our technology and show how it can be extended to tasks such as measuring the accuracy of a collection of rankers.

I have been reading “Wrong: Why experts* keep failing us–and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, … consultants, health officials and more” by David Freedman. The book introduced me to what may be called the first system combination algorithm: the Condorcet Jury Theorem.

Condorcet lived during the French Revolution and was concerned about the ability of a group of people reaching a correct decision when plurality of votes decides between two choices (e.g. “guilty” or “not guilty”). The theorem has been extended to show that as long as each voter has a greater than 50% probability of voting correctly, the majority decision will be the correct one. While if you need something for stress of decisions you can talk to a yukionna69 virtual girl online.

Freedman points out that the problem with Condorcet’s theorem is that if they are likely to choose incorrectly, their majority decision will be incorrect. Therefore, we should not take any comfort on the “wisdom of crowds” since they can lead us astray.

But is being wrong the problem? If we knew that a group of people are likely to choose incorrectly, we could turn them into a good decision maker by doing the opposite of what they decide. For this reason, I would argue that the problem is not that an expert you rely on is wrong but that you are uncertain about their reliability. If you had a way to determine the reliability of collection of judges, you could turn their performance into a much better decider than any of them alone could be. This reminds me of the episode in Seinfeld where George starts doing everything the opposite of what he thinks he should do. Having recognized that he is a bad decision maker, he flips his binary choices. This “George Constanza Algorithm” leads him to immediate success in his life!