Data Engines sells shiny new meters. We have no idea how to make information retrieval algorithms more accurate or better. We just know how to measure their retrieval accuracy when no ground truth for the correct relevance judgments is available. So we think of our algorithms for unsupervised inference as meters – sort of like voltmeters and ammeters. In our case, our meters measure how accurate your retrieval engines are. This number is just a statistic of the unknown ground truth – nothing more.
The interesting part comes when you think about the “nothing less” counterpart of what we just said. What can you accomplish when you are able to rank your retrievers correctly? For one, it gives you access to Bayes’ theorem to make better decisions than majority vote. And by the very same Bayes’ theorem, you get a probability estimate for your uncertainty – a particularly good way to spot the instances in your data for which your algorithms have less discriminative power. So, while we agree that ground truth can never be discovered by any automatic method – we can deduce statistics of the unknown ground truth. And in many cases, that is better than not having that measurement.