A Randomized Experiment Assessing the Accuracy of Microsoft's "Bing It On" Challenge

Abstract: In advertisements associated with its “Bing It On” campaign, Microsoft claimed that “people preferred Bing web search results nearly 2:1 over Google in blind comparison tests.” We tested Microsoft’s claims by way of a randomized experiment involving U.S.-based Amazon’s Mechanical Turk (“MTurk”) subjects and conducted on Microsoft’s own www.bingiton.com website. We found that (i) a statisticallysignificant majority of participants preferred Google search results to Bing search results (53% to 41%); and (ii) participants were significantly less likely to prefer Bing results when randomly assigned to use popular search terms or self-selected