Semantic Search Workshop 2010… and a bit of competition
Together with my co-organizers Haofen Wang, Thanh Tranh and Marko Grobelnik, we have been again given the fantastic opportunity to organize the next edition of our Semantic Search workshop at WWW 2010,. The papers are due soon (March 6), so it’s a bit too late to advertize this part of the event, but I definitely wanted to mention what will hopefully become another key part of the event: the Evaluation of Entity Search track, a competition we are organizing in conjunction with the workshop.
We have already noticed last year that our growing field has been lacking a firm ground to evaluate our results, so right after WWW 2009 we have to decided to put serious effort into thinking about the evaluation of Semantic Search. One of our talented interns at Yahoo Research Barcelona, Jeff Pound has spent his time last summer to formalize possible evaluation tasks in semantic search and we have a paper at WWW 2010 on this topic. Just as importantly, however, we were joined by other helpful folks (Harry Halpin, Daniel Herzig, Henry Thompson) to actually organize a public competition.
So what does this competition look like? Participants will be given queries sampled from a web search query log provided by the Yahoo Webscope program, and have to try to answer those queries using the Billion Triples Challenge corpus from 2009. The queries that are selected are all entity queries in that they are looking to find information about a single entity. (The queries may contain context information, e.g. peter mika barcelona). The results have to be URIs of resources from the dataset. In short, a competition with keyword queries over structured data (RDF), which also sets it apart from other competitions focused on entity retrieval.
Everyone interested will have until April 10 to submit results as explained on the website of the workshop. As we don’t yet know what to expect, there will be no prizes this time… other than fame and glory!