Agnostic learning vs. prior knowledge challenge

"When everything fails, ask for additional domain knowledge" is the current motto of machine learning. Therefore, assessing the real added value of prior/domain knowledge is a both deep and practical question. Most commercial data mining programs accept data pre-formatted as a table, each example being encoded as a fixed set of features. Is it worth spending time engineering elaborate features incorporating domain knowledge and/or designing ad hoc algorithms? Or else, can off-the-shelf programs working on simple features encoding the raw data without much domain knowledge do as well or better than skilled data analysts? To answer these questions, we organized a challenge for IJCNN 2007. The participants were allowed to compete in two tracks: The "prior knowledge" (PK) track, for which they had access to the original raw data representation and as much knowledge as possible about the data, and the "agnostic learning" (AL) track for which they were forced to use data pre-formatted as a table with dummy features. The AL vs. PK challenge Web site remains open: http://www.agnostic.inf.ethz.ch/.