these scores are not comparable because they are missing a critical piece of the puzzle: the likelihoods of your predictions.
I predict everyday for the rest of the year that the dow jones will be above 2 and below 4 million.
The right measure is to take the difference between market and pundit opinion (you know like a real market, except unlike companies these always end at 1/0 on a certain date - like a covered option) and average that.
This is definitely true. Plus, I think these are open to abuse. If the submissions are coming from a self-selected user base, surely its possible to self-servingly interpret certain statements as "predictions" that aren't, or a certain set of events as "proving" a prediction in one way or the other.
There is only so much fudging that can occur, of course. But this kind of on-the-margins behavior is a new challenge that this relatively new breed of sites will have to deal with.
The right measure is to take the difference between market and pundit opinion (you know like a real market, except unlike companies these always end at 1/0 on a certain date - like a covered option) and average that.
This is their real performance.