PredictionBook [1] does it that way. You get a chart of how many times you assigned a certain probability and what percent of times

[1] http://predictionbook.com/