How Well Calibrated is Predictit?

Since June 2020 my raspberry pi has been dutifully logging the Predictit market data every five minutes. I've run up to the 1 TB limit on the first hard drive I was using and ran up to the 64,000 file per directory limit the FAT file system imposes 4 or 5 times. Simply downloading the market data file Predictit provides was perhaps a little naive.

After almost a year I figured it was time to actually try to learn something from the data so I put it in a Postgres database and started asking some basic questions. I have data on 676 markets 567 of which are resolved and 109 still open. I have nearly 50 million rows in my prices table. I'm not quite at big data but I'm starting to have some medium data.

My question for today is: how well calibrated is predictit? In other words historically with what frequency f does a contract trading at a price p resolve to yes. In a perfectly calibrated context we would have p=f. For each resolved market in the data set I looked at its price one day, week and month before resolution and graphed the markets grouped by price against their yes frequency. You can see the results in the following three graphs. Note the green line is the frequency one needs to make a profit at a given price net of fees and bid-ask spread.

Some notes:

• I simply averaged the BB-BO to get the price. Most market spreads are 1-2 ticks [1].
• I smeared out my price data so if a contract was trading at 43 for example it added weight to 41,42,44 and 45 on the line.
• I did not smear out in time. I just took the price closest to a day, week or month before closing.
• Some markets you know when they'll close such as will Cruz by censored by 5/1. Others are closed either when the event happens or at a set expiry like Will the minimum wage be $15 this year. Being right a month before resolution means something pretty different in these two cases but I simply averaged them together in the graphs.

Some observations

• The graphs look a little crazy in the middle, especially the one day one. This is partly because there's a pretty small sample size in the middle. Almost all markets are either 'sure things' or 'long shots'. Especially a day before resolution.
• At all three timescales the markets seem fairly under confident at the upper tails. There's not a single contract that traded over 90 a month before resolution which resolved NO in my data set.
• The markets are also under confidant on the lower tails but not to the same degree. Note there is an asymmetry between yes and no because of multi-way markets where many contracts resolve NO but at most one resolves YES.

Let me know if you have any ideas for questions that would be interesting to answer with the data. I might investigate volatility next. If you're interested in receiving access to the data make a bid.

[1] Most might be an interesting thing to quantify next