Every year, we at HERO Sports want to know how BennettRank compares to what you consider to be the gold standard; the Coaches Poll. Every year in our post season surveys we ask you to tell us which polls you consider to be the most accurate and credible. Consistently across all divisions, all sports, you tell us you look to the Coaches Poll "first". We want to do this evaluation to see if and where we can tweak our formulas to be better and to eventually become your "go to first poll".
This year we have come up with an objective measure of how each poll performs; specifically in the year end NCAA tournament. We call it the Ranking Performance Accuracy Index (RPAI). To compare apples to apples, we compare how "our", Top 25 BennettRank compares to the Top 25 Coaches Poll in predicting tournament success, or for that matter to any Top 25 ranking system including the RPI. We compare the Top 25 rankings before the first tournament game is played as to how well their teams advance in each round.
We group each of the rankings systems Top 25 into the following buckets and assign weights to each category:
| # Top 25 Teams in the Sweet 16 | 23% |
| # Top 10 Teams in the Elite 8 | 23% |
| # Top 6 Teams in the Final 4 | 23% |
| # Top 4 Teams in the Final 2 | 23% |
| # Top 2 Teams in the Final 2 | 5% |
| # Top 1 Teams as Champion | 3% |
| TOTAL | 100% |
If every Elite 8 team is in the Top 10 for a given ranking system, they would get the full 23% allocation. If they had four of their top 10 teams in the Elite 8, they would get half the allocation 4/8×23% = 11.5%.
How does this look in practice?
The following is our live spreadsheet calculation for the D1W Soccer after the tournament. This matrix shows the Sweet 16 teams, their rankings before the tournament, and how they progressed to the tournament end.
These are Week 13 rankings of the Sweet 16 teams for each poll. BennettRank missed on Northwestern and North Carolina State, as did the RPI. The Coaches Poll missed on Santa Clara and North Carolina State. The following is the calculation table with the number of teams in each bucket and the extended calculation to give the final index:
2016 D1 Women Soccer Tournament Results by Ranking System:
In this situation, the ranking systems do not differentiate themselves very much; Bennett Rank had USC, the Champion, as a higher ranked team and got the points for that choice. The numbers are not always this close. Let's look at the 2016 matrix for D3 Men's Soccer…
2016 D3 Men Soccer Tournament Results by Ranking System:
In this case the Coaches Poll didn't have a single Top 6 team make it to the Final 4. The winner, Tufts, was unranked.
We will use this index starting this year for all sports, to make sure we are improving where we can, and doing what we say we can do: provide the best predictor of who will win a match between two teams on a neutral field.
Input and comments are always appreciated. You can email me directly at [email protected].
