Home / College Football Live / How did FPI fare in picking 2017 college football results?
How did the Football Power Index fare in predicting outcomes for the 2017 college football season? Though the model had its share of misses, FPI's error rate held its own among other public models.

How did FPI fare in picking 2017 college football results?

Jan 19, 2018

  • Seth WalderESPN Analytics

I’m going to let you in on a secret of the probabilistic prediction biz: We get to have it both ways.

Let’s say FPI believes a team has a 72 percent chance to win a game, much like it did for Auburn against UCF in the Chick-Fil-A Peach Bowl. If Auburn had won, no one would have much doubted FPI’s prediction powers: It said the Tigers probably would win, and they did. But when the Knights won, we were able to just shrug our shoulders and say, “Hey, things that are supposed to happen 28 percent of the time do sometimes happen. Twenty-eight percent of the time, in fact.”

See? We win no matter what.

OK, that’s not actually how it works. In reality, FPI wouldn’t take credit — or blame — for any single game’s outcome. That just isn’t (close to) enough of a sample to know if it’s actually doing its job.

But at the same time, ESPN’s analytics team feels strongly that if we’re going to make predictions, we sure better evaluate how those predictions actually fare. And so we do that — with a big assist from the folks at ThePredictionTracker.com.

We want to judge the college FPI model on how it fared predicting all college football games this season. And the way we want to measure that success is by considering its “error” — the difference between the predicted point margin and the actual resulting point margin. We mainly are concerned with two different scores: mean absolute error, and mean squared error (which punishes bigger misses more heavily).

And while FPI doesn’t want to brag, when we look at those prediction metrics, we see that it had a fantastic year in 2017. FPI’s average absolute error was 12.4, so the average margin of victory was 12.4 points different than our predicted margin of error. That might sound like a lot (a lower error is better), but when predicting 780 football games, that’s actually very good. And it’s all relative to other models. In both error categories, FPI had a lower (better) score than any other public model out there. Here, see for yourself!

In addition, in average squared error FPI also beat the Vegas opening line. Over the long haul, the Vegas lines are essentially a theoretical limit for a public model. If a public model were consistently beating Vegas, it would be used by bettors or sports books and the line would shift toward the model.

Here’s another way to verify that FPI is doing its job: calibration.

When FPI says a team has a 72 percent chance to win, like in the example above, we need to make sure that’s about how often that team actually is winning. If teams given a 72 percent chance to win are winning 95 percent of the time, then we have a problem.

If we group together teams by similar predicted win percentages, we can see how they fared as a set compared to their prediction.

All of prediction groupings fell in their expected range except for the 61- to 70-percent set.

Keep in mind, this uses arbitrary buckets and significantly reduces the sample of the season. Therefore, it is less important than the aforementioned error analysis above but is useful for a quick peek. And if we look at multiple years of calibration, the results end up much closer to the middle of those buckets.

Looking for a more concrete prediction from FPI? While it really has no mathematical bearing on the model’s accuracy, you can revisit our season win total predictions for five teams the model felt strongly about (relative to Vegas) in the preseason. Spoiler alert: All FPI needed to go 5-0 was for Syracuse to win a single game after beating Clemson. Alas…

About proncaa

Check Also

Tennessee honors late ‘Voice of the Vols’ Ward

On Wednesday night, generations of Tennessee fans and former athletes paid their last respects to radio announcer John Ward, who died on June 20 at the age of 88 after battling a lengthy illness.

Ex-AD: Baylor used black players as scapegoats

Baylor regents conspired to focus the school's decades-long sexual assault scandal on black football players and worked to put all the blame on athletics, former AD Ian McCaw said in a deposition taken in an ongoing Title IX lawsuit against the school.

College football’s surprising new cradle of coaches

Mount Union in Alliance, Ohio, turned out the current head coaches at Iowa State and Toledo, along with prominent assistants at Ohio State and Michigan, who all emerged from an intense setting in which about 10 percent of undergrads play football.

Ex-PSU president Spanier loses criminal appeal

An appeals court is upholding a misdemeanor child endangerment conviction against former Penn State president Graham Spanier over his handling of a 2001 complaint about Jerry Sandusky showering with a boy in the football team locker room.

Maryland introduces AD Evans, mourns McNair

University of Maryland officials struck a balance between celebratory and somber on Tuesday as they officially introduced Damon Evans as their new athletic director while also still mourning the death of 19-year-old football player Jordan McNair.