Paul Lujan <plujan@qqyahooqq.com> wrote: : Randy Heaton <randyheaton@hotmail.com> wrote:
[Warning to a.s.b.sf-g readers: no Giants content here, just discussion about predicting the standings. Adding r.s.b; the thread is about projecting standings]
:> In 2003, PECOTA's average miss was 7 wins +/-. :> The biggest misses were 16: Atlanta (PECOTA 85, actual 100), NYM(PECOTA 82, :> actual 66), SF(PECOTA 84, actual 100), KC (PECOTA 67, actual 83) :> :> I continue to think PECOTA is not substantially better than saying next year :> will be just like this year.
: You're right, PECOTA's margin of error is larger than my naive guess, but : it's still better than just relying on last year's results. If you did : *that* for '03, you'd end up being 22 wins off on the Angels, and 21 wins : off on the Cubs and Royals.
The standard error from plugging in last year in the NL 02->03 was 7.3; in the AL it was also 7.3, if I did the math right. So if PECOTA was 7, it isn't much better than plugging in last year.
OTOH, that's one year...let's see. The AL 03->04 was 12.5. The AL 01->02 was 8.8. The previous year was 9.4. Looks like 02->03 was unusually stable. OTOH, even if there's a two game/team advantage to using PECOTA, some of that you could get just from using last year's RS/RA instead of W/L. Do we know how PECOTA did last year?
The real question, I think, is whether PECOTA could have spotted things like the one year surges of the Angels in 02 or the Mariners in 01. Those are the years that are raising the error rate...I have no idea of whether PECOTA would catch them, but my guess is only a bit.
What BP is really doing is a combination of two things. First, they're predicting playing time based on the new season's rosters. Most of the time last year's playing time is going to capture the bulk of that; OTOH, for an extreme case like this year's DBacks, I'm sure there's a gain by doing this kind of thing. Overall, it seems highly likely that using this year's roster rather than last year's is going to yield a better projection.
The second thing they're doing is predicting how individual players will do. For that, I suspect the potential gains over using last year's stats are very, very small at the team level. The only time it's really going to matter is if everyone on the team figures to do better or worse, either because it's a very old or very young team, or because lots of players have fluke good or bad years. Since most teams are mixtures of all that, it'll tend to cancel out.
(The only other things they're doing that is helpful is projecting realistic results from minor leaguers, which you couldn't do without some sort of translation system, and converting stat lines into runs scored & runs allowed, which is just a question of math, but which virtually none of us would do on our own).
IOW: if it turns out that the standard error year-to-year is around 10 games/team, you could reduce that a lot by using RS/RA instead of W/L, and also reduce it by assuming regression to the mean. I'm fairly certain that doing a player-by-player projection with playing time projections would get you a bit closer still, with gains increasing as roster turnover increases. I doubt that individual performance projections help much at all.
In fact, odds are that if someone took the trouble to go back and plug in *actual* rate stats into the projections for a past season, it *still* wouldn't help that much. That is, take the before-the-season playing time projections, go back after the season, and plug in how the players actually did, converting it into RS and RA, and you're going to miss by about as much as you would if you use sophisticate PECOTA projections or dumb "everyone does the same" projections. My guess is that we project standings wrong mainly because our playing time projections are wrong, and secondarily because of random effects involved in converting raw stat lines into W/L.
I'd love to see someone do a study on that.
The second thing that I'd love to see someone do a study on is whether some GMs or some managers consistently outperform a PECOTA-like projection.
Jamal Bernhard 30 March 2005 07:23:06 [ permanent link ]
jhb@socrates.Berkeley.EDU wrote:
The standard error from plugging in last year in the NL 02->03 was 7.3; in > the AL it was also 7.3, if I did the math right. So if PECOTA was 7, it > isn't much better than plugging in last year.
<snip>
The real question, I think, is whether PECOTA could have spotted things > like the one year surges of the Angels in 02 or the Mariners in 01. Those > are the years that are raising the error rate...I have no idea of whether > PECOTA would catch them, but my guess is only a bit.
Looking at the standard deviation and not just the average error might say something about whether PECOTA is picking up anything on these teams that have a dramatic change from one year to the next.
The second thing that I'd love to see someone do a study on is whether > some GMs or some managers consistently outperform a PECOTA-like > projection.
Well *that* should be pretty easy to test out, at least. I would take a wild guess and say that the Giants have probably outperformed PECOTA in several recent years.
