The subjectivity of advanced defensive metrics

[MNBrew]

Going back 15 years or more, I've long felt that defensive stats are far more elusive and subjective than offensive stats. At least on sites like this, defensive stats have gained such huge clout over time and are often regarded as having gravity and credence comparable to any concrete offensive stat that exists.

 

So I've found it both refreshing and affirming to read two articles (the second linked from the first) discussing that defensive stats are, in some ways, still as elusive and subjective as they were viewed to be two decades ago.

 

Tim Marchman's SI.com article

 

For instance, fielding percentage often seems to be viewed as absurdly basic or incomplete if not antiquated or even obsolete. But, quoth Marchman, "From 2002 through 2009, for example, the correlation between team

winning percentage and team UZR was .19 -- less than half that of

fielding percentage."

Colin Wyers' BP article

 

Around here my viewpoint probably remains part of a rather small minority. But it is nice to know not everyone well-versed in so-called advanced defensive metrics believes them to be all they're cracked up to be.

 

Alright, rip away....

[Baldkin]

For instance, fielding percentage often seems to be viewed as absurdly basic or incomplete if not antiquated or even obsolete. But, quoth Marchman, "From 2002 through 2009, for example, the correlation between team

winning percentage and team UZR was .19 -- less than half that of

fielding percentage."

Not going to finish the quote?

This is ungenerous, as UZR correlates reasonably well to runs allowed without even accounting for pitching....

So, when taking into account the defensive part of the game, hey, UZR measures what defense does (prevents runs) really well!

 

Defensive stats are what they are, Dewan himself puts them at about "60% of the truth", but they are still much, much better than simply looking at fielding percentage, and they are improving every year. Fly balls are now measured by time in the air, along with location. Ground balls now include a speed component (slow, medium, hard).

Pos just made a long, rambling, awesome blog post on Dewan's Plus/Minus.

Here IS what they do: They look at every single play made (and not made)

on video. Every one. They watch the video and plot the plays right

there on the computer screen. They then compare each players ability to

make plays. I’ve explained this before, but let’s say with nobody on

base there’s a hard ground ball hit six feet to the left of second base.

Let’s say 20% of all shortstops turn that play into outs. Well, if you

make the play, you get a +.80 (because you’re making a play that 80% of

shortstops do not make). If you don’t make that play you get a -.20

(you’re NOT making a play that 20% of shortstop make). That, in a

nutshell, is the plus/minus system. Add up the plusses and minuses, and

you should get a pretty precise number of plays that the fielder makes

against average.

If you don't think that's better and more informative than "Oh, he threw it bad or bobbled it 9 times out of 243" then I suppose I don't know what to tell you.

Ryan Braun did not make an error last year. If you think he's even close to being a decent defender, I am not sure what to tell you.

 

I haven't heard much since this article was written, but if those cameras are in place everywhere (they were supposed to be for this season), there will be a huge coup in defensive stats once the data becomes available.

[igor67]

Cherry picking a quote on correlations is not a good call. I will agree that I think the defensive stats overstepped a bit on perceived reliability, but clearly there is a substantial and quick correction going on right now. I could give a halfway decent stab at explaining the correlation gap. Bigtime sluggers are more likely to get the benefit of the doubt on error plays, and at the lower end quality bats that have low error totals are going to have a playing preference over error prone guys with range. Call it the Jose Valentin effect.

 

And subjectivity is the wrong word. It's more accurate to say as Wyers did that their are hidden biases or reliability questions.

[Thurston Fluff]

They are a lot better than they used to be. I didn't know Dewan only thought of them as 60% telling. I thought it was higher. I think some of the problem is how they are being used. WAR for instance gives too much weight to single season results even while knowing it isn't as accurate as a single season of offensive stats. another common misuse is when people cite UZR for the season as evidence of how good a defender is when, to be accurate, they should count three seasons worth of info. The problem I think needs it be addressed is to make a single season's stats more telling. If you go by the proverbial three years necessary it seems to me many players ability can change in that time so the outcome is at least partially outdated by the time enough information is compiled.

[MNBrew]

No, Baldkin, I'm not going to finish the quote. I could've quoted the whole article.

 

Cherry picking, igor67? No, because I wasn't claiming that advanced defensive metrics are a crock. My point was that there was finally someone backing up at least to some small extent statistically that the "traditional" stat of fielding percentage isn't laughably worthless as many of you make it out to be.

 

I'm entitled to my opinion just as you folks are to yours. Because I don't agree doesn't make my viewpoint inherently wrong. Stats can often be made to say whatever you want them to say, regardless of how nobly they're approached.

[And That]

I don't remember many people saying fielding percentage is "laughably worthless."

 

To me, it is somewhat useful at the extremes (especially low extremes). If someone has played 100 games at third base with a .850 fielding percentage, I'm relatively certain they presently suck at playing third base (or official scorers have a vendetta against them). OTOH, if someone has a .950 fielding percentage at third, I'm relatively certain they aren't Ryan Braun bad there, but I need a lot more information to determine anything other than that. I think it's more of a point of entry stat than anything that should usually be used to make a determination about someone.

[1992casey]

The problem with fielding percentage is that too many people use it synonymously with the term "team defense." The stat is fine at measuring what it's supposed to measure, i.e. the percentage of balls that the defender handles properly out of the balls he actually reaches… according to the official scorer.

[Ennder]

Cherry picking, igor67? No, because I wasn't claiming that advanced defensive metrics are a crock. My point was that there was finally someone backing up at least to some small extent statistically that the "traditional" stat of fielding percentage isn't laughably worthless as many of you make it out to be.

 

That just isn't true at all from the article though. Winning and defense are two different things and they barely have anything to do with each other. If you were going to look at defense you would compare it to runs allowed which is the part of the quote you left off. Comparing fielding percentage to winning percentage is pretty much useless.

[MJLiverock]

And runs allowed will be heavily influenced by the quality, or lack therof in Milwakee's case, of the pitching staff.

[igor67]

No stats cannot be accurately made to say whatever you want. If you leave out relevant details, don't include all of the data collection procedures, use biased samples, or selectively remove outliers you can get all kinds of wacky results, but it is a massive problem to resort to "you can make them say whatever you want". That's as valid as saying I can make myself a king if I proclaim it true. You then shortened my own comment in your quote to make it appear that I said Cherry picking in reference to a stat when I was criticizing your limited quotation of the source. That's twice you've misquoted to support your position in this thread. And yes I'm a bit critical about it, because when those sorts of things break down so does rational discussion.

[Funketown]

Stats can often be made to say whatever you want them to say, regardless of how nobly they're approached.

Sure they can, but if one has an understanding of statistics, they can quickly be dismissed as asinine. Here's my classic example:

 

First, regress 1930's life expectancy with cabbage consumption. You'll surely get a model where as cabbage consumption increases, life expectancy decreases. Cabbage must be killing them, right? It doesn't take a genius to see that this is an idiotic conclusion (they ate cabbage because they were poor, and they had a shorter life expectancy because they were poor, not because they ate too much cabbage!).

 

My little rant above isn't really baseball related, but it shows that yes, you can pretty much say anything with statistics. However, just because you can say anything doesn't make it plausible.

 

 

[endaround]

So team UZR explains about 1/5th of a team's winning percentage and somehow this makes UZR bad?

[MJLiverock]

I think endaround's point is pretty valid, if UZR explains about 20% of the teams winning percentage that is plausible and passes the smell test. But the idea that equally weighting defense or UZR with offense is again shown to be not the case as defense isn't 50. Even the defensive stat gure Dewan only said UZR is about 60% reliable in explaining defense. It also doesn't let the excuse that the pitching staff is OK but the defense is really the problem fly. The defense isn't helping the staff but the defense isn't why Manny Parra can't throw strikes consistently, grooves fast balls that get rocketed, or Dave Bush gives up 4 bombs in a row. The pitchers either don't have the talent to do well, or can't harness their talent and locate pitches. UZR doesn't even fully explain the defense but putting an infield of all great UZR guys out there still wouldn't make this staff even average.

[Funketown]

^

 

I agree with you that the defense is certainly not as much to blame as we make it seem, but a terrible defense can definitely make a pretty decent pitching staff look bad. The difference between the best and worst team in the league in terms of UZR is typically around 150 runs, so even if we halve that, we're looking at a half of a run per game. So poor defense can pretty easily bump a 4.50 RA to a 5.00 RA.

 

That said, when comparing to an average defense, rather than an elite one, the Brewers' defense has probably only tacked on an additional .15 - .25 runs per game. They've certainly hurt the pitching staff, but it is wrong to say that defense has made an okay staff into one that looks terrible. I think it's more accurate to say that it's made a bad staff look terrible.

[sbrylski]

You gotta be super careful with correlation studies, as Funketown pointed out.

 

I've seen regression studies that conclude stolen bases have a negative correlation to wins, meaning each time you steal a base you are less likely to win. Total nonsense. All it illustrates is that bad teams tend to use the running game more often.

 

So any correlation of UZR to wins or even runs scored isn't very meaningful, at least in terms of determining how important defense is. Maybe teams with really bad pitching staffs compensate by signing good defensive players to try and get slightly better. What might result then is likely a negative correlation between UZR and runs scored, because the terrible pitchers are still giving up more runs than average, despite a great defense. But despite the poor correlation, that defense is still helping.

[LouisEly]

And runs allowed will be heavily influenced by the quality, or lack therof in Milwakee's case, of the pitching staff.

Yep. It's tougher to get to balls when the batters are hitting screaming ropes off of your pitchers, which impacts UZR.

 

I've felt that UZR is biased against leftfielders. The reason is because a majority of hitters are right-handed, and when they pull the ball it is usually hit much harder than if they hit it to center or RF. The harder it is hit, the less time you have to get to it; the less time you have to get to it the less balls you get to, which impacts UZR. I don't think that all LFs are outstanding defenders, but I don't think they negatively impact the outcome of games as much as some think. When I saw last year that Fangraphs had a higher value on Nyjer Morgan than Ryan Braun... well, that's all I needed to know about their valuation system.

 

I too think that fielding % is underrated and range is overrated. Don't get me wrong, I think UZR has value, but the point the original poster was trying to make was that fielding % is underrated. If you have range and get to a ball but can't make a play on it, it usually doesn't have any difference in outcome than if you can't get to the ball. The SS and 2B are positioned between the LF and CF and CF and RF respectively; if they can't get to a ball it usually goes straight to the LF, CF, or RF, and the batter is held to a single - unless the ball is hit really slow and the batter is really fast.

[logan82]

How hard balls are hit is taken into account in UZR.

 

The main problem with fielding % is that it is almost completely dependent on one person to make a judgement and can actually hurt better defenders. If you have good range you can make errors that a worse/slower defender wouldn't even be able to get to. The other problem is that even very crappy defenders like Braun cna have a good fielding% because unless you drop a ball you are almost never going to get an error. So in the case of outfielders, fielding % is almost totally useless. Unless a guy has an extremely bad fielding % it doesn't give enough information to be very useful because most players fall into a small range.

 

If you have range and get to a ball but can't make a play on it, it usually doesn't have any difference in outcome than if you can't get to the ball.

 

That is correct. However fielding % would actually punish a guy for getting to a ball and not making a play on it. UZR gives some credit for range.

 

 

 

I've felt that UZR is biased against leftfielders. The reason is because a majority of hitters are right-handed, and when they pull the ball it is usually hit much harder than if they hit it to center or RF. The harder it is hit, the less time you have to get to it; the less time you have to get to it the less balls you get to, which impacts UZR. I don't think that all LFs are outstanding defenders, but I don't think they negatively impact the outcome of games as much as some think. When I saw last year that Fangraphs had a higher value on Nyjer Morgan than Ryan Braun... well, that's all I needed to know about their valuation system.

 

I don't think that is a problem with UZR. That is more of a WAR or more specifically the positional adjustment of WAR issue. For the most part you get a bonus for playing a harder position. I can't argue against it much since most LF couldn't play in CF very well. I could be mistaken but as a LF your defense is set at zero by other LF. You sont' get compared to CF or RF.

[danzig6767]

I don't remember many people saying fielding percentage is "laughably worthless."

 

To me, it is somewhat useful at the extremes (especially low extremes). If someone has played 100 games at third base with a .850 fielding percentage, I'm relatively certain they presently suck at playing third base (or official scorers have a vendetta against them). OTOH, if someone has a .950 fielding percentage at third, I'm relatively certain they aren't Ryan Braun bad there, but I need a lot more information to determine anything other than that. I think it's more of a point of entry stat than anything that should usually be used to make a determination about someone.

I agree with you that fielding percentage alone often tells a very poor story about how a player is defensively, especially when it comes to outfielders. That said, i do think if you watch a player play long enough without having on team homer goggles, you can easily see that say Ryan Braun is a poor defensive LF regardless of his low error totals. You could watch Mike Cameron and see he was a very good defensive CF. Watch Fielder and see that he struggles scooping throws in the dirt which won't be reflected in his fielding percentage, but obviously hurts the Brewers because that's a key part of the job requirements for any firstbaseman. That Corey Hart has a very inaccurate arm in RF and that he also isn't very skilled at reading hit balls off the bat like the better rightfielders can do. That Casey McGehee doesn't have much range at third regardless if his error totals aren't excessive.

 

I'm not saying the human eye will always be correct when evaluating the defensive abilities of a player, but i do think it can be useful when mixed in with stats to see if what the stats say are vastly different than what your eyes are telling you.

[HiAndTight]

I don't remember many people saying fielding percentage is "laughably worthless."

Perhaps not "laughably worthless", but I've seen it said on this board too many times to count that they ARE in fact the worst way of measuring a defenses ability.

 

So I don't know how you want to equate that to "laughably worthless", but he's not exactly pulling this out of thin air.

That said, i do think if you watch a player play long enough without having on team homer goggles, you can easily

 

I have a bit of an issue with this, especially as it applies to most on this board. I think a lot of people use the defensive metrics, and if you believe in them, you're far more inclined to convince yourself that the player you're watching has the issues that the stats say than if it were the other way around.

 

I don't believe if you watched Corey Hart play for an entire year, this year in particular, that you would come to the conclusion that he's an awful defender in RF. In fact, he's made a lot of exceptional plays in RF, and being that you seldom see an OF'ers 1st couple steps, by far the most important part of playing OF defense, all you see is the last 10-15 strides.

 

So whereas you can see from time to time with Braun, he comes in on a ball and then has to retreat, or he's frozen and has to make an awkward catch, you do not see the same from Hart. And yet, again, I believe almost to a man, everyone on here would say Hart's a terrible defender in RF. So again, I believe this is a case of your eyes adhering to the stats rather than the other way around.

 

Same thing with McGehee. You can see he doesn't have good range, but he's also made a number of highlight plays this year. His range, in particular the last month, has actually APPEARED to be actually pretty damn good.

 

The main problem with fielding % is that it is almost completely

dependent on one person to make a judgement and can actually hurt better

defenders.

Yes, but as was noted in here, they have someone watching the games and charting the results. They say in this certain situation, if a ball is hit 6 feet wide of 2nd base, 80 pct of Shortstops get to that ball. Yet it doesn't take into account(as far as I know) how hard a ground ball is hit, if it's smoked up the middle, if the SS is shaded a bit further to 3rd base, who is running, Johnny Estrada, or Carlos Gomez. So there's also a lot of subjectivity in that as well. I understand with OF'ers they are trying to become more exact in measuring how hard the balls are hit, but to the best of my knowledge they do not do the same for IF'ers as it's simply too difficult to measure how hard a ball is hit.

Cherry picking a quote on correlations is not a good call.

 

Why? He certainly cannot be expected to find every quote on the subject. It'd be a ridiculously long post. Not to mention, he's logically going to post the quotes that agree with his perception on it, and without any doubt, the other side is VERY well represented on here.

No stats cannot be accurately made to say whatever you want.

He's saying the OFTEN can, and he's absolutely, 100 pct correct. If you want to present evidence that a certain player is a good player, there are a plethora of stats, particularly in today's game, that can support such a theory, and a bevy of stats that can disprove the same.

 

In the end, I think far-far too much emphasis is placed upon defensive metrics as a whole. They're useful, but as compared to OPS+, and..well, too many to list here, they're far too fallible, and simply are not as valuable as offensive rate numbers(Though I don't think anyone's suggested they're as reliable).

 

But the bigger problem I have is that WAR is used by many as the best way to compare Player A to Player B. It's being used more and more these days in HOF voting and what not, and Defense is a large part of that equation.

 

So, simply, I think they're far less reliable than many would have you believe.

[strawbossisevil]

If even the top statisticians claim that UZR is only 60% accurate in it's assessment of player's defensive ability, then I don't see how it is a very useful stat. At this point, there doesn't seem to be a way that genuinely provides an accurate picture of a player's defensive capabilities. I imagine a whole lot could be learned about a player's defensive ability from a good quality scout, then from any current defensive measurement.

 

Currently there are just far too many aspects that UZR and other defensive measures can not account for.

 

I think HiAndTight makes a good point. Where these statistics do have an impact is the perception of fans. If a fan buys into UZR which presents an unfavorable rating of a player they follow, the "eye test" of that fan is going to be skewed toward seeking out the player's defensive failings, because that is what they expect going in.

 

I'm not saying Corey Hart is a good defender or anything like that. I just think that letting one's opinion on a player's defensive capabilities be regarded primarily on a statistic that is only 60% accurate is folly. A person's eyes can serve as a far better judge of defensive capability than that. Especially so, the more one watches baseball.

 

And when posters state that we need 3 years of data to provide an even more accurate assessment of the player's defensive capablities, there are all sorts of problems with this. For example, let's imagine that in the first two years a player was battling with an injury that slowed them down significantly, but the third year they were finally healthy and able to run more freely. UZR would not provide a very accurate projection of that player's defensive ability. When you need a period of three years to provide accuracy, the problem becomes that a player can be completely different from year to year, depending on experience, health, the players around him etc.

 

Normally, I am a big fan of the statistic crowd and I was a big defender of OPS. But these defensive statistics have more holes in them than swiss cheese, and it's going to be a long time before I think that they are at the level to where we can actually trust them to fairly represent a player's talent level.

[logan82]

Again, balls are rated on how hard they are hit. From Balkin's link.

 

Now, do John and his people make subjective choices? Of course. For

instance, John says that ground balls are rated as “soft, medium or

hard.” That’s subjective. But it’s getting less subjective all the time.

Right now, they are charting balls hit into the outfield based on

“batted ball time” — that is, how long it takes for a batted ball to get

to the outfield (six seconds, John says, is almost always an out; three

seconds is almost always a hit). Batted ball time seems like a very

promising improvement. One thing I very much appreciate about John and

his people is that they are constantly trying to improve their system.

Fans watching are not watching as closely as the people watching the games to acquire data for these metrics.

.

 

Here’s the main point for me, though: Whatever subjectivity they have

in their intensive and painstaking and methodical approach to studying

defense seems to me to be NOTHING compared to the subjectivity that you

and I get watching baseball even night after night after night. They

look at every single play. More than look, they CHART every single play.

More than chart, they CONCENTRATE on the defense for every single play.

You and I don’t do that. Scouts don’t do that. Announcers,

sportswriters, managers, general managers, nobody else does this. We may

watch every single play (probably not), but we’re watching those plays

in the larger context. We’re watching the pitcher, the hitter, the base

runner, the umpires, the fans, the game. They are not. They are watching

only defense. It’s a different thing.

And while there may be individual choices they must make, the bottom

line is that what John and company actually do is objective: Did the guy

make the play or didn’t he? Did the guy record the out or didn’t he?

That’s what they record. They don’t give guys credit for almost making a

great play, or take away credit because the guy took a bad route but

still caught the ball. No, the Dewan plus/minus numbers — and they are

not perfect, even John says that our understanding of defense is now

about 60% — are NOT based on some sort of voodoo, and they’re not based

on some guy making rash judgments. They are based solely on what plays

are turned into outs and what plays are not turned into outs.

 

Even 60% accurate if far better than even the trained eyes of scouts. Scouts are good for seeing raw talent or future potential. Not so good for an assement on what actually happened. They also do not watch every play from every game so they can't do a very good job of making comparisons across every player in the league. Defensive metrics are not perfect and are sometimes used to say how good or bad a guy is on defense. I have no problem with some skepticism of defensive metrics but this thread seems like taking things way to far in the other direction.

 

[Crash2303]

don't know if this fits in the conversation on either side, but I have found it interesting that certain player's defensive reputation can so easily change within a season or two. I don't follow defensive stats very much, so i primarily rely on second hand info from what is assessed and talked about on brewerfan.

I'm not going to go back and look for links for this argument, but i remember at the beginning of the season- Carlos Gomez was touted as a tremendous defensive outfielder, so much so that most agreed even if he did slightly better than his offensive career averages it would justify his worth as a starter. there was a lot of talk that he should actually be in the conversation as one of the games best CFer.

the last couple seasons Braun was highly regarded in LF on this board. there was a lot of talk i believe during the 2008 season that he should be mentioned in the gold glove conversation- and how lame it was that all three postions usually go to CFers.

the same was true with Hart- around the same time period that he was an above average OFer, with above average speed and above average arm that didn't make a lot of mistakes. that he was totally solid.

Now, most of what i read on this board is how Gomez sucks, in general- few heralding his defensive abilities, and that Braun and Hart are horrible OFers.

If you concur with the changes in general sentiment on this board regarding these guys- is this because we have more data to see their weaknesses, or have they really started sucking more?

[Baldkin]

the last couple seasons Braun was highly regarded in LF on this board. there was a lot of talk i believe during the 2008 season that he should be mentioned in the gold glove conversation- and how lame it was that all three postions usually go to CFers.

I don't think anyone here, aside from maybe the one guy who said "Braun should win a gold glove because he's made no errors" last season, who has watched Braun would say he's a good defensive OF. He has the tools to be a good defensive OF, but he doesn't care enough to become one.

 

And yes, I say he doesn't care enough, because he doesn't adjust his position based on the batter or the game situation 95% of the time. Watch Gomez or Cain out there, moving in our out, shading LC or RC depending on who is up, then look at Braun, still standing in the same patch of replaced turf in LF.

[logan82]

he last couple seasons Braun was highly regarded in LF on this board

 

Only by a very small group of people. The general consensus seems to be he has the tools to be a good outfielder but he stinks.

the same was true with Hart- around the same time period that he was an above average OFer, with above average speed and above average arm that didn't make a lot of mistakes. that he was totally solid.

 

Maybe three years ago. Not over the last 2 years. hart has been considered pretty poor for a while now.

 

Now, most of what i read on this board is how Gomez sucks, in general- few heralding his defensive abilities, and that Braun and Hart are horrible OFers.

 

Not sure where you get the Gomez sucks stuff from. His offense yeah but only a handful of people have been critical of his defense. The second part you are right about. Braun and Hart are below average.

 

is this because we have more data to see their weaknesses, or have they really started sucking more?

 

In Hart's case he seems to have gotten much worse since his first year and a half. Braun pretty much has always sucked in the field. We have enough data that we can be reasonably sure the data is correct on those 2 and Hart is a little below average while Braun is worse. Both seem to be regressing and not towards the mean.

[mletto157]

Agreed with the last 2 posters. I think we all thought Braun would be a really good LF because of his excellent athletic ability. Unfortunately we were all proven wrong fairly early in his first year in LF. But still we thought he could become a good LF. This just never happened. I still think he has the tools to become a decent LF, but he is nowhere near good right now.

 

Hart is pretty much in the same boat. We all knew Hart was fast, and a good athlete, so we figured he would be a good RF. He has shown a couple times that he can do it. But he's had even more time than Braun now, and still hasn't become the OF we thought he could be. He is average-to-below average at best.

 

I think there's been maybe a handful of people on here critical of Gomez's defense this year - but the rest of us still know/think he is a phenominal defensive CF. He's just so bad at getting on base that we can't have that black hole in the lineup.