The problem with traditional performance measures

In past writings, I have used a baseball anecdote to help point out the drawbacks of traditional fire department performance measures, and I think that it bears repeating here.

The New York Yankees played the Pittsburgh Pirates in the 1960 World Series. The Series went to seven games, with the Yankees amassing 55 runs v. 27 for the Pirates. If we used aggregated data like total runs to determine the winner, it would be clear that the Yankees won the Series. They didn't.

The Pirates won the Series because they won more games (4-3). Here is the game-by-game breakdown:

                               Pittsburgh Yankees

Game One                     6            4

Game Two                     3          16

Game Three                  0          10

Game Four                    3            2

Game Five                    5            2

Game Six                     0           12

Game Seven               10             9

Total Runs                  27           55

The records show that the Yankees earned most of their runs in three blowouts in games Two, Three and Six. However, the Pirates won four of the seven games and took the championship.

The statistic of Total Runs for the Series tells us nothing about why each game turned the way it did. For that, we have to look at the outcome of each game. We can also evaluate each game to determine why the score turned out the way it did. Taking Game Two as an example, what factors allowed the Yankees to score 16 runs and hold the Pirates to three? Was the Pirates loss a factor of poor pitching, fielding or a combination of both? What did the Pirates have to change in order to win against the Yankees?

Total runs in baseball are similar to statistics like annual fire loss. They give us general information but do not tell us how responders faired at each incident. Without knowing whether a fire incident ended up as a "Win" or "Loss," we cannot figure out the most effective way to improve our performance. The role of the pitcher in baseball is similar to the role of fire prevention in public fire protection. Just as strong pitching reduces the number of hits, strong fire prevention reduces the number of fires that kill, injure and damage property.

Carrying the metaphor a little farther, manual suppression is similar to fielding. It is reactive instead of proactive and has its limits. If a batter overcomes a pitch and hits one out of the park, the best fielders in the world are helpless. Just as it takes a team of skilled individuals to win more baseball games than their rivals, it takes a team of skilled individuals to improve a fire department's record of Wins v. Losses. Fire chiefs who continue to rely upon aggregated data and ignore the outcomes of individual fire incidents cannot find the most cost effective ways to improve their department's performance.

Comments

[Steve S.]

I think this is good insight but raises a question... How does one go about "keeping score/statistics" for each "game"? What stats should be kept? Locally I'm trying to develop a plan to report $ "saves" instead of $ "losses". Over the course of a "season", we can go back and see how much money either the Operations side saved a homeowner/business or how the Prevention side (namely sprinklers right now) saved a homeowner/buisness.

[Robby Dawson]

Very good comparison. The issue is now what statistic is coparable to the opponents batting average or ERA to measure good pitching in fire prevention?

The statistics for sprinklers is easy to reach for, but either way you have to have a fire to generate the number. How do you measure how many times something didn't happen?

[Paddy]

Steve S. asked about what statistics to use for measuring successes v. losses. I suggest two, pre- v. post-flashover fires and life loss.

Dollar loss statistics are unreliable because they are as much a function of a building owner's tenacity in dealing with his insurer than they are a reflection of the actual value.

As for who had the most to do with a success, operations or prevention, an analysis of the incident will show what went right and what did not, and will point to why the fire was stopped before flashover. Did a smoke alarm contribute to early discovery? Credit codes and code enforcement.

[Paddy]

Robby asks how do we develop data similar to a pitcher's ERA?

We can do it in the future, but need to develop other measures first. ERA's are good data because each pitcher can be compared to other pitchers. We cannot compare fire departments like that right now. If we had a way to classify fire departments according to their level of risk exposure, we could rank departments as A, B, C or what have you. If two Class A fire departments have different levels of fire frequencies, we could reasonably attribute the difference to prevention.

RHAVE is one way to do that, but I am working on a simpler approach using GSI mapping.