Excellence in Fire Protection

The problem with traditional performance measures

In past writings, I have used a baseball anecdote to help point out the drawbacks of traditional fire department performance measures, and I think that it bears repeating here.

The New York Yankees played the Pittsburgh Pirates in the 1960 World Series. The Series went to seven games, with the Yankees amassing 55 runs v. 27 for the Pirates. If we used aggregated data like total runs to determine the winner, it would be clear that the Yankees won the Series. They didn't.

The Pirates won the Series because they won more games (4-3). Here is the game-by-game breakdown:

                               Pittsburgh Yankees

Game One                     6            4

Game Two                     3          16

Game Three                  0          10

Game Four                    3            2

Game Five                    5            2

Game Six                     0           12

Game Seven               10             9

Total Runs                  27           55

The records show that the Yankees earned most of their runs in three blowouts in games Two, Three and Six. However, the Pirates won four of the seven games and took the championship.

The statistic of Total Runs for the Series tells us nothing about why each game turned the way it did. For that, we have to look at the outcome of each game. We can also evaluate each game to determine why the score turned out the way it did. Taking Game Two as an example, what factors allowed the Yankees to score 16 runs and hold the Pirates to three? Was the Pirates loss a factor of poor pitching, fielding or a combination of both? What did the Pirates have to change in order to win against the Yankees?

Total runs in baseball are similar to statistics like annual fire loss. They give us general information but do not tell us how responders faired at each incident. Without knowing whether a fire incident ended up as a "Win" or "Loss," we cannot figure out the most effective way to improve our performance. The role of the pitcher in baseball is similar to the role of fire prevention in public fire protection. Just as strong pitching reduces the number of hits, strong fire prevention reduces the number of fires that kill, injure and damage property.

Carrying the metaphor a little farther, manual suppression is similar to fielding. It is reactive instead of proactive and has its limits. If a batter overcomes a pitch and hits one out of the park, the best fielders in the world are helpless. Just as it takes a team of skilled individuals to win more baseball games than their rivals, it takes a team of skilled individuals to improve a fire department's record of Wins v. Losses. Fire chiefs who continue to rely upon aggregated data and ignore the outcomes of individual fire incidents cannot find the most cost effective ways to improve their department's performance.

How fire risk and mitigation are related

Fire chiefs who want to achieve excellence must understand that public fire protection consists of much more than firefighters, apparatus and water supplies. They must also appreciate the value of the other elements of fire protection that mitigate fire risk.

I describe fire risk as the product of three probabilities. The equation for fire risk,

R, is:

R = (P₁) (P₂) (P₃)

 where:

   P1 = the probability of a fire starting

   P2 = the probability of a fire spreading

   P3 = the probability of a fire doing harm to property and people

The three elements of overall fire risk have three primary mitigation methods. Public education is the primary means of reducing P₁. Code enforcement also helps reduce P₁ but is not the primary means, and manual suppression plays no role at all.

Code enforcement, which includes effective building and fire codes, requirements for built-in suppression, knowledgeable plan reviewers and inspectors, is the primary means of reducing P₂. Public education also plays a role, followed by manual suppression. Suppression is the primary means of reducing P₃, but code enforcement and public education also play roles.

The fire risk equation and the related mitigation methods make it clear that fire chiefs who rely upon manual suppression cannot effectively control their fire risks. Fire chiefs who invest in quality public education and code enforcement can.

Excellent compared to what?

In my post titled "What is excellence in fire protection?" I mentioned the need for valid measures to gauge fire department performance. I suggested two measures in a later post.

At present, very few fire departments are evaluating the results of their public fire protection systems. However, those few show that some fire chiefs understand the need to catch up to other public services on this issue.

But even if a large number of fire departments switched to results-based evaluation, the following question remains. How do we compare apples to apples? Fire Department A may have a lower effectiveness score than Fire Department B, but what if it jurisdiction has higher risks than those in Fire Department B's locale? Moving to results-based management is important, but it is only the first step in comparing fire department effectiveness.

In 2001, my friend Ronny Coleman helped develop a method for measuring what I call a community's fire risk exposure. The program is called RHAVE, an acronym for "Risk, Hazard and Value Evaluation." Fire departments can employ RHAVE to assess the risk level of each structure using a variety of measures such as the potential demand on firefighters and equipment, the number and type of occupants, etc. The results are assigned to a matrix of frequency and consequences.

RHAVE's goal is to give fire chiefs data which they can use to describe their community fire risk and justify budget levels. I do not know how many fire departments are using RHAVE, but I would prefer a more straight forward method that does not depend upon personal judgment.

Part of RHAVE's risk analysis places structures in one of the following categories: maximum, high, routine and remote. It is similar to a set of categories that I have previously developed. Whatever the number of categories, we need a universally accepted way to describe risk. Once we do, we can than categorize fire departments based upon their overall degree of risk exposure. The information will let us interpret fire department effectiveness scores in light of their risk exposure scores. Then we will be comparing apples to apples.

Giving credit where credit is due

In an earlier post, I suggested that fire departments analyze fire incidents and categorize each one as either a success or failure of their public fire protection systems. By analyzing failures, fire departments can identify deficiencies and then find cost-effective ways to improve their Success records.

I did not mean to imply that we should limit our analysis to failures. Our successes tell an important story, too. For example, if an apartment fire was stopped before flashover because a smoke alarm gave early warning, code enforcement contributed to that success. If the occupant immediately called 911 instead of trying to extinguish the fire, credit also goes to public education.

In other words, every success means that some element or elements of the public fire protection system contributed to the success. Analyzing your successes lets your fire department give credit where credit is due.

When public fire protection succeeds, and when it fails

In an earlier post, I suggested that fire departments categorize the outcomes of fire incidents by the quality of the results. Doing so would place fire departments on an equal footing with competing municipal services that use results-based management to justify budget requests.

I offered two measures to determine if public fire protection succeeded or failed. When a fire is stopped before flashover and with no life loss, the fire protection system succeeded. Conversely, when a fire goes beyond flashover before being stopped, or takes a life, then the fire protection system failed.

Here is why I recommend those particular measures. Measures of results must be both valid and reliable. Fire departments provide many emergency services, but are primarily equipped, staffed and deployed to stop life and property loss when fires occur. Since preserving life and property are at the core of fire department missions, they are valid measures of how well fire departments are achieving those goals.

The other reason for choosing them is that they are easily identified. That makes them reliable measures of results. One question on the NFIRS report asks if flames extended beyond the room of origin. If they did, the fire is considered to be post-flashover before being stopped. If the flames did not extend beyond the room of origin, the fire is considered to be stopped in the pre-flashover stage.

Some may think that these measures are too harsh, maybe even arbitrary. They are not. Keep in mind that public fire protection consists of several components besides manual suppression. Public education, plan review, code enforcement and automatic suppression all play a role in whether a fire incident is a success or failure.

To illustrate, let's consider a fire that went to flashover before it was reported. Obviously, the responding fire companies could not stop it before flashover if it was post-flashover before it was reported. The reason we should call it a failure is that one or more components of public fire protection failed. Was the report delayed because occupants ignored the signs of fire or tried to extinguish it before calling the fire department? That is a failure of public education. Did the fire go to flashover because of improper containment, lack of automatic suppression, etc? That could be chalked up to deficiencies in plan review and code enforcement as well a public education deficiency.

Regarding the role of automatic suppression, not all structures require sprinklers. That being said, fire chiefs may decide to require sprinklers in certain occupancies when they recognize that sprinklers will improve the success rate for fires in those structures. Thus, their presence or absence must be part of evaluating why a fire was a failure of the public fire protection system.

I'll discuss the relationship between fire risk and mitigation methods in a later post. Suffice it to say here that public education, plan review, code enforcement and automatic suppression play key roles in the results of fires, and not just manual suppression.

Fire departments need to consider the role of each component when considering their rates of success v. failure. Such knowledge will point them to activities that will improve their Success/Failure records, and that will put them on the road to achieving excellence in public fire protection –achieving the highest level of success v. failure at the lowest cost.