After seven years of the air tanker effectiveness study, what have we learned?
08 April
Published by https://fireaviation.com/
USA- This year the Aerial Firefighting Use and Effectiveness (AFUE) study that started in 2012 will begin its eighth season.
The Government Accountability Office in 2013, a 2009 audit by the USDA’s Office of Inspector General, and Senators and Congressmen have asked questions about justifying the taxpayer’s funds that are annually allocated for firefighting aircraft by the federal government. When asked if aircraft were worth the cost and if they were effective the answers from the land management agencies have been, “Yes”. How do you know? “We just do”. (I’m paraphrasing here).
According to the Administration’s FY 2020 budget summary, over half a billion dollars was spent on fire aviation in FY 2017; $507,000,000.
The U.S. Forest Service started the AFUE in an effort to answer the question: “What are the best mixes of aircraft to do any fire suppression job?” Data collected from this study and other sources would be used to inform decisions about the composition of the interagency wildland firefighting aircraft fleet.
After seven years of the study, which costs about $1.3 million annually, very little information has been released about the status of the effort or any detailed findings that have been developed. It is almost as if the Forest Service is less than enthusiastic about what they have discovered so far. In fact, a reliable source told us that one or more high-ranking folks in the agency want it to “go away” and that detailed findings would never be released. The USFS refused to release the $840,092 RAND air tanker study completed in 2012 even after we filed a Freedom of Information Act Request. Finally RAND released it two years after it was completed, but as far as we know the USFS never did. That study recommended a heavy emphasis on water-scooping air tankers and fewer conventional air tankers, which would have been a monumental shift in the paradigm.
When we asked Christine Schuldheisz, a Forest Service spokesperson, when a report from the AFUE study would be released, she said, “The USDA Forest Service has not released a report and currently the agency does not have a timeframe to release a report. The Forest Service is collecting data to provide adequate information for a report that will be released in the future.”
We asked if the Forest Service wanted the study to “go away”, and she said, “USDA Forest Service has no plans to discontinue the AFUE program at this time.”
In stories like this, we often include the disclaimer that air tankers do not put out fires. Under ideal conditions aircraft can slow a fire enough to allow ground based firefighters an opportunity to contain sections of the fire’s edge by constructing a fireline. Strong winds or dense smoke can make it impossible for aircraft to operate safely or effectively.
The data is being collected by four “observation modules,” each comprised of three qualified firefighters, as well as a dedicated aircraft, to collect ground and aerial data at wildfires throughout the nation during fire season. In addition to the 12 firefighters, 3 analysts/managers evaluate the data. Ms. Schuldheisz said the annual cost of the project is approximately $1,300,000.
After the first three years the AFUE, the Forest Service found that the data collected from 2012 through 2014 could not be used to provide statistically defensible analysis and results. After making the necessary adjustments to their procedures, they made a commitment to begin releasing detailed annual aircraft use summaries several months after each collection season. The annual reports were scheduled to begin in the early months of 2017 for 2015-2016. By now three reports should have been issued.
Ms. Schuldheisz told us that annual reports have not been released but she sent us a copy of a two-page “Fact Sheet” about the program that she said was sent to Congress in March, 2019. (Another one-page “Fact Sheet” was released in 2017.) The recent document includes information about data collection and the preliminary information shown below about probability of success.
The two-page Fact Sheet has some preliminary information from 2015 to 2017 with enough data to report with high confidence, Ms. Schuldheisz said.
- Rotor-wing aircraft data indicates an 87% probability of success in direct attack drops, and 62% in indirect attack drops.
- Fixed-wing aircraft data indicates a 74% probability of success in direct attack drops, and 56% in indirect attack drops.
- Rotor-wing and fixed-wing have different mission profiles with a varying degrees of complexity. Both aircraft types fly direct attack missions the majority of the time.
When we asked how the researchers defined “success”, Ms. Schuldheisz replied:
Data is collected in multiple, nested scales which account for requestor objectives and then compare those to outcomes achieved at each scale and across various resource configurations. The AFUE developed hierarchical data groupings of: Resource Actions, capturing information about individual drops; Tasks, to aggregate multiple, coordinated individual resource actions, over the course of one shift or less, in support of the task work assignment; and Campaigns, to group multiple aerial and ground tasks, working in concert, for a measurable amount of time, in a defined geographic area, supporting incident objectives. By documenting outcomes independently of objectives, effectiveness can be accurately determined. To translate effectiveness into the observed probability of success, we divide the effective outcomes by the sum of effective and ineffective. Observed probability of success shows how often drops tested by fire meet or exceed their intended objective.
The Forest Service AFUE webpage includes these questions they hope to answer:
- The best mixes of aircraft to do any fire suppression job.
- Composition of the interagency wildland firefighting aircraft fleet.
- Track the performance of specific aircraft types.
- Assess the influence of the operational missions that drops supported and environmental factors that influenced outcomes.
We assume, with few details having been released by the Forest Service, that the study will collect data about four to five types of fixed wing aircraft (single engine, scooper, large, and very large) and at least three types of helicopters, Types 1, 2 and 3. Breaking it down by aircraft model and vendor would also be helpful. The type of fluid that is dropped should be recorded: water, long term fire retardant, or water with some other enhancement product.
If the study can determine the effectiveness of each of these seven types of firefighting aircraft, it should not only lead to answers about which ones are most effective, but also under what conditions of wind, terrain, fire behavior and fuel types they should be used.
Hopefully it will lead to answers to the questions from the GAO, Inspector General, and Congressmen about justifying the half billion dollars of taxpayer funds spent each year.
If the study can actually quantify the on-the-ground effective production rates of each type of firefighting aircraft, an analyst should then be able to develop a recommendation for how many of each type are needed nationwide and where they should be based.
And beyond that, algorithms or artificial intelligence could eventually, based on scientific data, make on-the-fly recommendations for which aircraft should be dispatched after a report of a new fire, based on availability of aircraft, aircraft production rates, location of the fire, fuel type, fuel moisture, terrain, scooping sites, location of reload bases, congestion at reload bases, weather, and predicted fire behavior.
At 10 a.m. EDT on April 9 the Senate Energy and Natural Resources Committee will hold a hearing “to examine the President’s budget request for the USDA Forest Service for Fiscal Year 2020.” It will be interesting to see if Chief Vicki Christiansen is asked questions about the AFUE. These hearings are usually live-streamed.
Gary (Bean) Barrett, a frequent contributor to the discussions on Fire Aviation, spent a career in U.S. Naval Aviation as a fighter pilot and served on the Navy Staff as a program sponsor responsible for planning, programming, and budgeting. Here are some of his thoughts about the information that has been released so far about the AFUE:
“This Australian air tanker effectiveness study defined results in terms of probability of success in meeting a common first [initial] attack objective of containment within 8 hours of detection. They didn’t try to differentiate between success and effectiveness in their report. They produced an excellent operationally useful study based on probability of suppression that begs for a follow-on study to compare different tanker types.
“Maybe the AFUE effort is suffering from excessive complexity by trying to address all the air tanker success, effectiveness, efficiency, and use questions on the first report. I would think it might be useful to get out an initial AFUE report with less complexity, get feedback, and then refine and expand it as more data becomes available each year. Here’s an example of an excellent report using partial data that produced an operationally useful document.
“AFUE might consider coming out with a partial report by focusing on IA objectives with basic variables. Simplify the process. If they are trying to get everything done in the first report, that just might be a bridge too far and the reason we haven’t seen any reports yet.”
