The first document we will examine in this series on assessment of high reliability seeking organizations is from the Transportation Research Board of the National Academies. It is their Interim Report on the Effectiveness of Safety and Environmental Management Systems for Outer Continental Shelf Oil and Gas Operations. The report was commissioned by the Minerals Management Service (MMS)-now the Bureau of ocean Energy Management, Regulation, and Enforcement (BoEMRE).
The report begins by summarizing the history and purpose for undertaking this review:
In 1990 the Marine Board reviewed the MMS inspection program and made several recommendations for improvement. The Board noted that the MSS inspections were compliance based checklists that focused on facility hardware, and that a strategy of exclusively compliance audits “diminish the operator’s recognition of his primary responsibility for safety.” The Board further noted that the “majority of accident events occurring in a representative year (1982) were related to operational and maintenance procedures or human error that (were) not addressed directly by the hardware-oriented” checklist. The Board called for increased analysis of incidents and accidents and data collected by inspectors suggesting that MMS should “place its primary emphasis on detection of potential accident-producing situations—particularly those involving human factors, operational procedures, and modifications of equipment and facilities.”
The committee recommended that the quality and quantity of inspection data be enhanced to with a goal of developing risk indices that MMS could use to focus on platforms at higher risk. The committee stressed that “the private operator was the primary responsible agent for ensuring safe operations and that MMS should structure its program to reinforce that awareness among operators.”
In April of 2009 the MMS asked the TRB to undertake a second review. However, subsequent to regulatory changes stemming from the BP Deepwater Horizon accident, the scope of the review was revised to consider the newly issued Safety and Environmental Management Systems (SEMS) rule and provide guidance on how SEMS programs should be evaluated to assure effectiveness.
The follow excerpts from the interim report illustrate key aspects of assessment for high reliability seeking organizations:
“The management of safety within an organization is ultimately a reflection of its safety culture. It is hoped that effective implementation of SEMS will have a positive impact on the safety culture of companies operating on the U.S. OCS; however, this will not be known until trend data are available and analyzed…
Safety management should be integrated into a company’s organizational systems and management practices to achieve a positive culture of safety. Safety management systems are more than a set of policies and procedures; they also include how policies and procedures are implemented through work practices and the commitment of resources and support in the workplace that can truly make an impact on safety culture.
A common problem for some companies is the tension between organizational mandates regarding safety and messages for efficiency in terms of time and money.
Companies are continually making decisions that trade off safety against other objectives (e.g., time and cost). Without a framework that keeps safety concerns elevated to an appropriate level, suboptimal decisions can be made. This can happen when the conflict of responsibility and accountability with respect to many different organizational goals (e.g., safety, time, and production) ensures that the target with the most forceful message from top management will prevail. Building trust that top management will support decisions to override other priorities with safety is the only way to achieve a culture of safety; however, SEMS alone cannot achieve this.
With its audit program, the Bureau of Safety and Environmental Enforcement (BSEE) will be in a unique position to influence how SEMS is implemented and integrated in an organization. To achieve reliably safe operations, more than a well-defined SEMS is needed – people in the organization must actually use SEMS and improve it on a continuing basis. Thus, auditing has to extend beyond the existence of a SEMS—and the existence of documentation that supports its use—to assuring that what is described in the SEMS is actually the way people work. An effective audit program would extend assurance beyond paper verifying records to how the SEMS is used to guide what individuals in the organization do to ensure safe and environmentally responsible operations.
…the committee has identified nine methods for assessing the effectiveness of an operator’s SEMS program: audits, compliance inspections, peer reviews–assists, key performance indicators, whistleblower programs, periodic lessee reports, tabletop exercises or drills, SEMS monitoring sensors, and calculating the risk with SEMS in place….
Properly auditing a safety management system is not only about finding and reporting deviations, but also about assessing the current state of how safety is assured and finding specific opportunities for improvement (i.e., identifying weaknesses in the system). An audit of performance standards should have the purpose not just of identifying uncontrolled or inadequately controlled hazards, but also of finding the strengths and weaknesses of the safety management system itself.”
So what might we take away from the Board’s interim report?
1) Compliance based audits are not sufficient.
2) Actual work practice must be assessed.
3) Assessments should produce data that informs a risk based profile of the organization
4) The value of assessment is not to identify what has happened but to detect precursors of what might happen
5) How you assess and what you assess shapes how an organization performs
6) Culture is the driver for how practice is aligned with policy, systems and standards
7) Multiple methods are necessary to generate rich performance data
8) An explicit framework is necessary for making sense of performance data
9) Performance based assessment should provide insights about the basic assumptions that underlie the safety management system, its strengths as well as weaknesses.
Those are my take-a-ways; what about yours? Did you see other items in the full report that should be included? The next posting will cover the 2010 report from VTT looking at evaluation in the nuclear industry.