Assessment of high reliability seeking organizations – an off shore petroleum perspective

The first document we will examine in this series on assessment of high reliability seeking organizations is from the Transportation Research Board of the National Academies.  It is their Interim Report on the Effectiveness of Safety and Environmental Management Systems for Outer Continental Shelf Oil and Gas Operations. The report was commissioned by the Minerals Management Service (MMS)-now the Bureau of ocean Energy Management, Regulation, and Enforcement (BoEMRE).

The report begins by summarizing the history and purpose for undertaking this review: 

In 1990 the Marine Board reviewed the MMS inspection program and made several recommendations for improvement. The Board noted that the MSS inspections were compliance based checklists that focused on facility hardware, and that a strategy of exclusively compliance audits “diminish the operator’s recognition of his primary responsibility for safety.”   The Board further noted that the “majority of accident events occurring in a representative year (1982) were related to operational and maintenance procedures or human error that (were) not addressed directly by the hardware-oriented” checklist. The Board called for increased analysis of incidents and accidents and data collected by inspectors suggesting that MMS should “place its primary emphasis on detection of potential accident-producing situations—particularly those involving human factors, operational procedures, and modifications of equipment and facilities.”

The committee recommended that the quality and quantity of inspection data be enhanced to with a goal of developing risk indices that MMS could use to focus on platforms at higher risk. The committee stressed that “the private operator was the primary responsible agent for ensuring safe operations and that MMS should structure its program to reinforce that awareness among operators.”

In April of 2009 the MMS asked the TRB to undertake a second review.  However, subsequent to regulatory changes stemming from the BP Deepwater Horizon accident, the scope of the review was revised to consider the newly issued Safety and Environmental Management Systems (SEMS) rule and provide guidance on how SEMS programs should be evaluated to assure effectiveness.

The follow excerpts from the interim report illustrate key aspects of assessment for high reliability seeking organizations:

“The management of safety within an organization is ultimately a reflection of its safety culture. It is hoped that effective implementation of SEMS will have a positive impact on the safety culture of companies operating on the U.S. OCS; however, this will not be known until trend data are available and analyzed…

 Safety management should be integrated into a company’s organizational systems and management practices to achieve a positive culture of safety. Safety management systems are more than a set of policies and procedures; they also include how policies and procedures are implemented through work practices and the commitment of resources and support in the workplace that can truly make an impact on safety culture.

 A common problem for some companies is the tension between organizational mandates regarding safety and messages for efficiency in terms of time and money.

Companies are continually making decisions that trade off safety against other objectives (e.g., time and cost). Without a framework that keeps safety concerns elevated to an appropriate level, suboptimal decisions can be made. This can happen when the conflict of responsibility and accountability with respect to many different organizational goals (e.g., safety, time, and production) ensures that the target with the most forceful message from top management will prevail. Building trust that top management will support decisions to override other priorities with safety is the only way to achieve a culture of safety; however, SEMS alone cannot achieve this. 

With its audit program, the Bureau of Safety and Environmental Enforcement (BSEE) will be in a unique position to influence how SEMS is implemented and integrated in an organization. To achieve reliably safe operations, more than a well-defined SEMS is needed – people in the organization must actually use SEMS and improve it on a continuing basis. Thus, auditing has to extend beyond the existence of a SEMS—and the existence of documentation that supports its use—to assuring that what is described in the SEMS is actually the way people work. An effective audit program would extend assurance beyond paper verifying records to how the SEMS is used to guide what individuals in the organization do to ensure safe and environmentally responsible operations. 

…the committee has identified nine methods for assessing the effectiveness of an operator’s SEMS program: audits, compliance inspections, peer reviews–assists, key performance indicators, whistleblower programs, periodic lessee reports, tabletop exercises or drills, SEMS monitoring sensors, and calculating the risk with SEMS in place…. 

Properly auditing a safety management system is not only about finding and reporting deviations, but also about assessing the current state of how safety is assured and finding specific opportunities for improvement (i.e., identifying weaknesses in the system). An audit of performance standards should have the purpose not just of identifying uncontrolled or inadequately controlled hazards, but also of finding the strengths and weaknesses of the safety management system itself.”

So what might we take away from the Board’s interim report?

1)    Compliance based audits are not sufficient.

2)    Actual work practice must be assessed.

3)    Assessments should produce data that informs a risk based profile of the organization

4)    The value of assessment is not to identify what has happened but to detect precursors of what might happen

5)    How you assess and what you assess shapes how an organization performs

6)    Culture is the driver for how practice is aligned with policy, systems and standards

7)    Multiple methods are necessary to generate rich performance data

8)    An explicit framework is necessary for making sense of performance data

9)    Performance based assessment should provide insights about the basic assumptions that underlie the safety management system, its strengths as well as weaknesses.

Those are my take-a-ways; what about yours? Did you see other items in the full report that should be included?  The next posting will cover the 2010 report from VTT looking at evaluation in the nuclear industry.


2 Responses to Assessment of high reliability seeking organizations – an off shore petroleum perspective

  1. wroege says:

    Earl, another excellent post. Could we have the link to the full report in the blog?

    I like the approach that the safety management system isn’t enough–one has to use it.

    I’ve always felt the short term management sell out to efficiency/schedule occurs because long term risk is not appropriately considered. Safety items are generally investments and address long term risks at the expense of short term revenues and time. However, when those prudent investments are not made, the unmitigated long-term risk eventually results in a large event that ends up costing much more to clean up and interrupts production for orders of magnitude longer than needed for the near term investment.

    In the long run, organizations that carefully consider and invest in safety systems and improved processes invariably outperform those that don’t. In many cases those that do not make the investments go out of business. If BP was a smaller company, it would be gone.

    Bill Roege

  2. Bill Mullins says:

    After reading the Transocean Investigation Report on the Mancado Well catastrophe I was struck by how the decision-making relationship (or lack thereof) between the drill operator (Transocean), the cement seal contractor (Halliburton), and BP (the responsible licensee) seemed to leave the platform operators very little say in the period of two weeks or so before the explosion engulfed the platform. During that period it would appear that BP’s management of change processes were overtaken by the dynamics of the subsurface structure.

    Unless these SEMS involve an effective knitting of mutual responsibilities among the several parties that control the conduct of platform work it seems that there is a gaping hole in the assurance framework.

    More study of this interaction of responsible parties is needed and perhaps I didn’t catch the relevant pieces of the Marine Board review report, but there is something about this piece that seems dated to me. I’m thinking there is a good deal more to these complex, high-consequence circumstances involving open systems poised against extreme forces of nature than the presumptive notions that production and protection can be managed as tradeoffs rather than comprehensively integrated under a single Duty of Care.

    We could start by dropping the substitution of the dubious construct “safety culture” for “safety climate” or “safety consciousness.”

    Bill Mullins

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: