What a Sunken Ship Teaches Us About Failed SCADA Projects

What a Sunken Ship Teaches Us About Failed SCADA Projects

By: Bradley Mull

5 reasons SCADA projects fail.

Isn’t the whole point of history to teach us how to learn from failures? I found a case study about the failures of the sunken warship Vasa, and made many comparisons to the reasons behind why some SCADA projects fail.

Here’s a bit of background: The Royal Swedish Navy’s warship, Vasa, sunk in the bay on its maiden voyage in the 1600’s. It hadn’t even made it out of the harbor, when a slight wind caused it to tip. The ship sunk, and 53 men died. After the Vasa was uncovered from the water, it was discovered it couldn’t have withstood winds over 8 knots, approximately 9 miles per hour.

Why did this happen?

Many parts of the design, building, and testing process were doomed from the start.

Here’s what we can learn from the Vasa in order to ensure SCADA projects aren’t doomed from the very beginning.

 

1. Lack of specifications

Building the Vasa was the King of Sweden’s idea. He wanted a giant ship to help him win the war against Poland. The king signed a contract with his shipbuilders, but never wrote out his specifications or even sketched a plan.

Without details, the shipbuilders started construction. They decided to roughly scale the ship based on a smaller version built years before. That decision turned out to be an absolute disaster.

Due to the lack of detail in the original specifications, zero documentation, and many further changes, the finished keel was too shallow for the ship’s relative size. Ultimately, there wasn’t enough space for the ballast required to stabilize the top-heavy ship.

Takeaway: Detail your specifications early

 

The specification is the system integrator’s Bible. If we have a question, we check the spec. There’s no such thing as too much detail. Often, the engineering company who writes the spec will provide a device/equipment brand name, but no guidance on the number of points required, or its ultimate function.

In those cases, we try to meet with the owner to discuss SCADA projects, but that’s not always possible. If we don’t have the details from the engineer, and don’t have access to the end-user, most of the time we take our best guess and make assumptions.

To be on the safe side, we will probably include more points. This obviously leads to greater expense in terms of design and licensing, and may not even be close to the original intention.

But without a specification, we’re blind.

2. Unrealistic deadlines

The king demanded the Vasa be launched by July 25th, providing just two and a half years of construction. With dozens of major changes within those two and a half years, there was no way it was going to be ready in time.

After a very rushed construction, the captain and the king’s admiral tested the ship. After only moments, the test was halted because the ship couldn’t be stabilized. With the king clamoring to launch the ship, the Vasa was launched on its first (and last) voyage anyway...even though the test indicated it needed more than twice the current ballast.

 

Takeaway: Manage expectations and involve the integrator from the beginning

 

The #1 reason projects take longer than expected is because the system integrator isn’t included in the beginning of discussions, when schedules are being decided.

A lot of new customers don’t understand exactly how much time SCADA projects and implementation actually takes. Most of what control systems integrators do takes prep work you might not see onsite. This means the amount of time allotted for the project is usually significantly reduced from its realistic length.

If the system integrator isn’t involved from the beginning, they usually get their information second or third-hand from the electrical contractor. Through the grapevine, important information (like scheduling detail) is easily lost or miscommunicated.

Ensure those that manage your projects dig into everything that could possibly impact the schedule (holidays, other subcontractors, extended timelines, etc.) Take away the red tape and let your integrator communicate with all project stakeholders (client, construction team, electrical contractor). Better yet, include the integrator in your scheduling meetings and CC them on every scheduling email.

 

3. Excessive change orders

In the king’s original contract, the Vasa was a small, traditional Swedish warship. Over time, it became a monster with massive, untested, unnecessary innovations.

During the entire year of 1625, the king of Sweden made confusing changes to the Vasa’s plans. After hearing Denmark was building a bigger, better ship, he commanded the ship’s size be enlarged. He also kept changing the amount of artillery onboard, in order to ensure his warship was better. The first version of the Vasa had 32 24-pound guns, then it scaled up to 36, then 64, but it ended up launching with 48.

The problem is, gun weight and placement drastically change how a ship should be built. Because parts of the Vasa were already constructed with original sizing, the shipwrights were challenged to somehow accommodate more weight in the remaining construction. They didn’t succeed. The added weight upset the ship’s delicate center of gravity, leading to its ultimate demise.

 

Takeaway: Keep change orders to a minimum through detailed specification

 

One of the most expensive parts of system integration are change orders. The reason change orders exist is because the customer/owner/engineer didn’t provide enough detail up front, or didn’t clearly outline assumptions.

Most of the time, owners don’t realize the effect their change orders have on the entire project. Minor changes along the way are ok, but big changes can end up nearly doubling the original cost and scope.

We’ve had projects that end up lasting years, when they were originally supposed to last just a few months, because customers keep submitting change orders.

If you decide a change order is necessary, make sure you understand the implications and decide if it’s really worth it.

 

4. Unnecessary bells and whistles

The Swedish king ordered that the Vasa be decorated with beautiful (but bulky) carvings throughout the ship. In the 17th and 18th centuries, it was believed the more impressive the warship, the more indestructible it would be. He wanted it to be the most ornate ship on the planet, and it was.

But the weight from the carvings further disrupted the center of gravity, and added to the long list of issues that made the ship sink to the bottom of the harbor.

 

Takeaway: Don’t overspecify. You’ll end up paying for it later.

 

You don’t always need all the bells and whistles in your SCADA system. More is not always better…it just costs more.

Don’t think you’re overspecifying? Your engineer might be. There’s a tendency for some engineers to use cut-and-paste or boiler plate instrumentation specifications. Even if you only need four of 20 device features, you might be paying for all 20 because the engineer is too lazy to write up the specification himself. Or he’s too busy to dig deep and find out what makes sense in your individual environment.

One key to a low cost SCADA system is tailoring a specification in terms of what you need to operate your facility. Specify only the functions absolutely required, and eliminate the rest.

A good integrator will work with you ahead of time to ensure they adapt the specification to your unique and individual facility needs. They may even question your specification when something seems out of place or unnecessary, helping you achieve lower cost SCADA projects.

 

5. Poor project management

After the Vasa was salvaged back in the 1960’s, thousands of research hours have examined why it failed so embarrassingly. In all those hours of analysis, researchers have never been able to find a documented project plan.

The lack of documentation is likely due to the Vasa’s quick construction, but it also speaks to the lack of project management. After the original shipwright died halfway through construction, the new project manager (the shipwright’s assistant) had no prepared plans, milestones, or goals to work off of…and didn’t bother creating any.

Nobody successfully acted as the intermediary between the king and the shipbuilders. Nobody set goals, kept shipbuilders on task, or kept the king in check. New construction aspects were added without asking: Why? Is there a better way?

 

Takeaway: Make sure SCADA projects are overseen by a project manager

 

I know a lot of control systems integrators without devoted project managers, who trust engineers to manage their own projects. In some cases this may work, but in others it can take much more time and lack the necessary organization needed for a successful project.

At Affinity Energy, we have dedicated project managers devoted to making sure projects are completed on time, under budget, and within original specifications. With a project manager at the helm of the project, quality, consistency, communication, and visibility are higher.

With good project management comes risk reduction, consistency, and ultimately, project success.

 

Takeaways from the Vasa

The whole point of reading historical accounts is to learn from the mistakes of our ancestors. Let’s never forget what the Vasa taught us about avoiding failed SCADA projects:

  • Specify early and in detail
  • OK deadlines with system integrators ASAP
  • Minimize change orders
  • Stop unnecessary overspecification
  • Trust an integrator with dedicated project managers

 

Brad Mull - Engineer | Affinity EnergyBradley Mull is an Application Engineer at Affinity Energy, with over 17 years in software development and systems integration.

Bradley’s achievements at Affinity Energy include significant contributions to the data center company Quality Technology Services (QTS), the CEP SCADA system for Carolina Healthcare System, and the Duke Energy OSG SCADA system for monitoring emergency power generation at customer sites.

Prior to joining Affinity Energy in 2008, Bradley worked as a Software Developer at AGV Products, and as an Electronics Engineer for the U.S. Army Research Laboratory.

Bradley graduated from the University of North Carolina at Charlotte with a B.S. in Electrical Engineering, so he understands there are 10 types of people in the world: those who understand binary, and those who don’t.