As software struggles persist, consultant says Muskrat Falls completion impossible to predict

In one of its most damning assessments yet, a consulting firm monitoring progress at Muskrat Falls says it’s now impossible to set a completion date for the troubled hydroelectric project.

In its most recent quarterly report to the Newfoundland and Labrador Public Utilities Board, Liberty Consulting focused on efforts to develop power and control software for the Labrador-Island Link (LIL), which has suffered repeated setbacks and been plagued by bugs.

The software is critical to the safe and efficient operation of the 1,100-kilometer Labrador to Avalon Peninsula line, which is capable of transmitting up to 900 megawatts of power over two separate lines, or poles.

But the contractor, GE, has repeatedly missed deadlines on the software, and each new release uncovers more and more glitches.

Therefore, the report’s authors say that “no reasonable forecast of commercial operation of LIL at full capacity can be of substantial credibility.”

According to the report, it may take another year, “and maybe significantly longer” to complete the project and that NL Hydro must now prepare for “another winter season ahead without a reliably functioning LIL”.

“Currently, the software does not support operation at any power level in a way that system operators can consider reliable,” reports Liberty.

Liberty has “great concern” about methods

In its March 3 monthly PUB update, NL Hydro expressed similar concerns, writing that the trial operation was behind schedule.

“It is not possible at this time to predict the extent of a possible delay,” the update said.

Some 3,200 transmission towers like this one have been erected from Muskrat Falls to Soldiers Pond as part of the $3.4 billion Labrador Island link. (Nalcor Energy)

It’s another somber portrait of a project that is years behind schedule, billions over budget and the subject of a public scrutiny that Muskrat called a misguided project and led to the dismantling of Crown-owned Nalcor Energy.

The project has grown to $13.1 billion, nearly $6 billion more than the original 2012 estimate, and has required a multibillion-dollar federal government bailout to prevent muskrat-era electricity prices from skyrocketing fast.

Liberty Consulting is a US-based company hired to oversee the progress of the project and its integration into the province’s electrical system.

After recent meetings with NL Hydro officials, Liberty said it had “major concerns about the methods used to overcome very longstanding problems with the LIL software”.

Jennifer Williams is President and CEO of Newfoundland and Labrador Hydro. (Terry Roberts/CBC)

Liberty said it takes an “extraordinarily long time” to perfect the software and that it’s not normal to launch trial versions of the software through the link, rather than using simulations.

“Testing has continued to fail in its purpose of finding and correcting errors before the system and customers are deployed as a testbed for discovering remaining problems,” Liberty wrote.

That the software here still has significant flaws after so many years of development and failed testing can put customers and HVDC system equipment at risk. Good management should not see this as normal.– Freedom Advice

“That the software here still has significant flaws after so many years of development and failed testing can put customers and HVDC system equipment at risk. Good management should not see this as normal.”

The link was powered in October and for several months provided up to 425 megawatts of power, or about half Muskrat Falls’ rated output.

In mid-December, an unexpected shutdown dubbed the Bipole Trip occurred, affecting approximately 22,000 customers. Liberty said such an incident should occur at most once every 10 years.

A more serious incident occurred on February 20 when a fire sensor triggered one of the two lines and the entire connection has since been offline due to the software not responding properly.

There was no fire, but Liberty said “this outage has resulted in another indefinite shutdown of LIL while GE investigates the additional discovered and critical software defect, while continuing to address other issues with the software.”

GE will be held accountable

Hydro’s President and CEO Jennifer Williams responded to the report Friday afternoon and said there was frustration at the ongoing delays. However, she said Hydro will not compromise on its software requirements and that GE will be “held accountable” for its contractual obligations.

“I’m not so focused on the exact date. Our hard line is that it has to be the right software package,” Williams said.

Williams also denied that Hydro was taking unnecessary risks by running the link on preliminary software. She said extensive offsite testing is conducted before the software is used on the link.

“We are not careless. When we have an outage or a trip, it doesn’t feel good. We don’t use the system as a test bench, it has to be put into operation at some point,” she says.

Williams declined to comment directly on Liberty’s pessimistic outlook on the software, but stressed that a new version is expected in the coming weeks.

Although it’s possible the new connection won’t be fully operational by next winter, customers shouldn’t worry about the reliability of the power grid, Williams said. She said Hydro is keen to wait Holyrood thermal power station until it is certain the system can meet demand.

Meanwhile, she said an update on project costs is expected this summer. While the $13.1 billion price is expected to increase, she doesn’t expect it to be “significantly” higher.

Read more from CBC Newfoundland and Labrador

Leave a Comment