Cisco helps Albuquerque Water Authority Modernize, Protect Critical Infrastructure

Innovation authority

More than 650,000 users count on the Albuquerque Bernalillo County Water Utility Authority, the largest water and wastewater utility in New Mexico. It operates a dual groundwater/surface water supply system relying on the local aquifer and water imported from the Colorado River Basin via the San Juan-Chama Project. And the organization maintains more than $5 billion in assets―including 3,000+ miles of water supply pipeline.

Like most water utilities, the Water Authority was running legacy, often proprietary assets that required careful maintenance to ensure efficient operations and effective conservation of water. The organization has also been focused on mitigating growing threats to cyber and physical security while addressing the new regulations within America’s Water Infrastructure Act. Passed in 2018, the law requires water utilities that serve more than 3,300 people to develop or update risk assessments and emergency response plans.

In February 2021, the entire industry received a stark reminder of why those regulations are so important. A hacker successfully breached a Florida water utility. Having gained unauthorized access to its water treatment system, the hacker boosted the level of sodium hydroxide (lye) in the water to 100 times higher than normal. An employee was able to correct the level before it impacted the water supply. Had the lye successfully made it into the water supply unnoticed, it could have caused burns and severe sickness among those who consumed it.

Fortunately for the communities served by the Albuquerque Water Authority, their utility was already at the forefront of the industry. By the time the breach in Florida occurred, this team in Albuquerque had made significant progress in modernizing―and better securing—its infrastructure.

Modernizing the network

Water utilities usually count on air gapping—that is, separation between their IT network and physical infrastructure—to safeguard mission-critical assets. But as utilities embrace smart water capabilities, the deeper integration between cloud, information (IT), and operational technologies (OT) is creating new security issues requiring a new generation of cybersecurity.

"The benefits of smart water and Internet of Things (IoT) sensors are compelling in terms of proactive maintenance, customer experience, and water conservation. The flip side is that they dramatically expand the threat surface.”

- Kristen Sanders, Chief Information Security Officer, of the Albuquerque Bernalillo County Water Utility Authority

“The benefits of smart water and Internet of Things (IoT) sensors are compelling in terms of proactive maintenance, customer experience, and water conservation. The flip side is that they dramatically expand the threat surface,” says Kristen Sanders, Chief Information Security Officer, of the Albuquerque Bernalillo County Water Utility Authority. “We needed a way to gain visibility and implement controls across all connected assets.”

To realize the benefits while mitigating the risks, the Water Authority has implemented Cisco industrial Ethernet switches and switching platforms that connect water sensors throughout its infrastructure as part of the Cisco Country Digital Acceleration. To segment and secure that OT network, Cisco Industrial Security Appliance (ISA) 3000 provides advanced threat protection built on the proven Cisco Secure Firewall and other network security capabilities. This provides a ruggedized solution that helps ensure safe, reliable service delivery.

The Water Authority also invested in Cisco Cyber Vision, a solution designed to gain visibility into all OT assets, and to detect cybersecurity threats and abnormal process behaviors.

“We found that our OT team didn’t necessarily understand the nuances of cybersecurity, while we in IT were unfamiliar with the programmable logic controllers (PLCs) and human machine interfaces (HMIs) that our OT colleagues use every day,” Sanders says. “Cisco Cyber Vision helped bring us together—giving us visibility into exactly what devices we have and what they’re doing.”

To accelerate that connection while also reducing the risks, Cisco CX Advanced Services connected the supervisory control and data acquisition (SCADA) system to the network, deployed Cisco Cyber Vision, and integrated it into the water treatment facility. The CX team completed the planning and execution in just 4 short months when the project was scoped to take as long as 8–12 months.  

But the benefits don’t end there. Cyber Vision also makes it easier to secure all parts of the utility’s network. Because it can be embedded into select Cisco network equipment, it was quick and simple to deploy at scale, especially in locations where space could be an issue. Not only did this architecture considerably reduce the total cost of ownership, but it was also able to provide comprehensive visibility on the smallest details of the industrial infrastructure.

Using Cyber Vision, the Water Authority could establish a baseline for “normal” device behaviors and network activity. Managers now receive alerts when anything occurs outside the norm. IT also has access to a detailed and accurate asset inventory, making it easier to identify software vulnerabilities to patch and build security policies to enforce.

Positioned for security, compliance, and flexibility

The Water Authority’s collaboration with Cisco has resulted in stronger, more future-ready infrastructure. Modern infrastructure and better visibility across IT and OT have positioned the organization well when it comes to addressing compliance with America’s Water Infrastructure Act and in preventing a security breach that could put its customers at risk.

Seeing is one thing but doing something to limit risk is another. To take advantage of the visibility and awareness into device behavior supplied by Cyber Vision, the team extended their current deployment of Cisco Identity Services Engine (ISE) from their IT environment to OT. Cisco ISE leverages Cisco Industrial Ethernet switches to restrict communications between groups of assets as defined by the OT team in Cyber Vision. ISE makes it easy to implement zones and conduits in industrial networks to contain potential threats and maintain compliance. Armed with Cyber Vision’s deep awareness of the device, the team can now confidently ensure that security doesn’t get in the way of supplying water to the 650,000 users who count on a safe and reliable water supply.

In addition, the smart water sensors are enabling the organization to proactively identify potential issues―from water leaks to excessive vibration levels that could suggest the need for repair or replacement.

“Rather than having a pipe burst and cause extensive damage, we can identify issues well before they impact the public,” Sanders explains. And their public focus also applies to their critical billing applications. The Water Authority enhanced their security posture by deploying Cisco Secure Workload (Tetration) to protect their important applications through comprehensive visibility and control, ensuring those applications are available, resilient, and do not create a vulnerability that could be exploited.

Sanders also adds that several components of the updated network enabled the water authority to pivot quickly to remote work during the Covid-19 pandemic. That includes a robust voice infrastructure, multifactor authentication, and cloud security capabilities―all of which made it possible for staff to work from their homes.

“When staff started working from home, we really expanded our digital footprint, creating a larger attack surface for cybercriminals to try and exploit,” Sanders says. “Our earlier technology investments made this transition seamless and kept us secure.”

The end result

Ultimately, these investments helped the Water Authority meet its goals, physically and digitally, securing its entire operation. In a desert environment, where water is a sparse and precious commodity, visibility across both OT and IT, enabling predictive analytics for proactive agility, and strong cybersecurity ensure that water makes it safely to the faucets of the utility’s customers.