Flaws in networking devices highlight tech industry's quality control problem

Researchers have uncovered security vulnerabilities in widely used remote power management equipment that many say is the byproduct of a technology supply chain plagued with quality control issues. 

Security flaws discovered in common networking equipment could give malicious hackers a direct pipeline into data centers and business applications, even allowing them to remotely turn off power to critical information systems and industrial machinery.

Researchers at the Georgia cybersecurity firm BorderHawk revealed to Passcode that vulnerabilities in a widely used type of business hardware known as remote power managers (RPM) may affect thousands of companies across the country.

BorderHawk would not reveal the name of the company that makes the flawed hardware. But it is advising businesses, which often rely on these kinds of network-connected devices to remotely manage equipment, to ensure they aren't accessible from the Internet and to make sure they have been updated with newer software and firmware.

Unfortunately, security researchers say these types of vulnerabilities are not uncommon and are often difficult to detect. As companies add more networking devices or control system equipment to their overall business operations, especially those that are cheaply made overseas, they are often plugging in insecure equipment rife with vulnerabilities.

"We see lots of different devices, but a lot of the same problems," said Billy Rios, chief executive officer of the security startup Whitescope.

The issue can often be chalked up to poor quality control in the supply chain of manufacturing business networking equipment, which largely takes place in China, say experts.

"Hardware is a misunderstood, unknown territory," said noted electrical engineer and inventor Joe Grand of Grand Idea Studio. "People buy a piece of hardware and take it for granted. They assume it is secure. They assume it does what it does and only does what it does."

Small, inexpensive, and insecure

BorderHawk didn't set out to search for vulnerabilities in RPM devices. While working on another project at a large energy firm, its researchers noticed a steady stream of alerts about unusual traffic on their client's network, said Matt Caldwell, the company's chief security researcher. 

He said the traffic was disguised to look as if it came from a well-known defense contractor with no known connection to the client. It was destined for computers in France, South Korea, Russia, and Britain. It also appeared the traffic had been on the company's network for as long as a year.

That discovery set off a hunt for the origin of the traffic that ended with the 5-by-6 inch RPM device: simple network hardware containing two power outlets to plug in equipment as well as an Ethernet and serial ports for connecting to the network or directly to another computer. 

Caldwell said it is difficult to know whether RPM devices such as those studied by BorderHawk are merely the first entry point hackers can detect in an organization or whether hackers are targeting the devices specifically.

After discovering the flaw, Caldwell's team attempted to contact the manufacturer, to little effect. "They were elusive," he said. "They kept asking us what the [unique machine address] of the device was or demanding that we send the hardware back to them."

Since the vendor was uncooperative, BorderHawk wrote its own, custom tool to extract the software from the device and analyze it. Researchers also went online and purchased different versions of the same device to analyze those.

They found more reasons for concern. A help file in the product contained a link to a known, malicious domain located in China. An analysis of the device firmware found undocumented features: hidden commands that could be used to dump a list of user accounts and passwords to access the device, and other commands whose function was unknown, said Caldwell.

BorderHawk's discovery isn't the first time that security researchers have uncovered problems in RPM devices.

For instance, Shawn Merdinger, chief information security officer at Valdosta State University in Valdosta, Ga., discussed the security exposure posed by iBootbar RPM devices deployed on corporate networks, but accessible from the public Internet, at a recent security conference in Tampa, Fla.

More recently, the security consulting firm Senrio Inc. (formerly called Xipiter) found similar problems to those identified by Border Hawk in an RPM device – the NetBooter NP-02B – made by the Arizona firm SynAccess Networks. 

One hidden feature in the device's firmware lets anyone remotely reset the NetBooter device to its factory default configuration – an action that would sever it from the network. Another allows anyone to modify network and system settings. A third, hidden function could be used to extract data (like a recently entered password) stored in the device’s memory, according to Stephen Ridley, a principal at Senrio.  

In many cases the hidden functions can be used without needing a user name or password, Senrio researchers found. That means anyone who could connect to the NetBooter device and knew the proper syntax of the commands could control it, Ridley said.  

When Senrio researchers looked for NetBooter devices on Shodan, a search engine that catalogs devices connected to the Internet, they found 83 of them in the US reachable from the public Internet. The firm identified another nine in Canada and one each in Panama and Australia, Ridley noted. A search, more broadly, for SynAccess devices using Shodan identified more than 400 devices.

When contacted about the flaw and Senrio's findings, SynAccess Network Chief Executive Officer Shan Han said he was only willing to speak with the company's customers about problems with its products. "Please stop calling," he said.

Web of vulnerabilities in global supply chain

Many security experts say that the kinds of flaws uncovered by BorderHawk and Senrio are not limited to RPM devices or even to inexpensive hardware from small firms. Rather, they can be found in a wide range of hardware including networking equipment, industrial control systems, and medical devices.

The problem is a byproduct of changes in the way that technology firms source and build their products, often relying on far-flung networks of manufacturers and suppliers who operate with little oversight or quality control. 

Computer products 25 years ago were assembled in Texas from parts made in Silicon Valley and shipped directly to retail stores and companies in the US, noted Caldwell from BorderHawk. Now, he said, finished products are made of parts manufactured in China, Taiwan, the Philippines and Indonesia, assembled in China and shipped via a web of importers and distributors to stores and customers. 

When his firm began investigating RPM devices, they noted that many products that were labeled "Made in the USA" but were clearly sourced overseas. Even casual, visual inspection of purchased RPMs turned up red flags, like misspellings on product labels and compliance certificates on the products that were outdated. 

Ridley of Senrio said that his company's research on the NetBooter device even revealed the existence of a knock-off version of the SynAccess product they were analyzing, the NP-02R. Sold mostly in China and uses almost identical hardware and software. "The goal is to trick people into thinking this is a SynAccess device," he said. Such counterfeit products could eventually make their way into firms outside of China, further exposing them to risk, he said.

The problem, said Mr. Grand of Grand Idea Studio, is often that buyers aren't examining components going into much of the industrial equipment that's on the market today. 

"They just buy the hardware from a vendor that meets their specifications and that’s just accepted as good," he said. "Whatever hardware is in it, whatever software it’s running, that just goes into the final product."

Instead, he said, the supply chain for electronics should be examined as closely as the supply chain for food. “If I’m sourcing a module, I want to go and see where it's made," he said. "I want to make sure it’s a legitimate package and that the company meets my standards."