Tale of 'Bob': Does outsourcing new software pose cyber security risk?

Many US companies hire foreigners to build new software for their computer networks – a practice that may raise their risk of cyberattack, some experts warn. Even firms that do not outsource software development may find an occasional employee doing it on the sly, as in the case of 'Bob.'

A software developer at a US company providing "critical infrastructure" – transportation, electricity, water, or the like – last year secretly outsourced his job writing computer programs to software engineers in China. Dubbed "Bob" by investigators – to keep his identity and that of the firm private – he even overnighted his electronic Secure ID token to China so the workers there could log into his company's network. 

That left Bob, who paid the Chinese software engineers a fraction of what he earned to do his work, plenty of time to surf the Internet and watch cat videos. But it also left Bob's company vulnerable to having its computer network compromised, possibly in ways that interfered with company operations or jeopardized public safety, some cybersecurity experts say.

In this case, the Chinese workers to whom Bob outsourced his work have so far not been identified as cyber monkey-wrenchers, according to a Jan. 14 blog by those who investigated Bob's exploits. But the episode serves as a warning to the thousands of US companies that opt to outsource their software development work to firms abroad, in an effort to cut costs, cybersecurity experts say. The practice, they warn, represents a big hole in the cybersecurity shield America needs to build to protect itself from cyberattack.

"If an attacker is part of your organization as an outsource contractor – writing code, or building the chip – they are in effect insiders with all kinds of advantages that enable them to cause you and your customers all kinds of grief," says Seymour Goodman, a professor of international affairs and computing at the Georgia Institute of Technology.

The cybersecurity risk from outsourcing isn't new. Back in 2005, Dr. Goodman chaired the cybersecurity panel for the Association for Computing Machinery, which found that "offshoring [of software development] magnifies existing risks and creates new and often poorly understood or addressed threats to national security, business property and processes." But the threat continues to grow as companies outsource not just software for smart phone apps, but also software tools that run corporate websites, networks, and databases.

The "Bob" episode came to light during a review of his company's data logs, which revealed that an unknown intruder was connecting daily to the company's network from Shenyang, China, according to "risk team" investigators from Verizon, a provider of cybersecurity services, hired to look into the breach. Bob had received sterling performance reviews, but his Web browser history revealed that he spent a typical work day as follows:

9 a.m. – Arrive and surf Reddit for a couple of hours. Watch cat videos.

11:30 a.m. – Take lunch.

1 p.m. – Ebay time.

2-ish p.m Facebook updates – LinkedIn.

4:30 p.m. – End of day update e-mail to management.

5 p.m. – Go home.

"They’re a US critical infrastructure company, and it was an unauthorized ... connection from CHINA," the investigators wrote with emphasis. "The implications were severe and could not be overstated."

While Bob outsourced his software work without his company's knowledge, many other suppliers of "critical infrastructure" offshore such work as a matter of course.

"We are aware of several critical infrastructure organizations that outsource development projects overseas," says Robert Huber, a principal investigator with Critical Intelligence in Idaho Falls, Idaho, a company specializing in security for critical infrastructure providers. "Without a thorough security review by someone in your organization, you have no idea of the issues that are being introduced to your networks that may expand your attack surface." Malware inserted into software in the "software supply chain," as it is being written, can leave companies vulnerable to theft of their intellectual property, he says.

Software products that defense contractors supply to the Pentagon, for use in microelectronic and telecommunications, are also at risk. Most contractors have geographically dispersed supply chains that create "a vulnerability of potential insertions of malicious hardware or embedded software on the hardware components," the US-China Economic and Security Review Commission warned in a report last year to Congress. 

Problems the report cited included a desktop computer purchased by the Army and made in China by Lenovo. The new computer was discovered to be "beaconing" (attempting all by itself to establish a connection) "to a suspicious foreign entity," the report noted, citing a US Army official who revealed the 2007 incident last February.

The software export business worldwide is booming, as companies around the globe look outside their own national confines to fulfill their software needs as cost-effectively as possible. Ireland, a leading exporter of computer software and services, saw its exports soar to $37 billion in 2010, up from $7 billion in 2000. India's software export sales nearly tripled in five years, hitting $45 billion in 2011. China's software export sales soared to $30 billion last year from $10 billion in 2007, the lion's share headed to the Japanese market, according to the UN Conference on Trade and Development's 2012 report on the global software industry.

American firms are major buyers of software development services from abroad, say researchers at Duke University, in Durham, N.C. Among US software companies, half of all development projects were headed to India and 13 percent to China, a 2008 Duke survey found. Nearly one-quarter of all US companies expected to outsource software development to China.

Against that baseline, US software outsourcing has only accelerated, suggest unpublished Duke data from last year. Helping drive the trend is the emergence of at least 120 eBay-like Internet platforms such as freelancer.com, where software developers worldwide can bid on software projects large and small, Duke researchers say.

"What's amazing to me is that roughly one-third of those bidding on such forums for software development projects are people in full-time jobs – and I'm sure the companies that employ them have no idea," says Arie Lewin of the Duke Center for International Business Education and Research, citing yet-to-be published survey results on software outsourcing by US companies.

Dr. Lewin's "Offshoring Research Network" 2008 survey showed that "data security" and "lack of intellectual property protection" in the software development cycle are among US software companies' top five concerns about outsourcing.

"We were quite amazed about the low maturity level of companies managing these software development projects," Lewin says. "Opportunities to penetrate them must be amazing. What you need to be able to do is have capabilities in place to manage and monitor these vendors. But in my opinion, top management doesn't give high priority to this."

One trend that alarms Goodman, Lewin, and other cybersecurity experts is that US companies are not adequately inspecting outsourced software for security flaws. Among software-outsourcing companies, the share that also farmed out their quality-control and security testing jumped from 72 percent to 87 percent in a year, a 2012 InformationWeek survey found.

A company that does not do its own security testing is like letting the fox guard the hen house, says Richard Hoffman, a software developer who helped conduct the InformationWeek survey.

"It's understandable that companies are seeking cost savings," he says. "But these companies writing the software are also often inspecting and testing security, too. In many cases, the same people charged with keeping costs down are also supposed to catch security holes."

Still, not everyone in the cybersecurity realm is ready to hit the panic button over software outsourcing. While the practice carries a risk, the threat may have waned at bit, says James Lewis, a cybersecurity expert for the Center for Strategic and International Studies, who wrote a 2007 study on software outsourcing. That's because cyberattackers have cheaper ways to penetrate a company's or agency's computer network, he says.

"Hacking in from the outside is so easy now that in most cases it's probably just not cost effective [for cyberspies or cybercriminals] to insert malware when the software is written," Dr. Lewis says. "Still, this isn't a hypothetical problem. ...As the US begins to get its act together on cybersecurity, you'll see the cost and benefits of hacking change. Then those attackers might look to more costly approaches."

As for "Bob," described as a "family man, inoffensive and quiet," the digital trail eventually revealed that he was freelancing for other US companies – and shipping those software code-writing assignments to China, too, according to the Verizon investigation team's blog. While "Bob" was paid hundreds of thousands of dollars a year from his company and for freelance work, the Chinese firm got perhaps $50,000, investigators estimated in their blog.

Bob's fate is not publicly known. Some people who left comments on the investigators' blog declared him unethical for secretly farming out the work and breaching company security. Others complimented him.

"Sooo… where’s the problem?" reads one comment. "He improved his personal profit and the quality and efficiency of his work, obviously. And all that by using standard business practices – get money to do the job, then pay someone else less to actually do it. This guy is an American hero and deserves a medal."

Others declared the Verizon blog post to be a hoax. After all, wasn't there also a report in The Onion, the satirical online news website, headlined "More American Workers Outsourcing Own Jobs Overseas"? Yes, there was.

Responding to doubters, Verizon's team on Jan. 18 followed up its original blog post with another one declaring that "the case is factual and was worked by one of our investigators."