How big data helps big cities

When troves of information are opened to programmers, problems get solved.

By order of the White House, June 1 marks the first National Day of Civic Hacking. Cities across the country will invite programmers to rally together and improve local government one line of code at a time.

In New York, more than 80 teams have signed up for NYC Big Apps, a competition in which people hunt for digital solutions to the city's problems. Projects range from the serious (software that compiles crime, garbage, and other city statistics to create a quality-of-life index) to the amusing (a game in which players race to claim territory in the city by snapping photos of billboards with their cellphones – if an ad isn't in New York's official database, the billboard is flagged as likely being illegal).

Many of these projects would be impossible without a recent shift in American cities. Mayors in New York, Chicago, and elsewhere have opened up a fire hose of data about their cities. This flood of information has allowed city workers to stay on top of problems – some of which they never knew existed – and has helped launch new businesses within the city limits.

"When I joined as chief data officer, the initial goal for releasing all of this data was transparency," says Brett Goldstein, who heads Chicago's Department of Innovation and Technology. "But along the way, we learned a couple of really interesting things. We learned that open data can do so much more than that."

About a year ago, Scott Robbin, a programmer in Chicago, approached Mr. Goldstein with an idea. He wanted an application that could send e-mails or text messages to residents as a street sweeper approached their homes, giving people an extra reminder to move their cars. Goldstein liked the idea – he'd received a few tickets of his own – but his team would likely never find time to create such an app. It didn't need to. The city found the data, published it online, and let the community take it from there. Shortly after the database went public, Mr. Robbin launched SweepAround.Us as a free online service.

Another app, SpotHero, uses a mixture of city data and business partnerships to sniff out nearby parking spots. The local company recently raised $2.5 million in venture-capital funding and has expanded its operation from Chicago to Milwaukee and Washington.

Goldstein says services such as SweepAround.Us and SpotHero are a win-win for Chicago. The newly released data made the city government more transparent, helped start businesses, and led to useful tools for residents.

Chicago is not the only city trying to code its way toward better government. In addition to hosting competitions such as NYC Big Apps, New York has hired a team of digital sleuths. These number crunchers – mostly economics majors just a few years out of college – try to turn massive amounts of data into predictions about the future.

Among their early assignments was "illegal conversions," when landlords divide up housing units in an attempt to cram more people into a building. Such tight living quarters often lead to fires, infestations, and preventable deaths. The city could only follow up on a fraction of the complaints it received each year. It needed a better way for inspectors to decide which complaints were worth their time.

According to the new book "Big Data: A Revolution That Will Transform How We Live, Work, and Think," New York's geek squad "started with a list of every property lot in the city – all 900,000 of them. Next they poured in data sets from 19 different agencies indicating, for example, if the building owner was delinquent in paying property taxes, if there had been foreclosure proceedings, and if anomalies in utilities usage or missed payments had led to any service cuts."

The team discovered clear patterns. Before they dug into the data, inspectors had found major violations at about 13 percent of the buildings they searched. Now, it's more than 70 percent.

"We need to tell developers, 'let's solve the city's problems,' " says Noel Hidalgo, New York City program manager for the nonprofit Code for America. "Great ideas come from admitting that we don't live in a siloed world. Having people in government that understand the power of open data will be crucial to life in the 21st century."

For more on how technology intersects daily life, follow Chris on Twitter @venturenaut.

[Editor's note: This is an updated version of an article that appeared in the June 3 issue of the Monitor weekly magazine.]