Markets fail. That’s why we need markets.

Two camps have fought the political and philosophical battle for influence over the economy in the United States for the past 100 years. They differ in their views over the nature of markets and government. And both are wrong.

One camp makes it sound as if markets can do no wrong. Their views were ascendant in the 1920s and again in the 1980s. 

The other camp argues, “Markets fail, and that’s why we need government.” The idea is that markets are prone to excesses and imbalances and need the thoughtful, steadying hand of government to protect consumers and investors from flaws and uncertainties of the market. This camp believes that wise technocrats can and will bring order to the markets. 

In the wake of the financial crisis that gave way to the broader economic downturn, the advocates of government involvement in the economy are once again on the march and traditional defenders of markets are in retreat. And so we have seen government advance its role with partial ownership of many big banks, with a take-over of automotive firms, with a large “stimulus” program, with proposals for cap-and-trade for carbon emissions, and with a major initiative on healthcare.

The traditional defenders of free markets have had a rough time getting a hearing lately. After all, it certainly appears as if markets don’t work in any meaningful sense. The Dow rises and plummets in harrowing fashion, the housing market balloons and then craters, financial services firms teeter on the edge of extinction one minute and swing to record profitability the next. 

Surely, government can do better. 

Or can it? Over the past two generations, a different view of markets and government has begun to emerge, one whose moment may have arrived. It is a view that believes both traditional camps have overlooked some important aspects of markets. 

What’s more, it is a view that, if embraced, helps reinterpret market gyrations and government interventions in a way that better reflects reality. The view is subtle, but it has a profound influence on how the public and policymakers should think of markets and, ultimately, the role of government in the economy. 

This view can be summarized as “Markets fail. That’s why we need markets.” 

This seemingly paradoxical view is based on several overlapping strands of research in economics as it pertains to development, history, technology, business expansion, and new-firm formation. According to this view, entrepreneurs at work in the economy – in finance, high tech, manufacturing, services, and beyond – are constantly experimenting, creating new business models, techniques, and technologies that upend the established order of things. 

Some new technologies and innovations are genuine improvements and are long-lasting welfare enhancers. But others are the basketball equivalent of pump fakes – they look like the real deal and prompt market actors to leap hastily into action, only to realize later that their bets were wrong.

Given this dynamic, markets are unpredictable, prone to booms and busts, characterized by bouts of exuberance that are rational or irrational only in hindsight.

But markets are also the only reliable mechanism for sorting out this messy process quickly. In spite of the booms and busts, markets drive genuine long-run innovation and wealth creation. 

When governments attempt to impose order on this chaotic and inherently risky process, they immediately run up against two serious dangers. 

The first is that they strangle new innovations before they can emerge. Thus proposals for a Consumer Financial Protection Agency, a systemic risk regulator, a public health insurance plan, a green jobs policy, or any attempt at top-down planning may do more harm than good.

The second danger has to do with the nature of political economy. Politics creates its own kind of innovators who can be as destabilizing to markets as market actors themselves – but in far more pernicious ways. 

Economists call these political entrepreneurs “rent-seekers.” Rent-seekers gain wealth, not by creating it, but by channeling it through political favors. Examples include government-sponsored monopolies, “targeted” tax breaks for special industries, and legislative loopholes inserted by lobbyists. 

The boom in housing and mortgage securities that ended so badly was fueled by government policies that were encouraged by rent-seekers in the real estate, home building, and mortgage finance industries. 

Rent-seekers aren’t partisan. They used President Bush’s push for an ownership society to promote sketchy mortgage products. Before that, they used President Clinton’s push for a fairer economy to compel banks to make loans to poorer neighborhoods. In both cases, rent-seekers turned political slogans into profit, but at a steep cost to society when the boom ended.

The response to the current economic crisis has perpetuated and even intensified this process, as hundreds of billions of dollars of taxpayer funds have been used to prop up the very firms that took such reckless risks. The bigger the bad bet, the bigger the bailout. 

This gets to the key difference between markets and governments. When innovation-driven excesses and imbalances are recognized in the marketplace, the system can correct itself quickly. This is less the case when government policy failure occurs. 

Because political failure is less publicly tolerable than market failure, the temptation becomes for policymakers to avoid acknowledging their role in creating or perpetuating problems. Or they double down on bad bets. So rather than recognize the government’s central role in the housing boom and bust and quickly changing its ways, we see the federal policy apparatus continuing to throw good money after bad in the mortgage market and on Wall Street.

Markets fail; but they learn from their failures. That’s why we need markets. Government can promise to guarantee our prosperity; but only markets can really deliver.

Arnold Kling is an economist with the Mercatus Center’s Financial Markets Working Group at George Mason University. Nick Schulz is DeWitt Wallace Fellow at the American Enterprise Institute for Public Policy Research. They are coauthors of “From Poverty to Prosperity: Intangible Assets, Hidden Liabilities and the Lasting Triumph Over Scarcity.”