Horrors, darling! A mansion shortage.

Wealthy home buyers are quickly running out of mansions to buy.

While housing inventory is falling throughout the country, it's falling especially fast in some of the country's richest ZIP codes. A study from Altos Research, the Mountain View, Ca., real-estate research firm, found that inventory in the nation's 90 wealthiest ZIP codes fell 15 percent over the past year, slightly faster than the broader market.

But in the richest ZIP codes, inventory is down more than 50 percent. In a ZIP code in Carmel, Calif., inventory fell 76 percent over the past year. There were only four homes left on the market priced at $1 million or more as of the end of May, according to Altos.

In Palm Beach, Fla., the number of $1 million-plus homes has plunged by 70 percent, falling from 89 to 26. And in the Old Greenwich, Conn. ZIP code, there are only 10 homes left priced at $1 million or more, down 58 percent, according to Altos. (Read More: $1 Million Hamptons Rentals on the Rise)

"I don't recall seeing the market like this, and it's come so quickly," said Cristina Condon of Sotheby's International Real Estate in Palm Beach. She said buyers have poured into the market in recent months, many from overseas. American buyers are also piling in—some from higher-tax states like California, lured by low taxes and still-low prices in Florida.

Condon said one of her listings that sold in the past year is a $11.3 million property on the Intracoastal that had six bedrooms, two baths, and Tuscan-inspired gardens, along with a pool and a boat dock. 

Interest in her remaining listings remains strong. She cited strong interest in a $34.9 million lakefront estate in Palm Beach as an example. The 13,278-square-foot mansion has 7 bedrooms, 8 baths, a sprawling pool overlooking the Intracoastal Waterway with outdoor loggias. (Read More: Biggest US Home to Be Completed in 2015)

In Connecticut, some $1 million-plus homes are selling just days after being listed. David Oglivy of David Oglivy & Associates in Greenwich said he had a listing at $1.38 million that sold in just two days. He said the home was viewed 14 times.

"The Old Greenwich market right now is just super hot," he said. About half the buyers are people moving within Greenwich and others are moving from New York City and other areas, he added. While some realtors worry that shrinking inventories could crimp future sales, Oglivy says he's not worried.

"The inventory is fine," he said, adding that luxury inventories in the broader Greenwich area remain higher.

Carmel and Pebble Beach, Calif., have seen a rush of buyers from Silicon Valley as well as from Europe and Asia. But the number of new, high-end listings has been limited.

"We are seeing an influx of luxury second home buyers coming into the market including venture capitalists, tech money, oil and gas, developers and CEO's," said Tim Allen, of Tim Allen Properties in Pebble Beach. (Read More: Wealthy Horse Owners Jump Back Into Equestrian Estates)

Pebble Beach and Carmel have seen a total of 106 sales in the past four months—well above the previous four months. A 3,000 square-foot ocean-front home in Carmel sold for $16.5 million this year, which marked a new record for Carmel.

A home in Pebble Beach sold for $22.5 million, though the new owners plan to remodel. Allen said that an added factor in the high end of the real-estate market is that sellers are under less financial pressure to sell, which leads to less inventory. He said inventories remain healthy, with some high-end neighborhoods still filled with "for sale" signs, but some enclaves are selling out fast.

"These sellers can hold on until they see prices where they want," Allen said. "In some of these areas, all it takes is five buyers and you can sell out."