That tattered old currency can't hack it with the automatic tellers

It's the old Red Riding Hood scenerio played out in the marketplace.

One Saturday afternoon, the unsuspecting consumer visits her bank's automatic teller machine (ATM). She feeds her bank card into the machine, expecting to withdraw money from her account. She waits a few moments and, when nothing happens, starts pushing all the buttons on its little computer face. Having swallowed the card, the machine smirks, ''Transaction cannot be completed.''

With an average of 3,800 ATM withdrawals per month, or 45,600 a year, there could be many such scenes, particularly when the machines are asked to serve up worn-out money.

And who is the woodsman who comes to the rescue? Not the banks or the ATM manufacturers. The woodsman is the Federal Reserve System.

ATMs have mushroomed around the country. As of August 1982 there were 23,600 - an increase of 40 percent a year since 1975, estimates H. Spencer Nilson, president of the Nilson Report, an advisory service for payment-systems executives.

Breakdowns occur when ''unfit'' currency - bills with holes, tape, tears, or excessive soil - jam the machine. The simple solution is to use new currency. An ATM can accept either new currency or ''fit'' currency, which is still in good condition, but not both simultaneously; it takes three minutes to reset the machine for one or the other. But new currency is in limited supply.

The number of unfit bills in circulation has grown rapidly since the mid-1970 s. Today, there are at least 3.8 billion, estimates a Treasury Department official; 95 percent of all new currency replaces unfit currency.

Before 1974 people checked each bill manually. What this system lacked in speed - an operator could sort through only about 40,000 bills a day - it made up for in quality. In 1970, for example, the Fed destroyed 35 percent of all currency because it was unfit.

By 1978 that figure had dropped to 30 percent. In an effort to increase productivity, the Fed bought equipment in 1974 which sorted twice as fast as people. But it inspected money package by package, not bill by bill. This ''strap sorting,'' which missed many unfit bills, is the main culprit for the influx of poor-quality currency now circulating.

Since 1978 the Fed has invested $35 million in developing an accurate, high-speed sorting system. Today 37.6 percent of all currency is destroyed, says John Kerr, vice-president of the Federal Reserve Bank in Atlanta. The Fed has installed 95 machines manufactured by Recognition Equipment Inc. (REI). These can read 400,000 bills individually each day and will add 15 more by mid-1983. Two-thirds of all bills are checked by REI machines, up from 15 percent in 1979.

Until the unfit currency problem totally disappears, banks might adjust their ATMs to process new currency. However, since 1977 the Treasury has stepped up its production of new $5, $10, and $20 bills - the denominations used in ATMs - by only 10 percent per year, from 1.1 billion notes to 1.8 billion, according to Treasury Department statistics. That hardly keeps up with the 31 percent annual growth rate of ATMs.

Rumors that the Fed would begin charging banks for new currency meet vigorous denials, however.

''Providing new currency is considered a governmental function,'' says an official at the Federal Reserve Bank of San Francisco. ''We could not charge for currency unless Congress passed such a bill, and I don't think that will happen.''

Nor will large ''correspondent'' banks be able to charge their ''respondent'' banks - small banks which get money from other banks rather than directly from the Fed - because the money was given to correspondent banks free. What large banks can and have been doing, however, is funneling little or no new currency to banks without ATMs.

Fed officials say this is a problem to be worked out in the marketplace. Since private banks can't charge for new currency, banks and the Fed believe ATM manufacturers should develop more flexible machines.

Such observations are rapidly becoming moot, for the Fed has almost solved the problem. In addition to investing in REI sorters, the Fed has taken other steps.

* The Federal Reserve System's Board of Governors approved a change in its 12 branch banks' accounting system, which had deterred them from issuing new currency.

The Fed pays the Treasury Department $20.60 for every 1000 notes. Currently, the cost of the bills - which is expected to be $100 million this year - is an operating expense. That means branch banks' net income is reduced by that amount.

As of Jan. 1, 1983, the banks will put the cost of new currency below the bottom line of their income statements. So the amount they pay the Treasury for new bills will no longer have a negative impact on their performance measures.

* The Fed has invested $600,000 to improve the fitness sensor on the REI sorters, which is good at detecting excess soil, but not holes, tears, and tape.

* For the first time Fed banks will sample the sorted bills for fitness. But the Fed suggests banks also ''deep fan'' packages of bills before putting them into their machines.