Annenbergs: rags to riches; The Annenbergs, by John Cooney. New York: Simon & Schuster. 428 pp. $18.95.

Many of us who aren't sitting on a pile of greenbacks tend to look at the very rich with a mixture of fascination and suspicion. We're tempted to agree with the cynic who said that ''behind every great fortune lies a crime'' - or at least some dirty deals.

''The Annenbergs'' describes the sort of case Balzac would have cited. It chronicles the rise of the Annenberg family from the slums of Chicago to palatial Palm Springs; from a horse-racing news service to a publishing empire that includes TV Guide and Seventeen magazine; from the disgrace of a tax-fraud conviction to the prestige of a US ambassadorship.

The first part focuses on the family's patriarch, Moses, a complex, ruthless immigrant whom gangster Lucky Luciano once praised as ''my kind of guy.'' Moses built his fortune in the borderland between legitimate business and organized crime, only to spend his last months in prison after an FBI probe.

The second part spotlights Moses' son, Walter. Taking as his credo the line from a prayer, ''Cause my works on earth to reflect honor on my father's memory, '' Walter strove to purge Annenberg businesses of any taint of racketeering - while new entrepreneurial gambles reaped fabulous profits. Today he enjoys a position of eminent respectibility, knighted by the British Queen for his contributions to Anglo-American friendship.

The Annenberg saga is skillfully written by journalist John Cooney, who combines the dramatic flair of a good storyteller with the researching skills of a former Wall Street Journal staffer. With real-life histories like this, who needs John Jakes?