Solve a mystery, win a prize: a new trend in book promotion?

''What would you think about doing a book in which there were eight murders and no solution . . . and the publisher should give a $10,000 reward (to the reader who solved the mystery)?''

When mystery writer Thomas Chastain was asked this question by his literary agent, he fell off his chair. After he picked himself up, he wrote ''Who Killed the Robins Family?'' (152 pp., $9.95). His agent, Bill Adler, took the book to William Morrow & Co., which published it in August.

It has been on the best-seller lists for almost three months. The publisher is offering a $10,000 cash reward to the reader who comes up with the best solution to the mystery: who killed the eight members of the wealthy Robins family - and where, when, how, and why each was murdered. The contest closes April 15, and entries will be judged by an independent organization, Ventura Associates Inc., of Lowell, Ind.

The winner will be announced at the American Booksellers Association convention in Washington on May 28. And next summer Warner Books will release a paperback edition that will include the solution.

Recently, Chastain and I discussed ''The Robins Family'' over lunch at Billy's, his favorite New York restaurant. He was dressed for the occasion as a mystery writer should be: tweed jacket, rep tie, and glasses that failed to hide the twinkle in his eye. I asked him if he and Adler were worried about someone kidnapping them for the solution to the mystery and the $10,000 reward. With a smile, he replied that he was not worried: For now, their solution to ''The Robins Family'' is being held in a sealed vault in the US Safety Deposit Corporation in New York.

Chastain feels it is not impossible that a reader could come up with a solution identical to the one he and Bill Adler developed. He says that ''the answers are there, and to me they're there in a logical sequence. . . . As far as I was able, I tried to be fair.'' Part of this fairness involved skimping on details of characterization and atmosphere so typical of his other mysteries.

Yet following the traditional rules of the mystery ''as he sees them'' meant including every clue in ''The Robins Family'' but omitting any unnecessary details about the characters and plot that might confuse the reader. The result: a tale that lacks the richness of his other mysteries and reads like a novelization of a movie script.

I asked Chastain if the word ''gimmick'' could be applied to ''The Robins Family.'' He replied, ''It bothers me that it should be perceived that way.'' He was excited about the idea of ''The Robins Family'' because it was something new. ''Up to now you could not have written a book and not solved the murders and expected anybody to buy it.'' He sees the $10,000 reward as a necessary mechanism in the selling of the book and feels that ''The Robins Family'' is significant because ''it is something entirely new; it has not been done before'' in the mystery genre.

Part of Bill Adler's idea for ''Who Killed the Robins Family?'' came from Kit Williams's ''Masquerade,'' which provided a similar reader involvement in the search for a buried rabbit made of precious stones and metals. After following the clues in the book and digging up various English locations, a reader finally found the rabbit last year.

''The Robins Family,'' however, is a mental puzzle and involves no actual digging. Chastain points out that ''it's an exercise in logic - that's what all mystery stories are.'' ''The Robins Family'' does what other mysteries do - except that it involves the reader more deeply and provides the chance to win $ 10,000. He says he wrote ''Who Killed the Robins Family?'' ''in the tradition of the mystery as well as I could do it . . . . It gave me a chance to play with the whole (mystery) form in a way I've never been able to before.'' There are references in ''The Robins Family'' to a number of classic mystery situations, including a locked room, a lady vanishing, and a murder on the Orient Express.

Chastain says he intends to read every answer submitted in the contest. ''I think it's going to be great fun for me. There may be some who have better solutions than I had.''

That is unlikely. Chastain has served on the board of directors of the Mystery Writers of America and has written six mystery-suspense novels besides ''The Robins Family.'' In his first mystery novel, ''Pandora's Box,'' he introduced two characters, NYPD Deputy Chief Inspector Max Kauffman and private detective J.T. Spanner, who also appear in ''911,'' ''Vital Statistics,'' ''High Voltage,'' and ''The Diamond Exchange.'' Chastain treats New York City almost as a character in all his mystery novels except ''The Robins Family.''

Is ''Who Killed the Robins Family?'' the beginning of a trend? Thomas Chastain says, ''It's possible.'' His friend and fellow mystery writer Mary Higgins Clark thinks it may be the beginning of a new genre. And there will be a sequel to ''The Robins Family'': Mr. Chastain will soon begin writing ''The Revenge of the Robins Family,'' to be published next summer. This time the reward for solving the mystery will be $10,001.