Stretching the limits of the mystery genreThe Killing Doll, by Ruth Rendell. New York: Pantheon Books. 258 pp. $12.95.

A character in the latest Clint Eastwood movie, ''Tightrope,'' comments that we all have darkness in us. Some of us keep it under control, some let it take over, and others walk a tightrope between light and darkness.

The same could be said of the characters in Ruth Rendell's psychological mystery novels. Many of them seem to be walking that tightrope, and some of them fall off it into the darkness.

The mystery in Rendell's psychological novels is not ''who done it,'' but who will fall off the tightrope, and what will the consequences be? In dealing with these questions, Rendell continually pushes out the limits of the mystery genre. In her latest novel, ''The Killing Doll,'' she pushes the limits further than ever before.

Rendell begins her chilling novel with this striking sentence: ''The winter before he was sixteen, Pup sold his soul to the devil.'' The heart of the mystery in ''The Killing Doll'' lies in that sentence. Did the devil buy Pup Yearman's soul in exchange for giving Pup whatever he asks for? Does Pup have magical powers? Are the events that follow attributable to his powers, or are they merely coincidental?

Pup, whose real name is Peter, lives in a shabby, yet respectable middle-class London neighborhood with his recently widowed father, Harold, and his reclusive older sister, Dolly. Harold, who owns a typewriter business, spends every spare minute reading biographies of European royalty. Dolly, whose right cheek is disfigured by a prominent birthmark, works as a dressmaker, attends seances, and keeps house for Harold and Pup.

Pup decides to sell his soul after reading Marlow's ''Faustus'' in school. The first thing he asks for is to grow taller. Pup does grow, and he devotes himself to becoming a magician. Eventually he realizes he would have grown anyway, and his interest turns from magic to his job and his girlfriends. He doesn't believe he has magic powers, and everything that happens, including the deaths of two people and Pup's success in love and business, can be explained rationally. Yet his sister, Dolly, believes he has mystical powers, and that is the crux of the novel.

Ruth Rendell specializes in examining the inner guilt and darkness of her characters. In ''The Killing Doll,'' she takes this a step further than usual. One of her characters, a young man named Diarmit Bawne, is suffering from a severe mental illness. Dolly, who becomes more and more emotionally dependent on Pup and his supposed magical powers, retreats further and further into a guilt-ridden fantasy world. Although Rendell treats Diarmit and Dolly with great sympathy, I found her use of these mentally disturbed characters distressing.

Pup, on the other hand, appears to be the most sane and sunny character in this novel or in any of Rendell's psychological novels. Or is he? Everything works in Pup's favor. His worst sins seem to be some well-meaning deceptions of Dolly and his girl friends, an ambitious self-interest, and an ignorance of the seriousness of Dolly's mental state. But is his cheerfully guiltless acceptance of his own good fortune at the expense of others the natural reaction of an optimistic, callow young man, or the hardness of one who has sold his soul? It is this uncertainty about Pup and the things that happen to him that sets ''The Killing Doll'' apart from other mystery novels.

At the same time it has all the hallmarks of the best mystery novels. As well as being suspenseful, the plot displays a tidy inevitability. The writing, atmosphere, and characterization are all excellently done. The shabby London neighborhoods, moody atmosphere, and eccentric, slightly pathetic characters are wonderfully evoked.

Reading ''The Killing Doll'' is a thought-provoking and, at times, disturbing experience. If you enjoy reading mystery novels, it's an experience you shouldn't miss.