A university prize winner; Memorable stories from a talented new writer; The Death of Descartes, by David Bosworth. Pittsburgh: The University of Pittsburgh Press. 182 pp. $9.95

One of this season's finest fiction debuts is David Bosworth's collection of five stories, published under the title of the longest, ''The Death of Descartes.''

These memorable stories are peopled with believable characters who catch us up in their bleak contemporary dramas, even though Bosworth doesn't give them much in the way of plot to help. His drama often turns on a subtle change of mood based on a moment of revelation or self-discovery. But in his hands this becomes as gripping as fiction on the grand scale.

These are characters haunted by betrayal, whose friends, trusts, or ideas of happiness have failed them, ''born it seemed with something missing, stranded here without an answer,'' to borrow the author's poetic language. In these stories they're learning why, how, and who's to blame. Though swathed in black, this is fiction that flashes with insight.

Bosworth's book is the first winner of the Drue Heinz Literature Prize, established in 1980 by the University of Pittsburgh Press ''to recognize and encourage the writing of short fiction.'' Three of the stories first appeared in small literary magazines, and it's heartening to see the universities becoming allies of the small independent publishers in helping talented newcomers break into print.

I found the title piece to be the most riveting and polished. In it a retired and infirm detective is called to the scene of a probable murder, only to discover that ''truth'' has more shades than are recognized by the law and that occasionally a mystery may be best left unsolved.

Even if the symbolism is overdone, Bosworth's characters in this piece are wonderfully rich and believable. His skill in disguising a philosophic voyage as a murder mystery proves him to be a gifted writer whose work is a welcome alternative to the hollow fiction so common today.

Yet, despite the appeal of ''Descartes,'' I found the most haunting story in this collection to be ''Excerpts from a Report to the Commission.'' This deeply-felt saga of the Vietnam generation traces one young man's journey from naivete and shallow idealism in the '60s through a devastating chain of betrayals - by institutions, friends, family, and a lover - that leave him utterly bereft in the '70s. Though sometimes overwrought, this is a true and searing account of the disillusionment of the '60s.

The other stories excel in revealing the essence of fleeting moods. ''Psalm'' deals with subtle interplays of feeling, as a young husband and wife drive their terminally ill child to a hospital for a treatment that neither of them believes can save him. ''Dice'' relies on an O. Henry-like twist to unite the lives of partially sketched characters in a web of tragedy. And, with a crank phone call as its catalyst, ''Alien Life'' examines the sense of isolation in contemporary life.

Though Bosworth's subjects are grim, he hurls penetrating beams of compassion or justice into the gloom, questioning, if not challenging, the cruelty and absurdity of the blackness.

And what of Descartes?

He's an omnipresent villain in this fiction, that lofty rationalist who could ignore the heart but not the intellect, who robbed the world of ''meaning and mystery'' and ''killed for those who came after him what he valued most.''