More than an adventure, this ambitious novel deserves attention; Mysteries of Motion, by Hortense Calisher. New York: Doubleday. 517 pp. $17.95.

Hortense Calisher's fiction has intrigued and baffled readers and critics alike since her first stories began appearing more than 30 years ago. Her work encompasses an impressive, bewildering variety: autobiographical stories about her own (Southern American German-Jewish) heritage; ''problem'' novels dealing with marital and social crises; and philosophical books about imperfect communication among people nestled in their own private worlds.

There is throughout Calisher's work a strain of imaginative unconventionality , perhaps best seen in her brilliant novella ''Standard Dreaming'' (1972), which treats the generation gap as possible evidence that the human species is in decline. The unconventionality is also displayed in the proto-science-fiction novel ''Journal from Ellipsia'' (1965), whose heroine's decision to escape her imprisoning life takes on extraterrestrial, feminist, and ingeniously comic dimensions.

The difficulties inherent in her work are magnified, not just by Calisher's interest in formal philosophy and the new sciences (such as psychotherapy and astrophysics), but also by the mandarin style which bespeaks her allegiance to the example of Henry James. Her prose is, at times, either sonorously rich or else terse, crabbed, and urgent. Her sentences are confusingly, densely packed.

All these qualities and emphases are formidably present in ''Mysteries of Motion,'' Calisher's ninth and most ambitious novel. It takes the form of a ''logbook'' kept by Tom Gilpin, who's a passenger on the first American space shuttle for civilians, the Citizen Courier, en route to ''the first public habitat in space,'' known as ''Island US.''

The time is the 1990s. The main narrative - a small, sturdy heart pulsing inside this book's oversize body - concerns the Courier's troubled voyage and moves suspensefully toward resolving the mysteries that gradually trouble its travelers: the identity of the unseen other passengers; the nature of the payload that threatens to deflect the vehicle off course; their suspicion that the spacecraft is piloted but by remote control and that NASA may have known that the mission would fail.

But the major space is taken up by extended flashback accounts of the lives of the inhabitants of Cabin Six. Tom Gilpin, our narrator, is a wealthy journalist, whose editorial urgings have forced the government to democratize the Courier's passenger list. People from all levels and walks of life are on board.

In Cabin Six there are Veronica Oliphant, a black journalist who shares with Gilpin a marriage of convenience; John Mulenberg, a businessman with Mideast connections; Lievering, a German refugee intellectual who may be a survivor of a Nazi concentration camp; William Wert, a diplomat chosen to be civil administrator on Island US, and Wert's Iranian wife, Soraya. And there is a stowaway - Moleson Perdue, teen-age son of the NASA admiral who, Gilpin comes to believe, may have knowingly sent them to their deaths.

The novel's complicated structure presents these people's interrelationships as examples of ''the movements we make toward one another's mystery.'' Backward and forward we're tossed in and out of both first-person and omniscient narration, as we listen to Gilpin imagining what's going on in their minds.

An example: The opening section gives us Gilpin describing his present circumstances; then recalling, first, his boyhood on a Maine island, next, his stay in Italy during the time of the 1969 Apollo moon mission. Then we forward-track to Gilpin's last night on earth. Calisher handles this material with perfect adroitness, but the reader may find himself struggling to keep pace.

Calisher's characters live for us because there's a weight of relatedness, an impression of depth and extent about them; we willingly follow as she explores them in all their aspects, even with her slow, meditative pace and the frustrating thickets of her heavily stylized prose.

Whenever it isn't near-turgid, her style can be arrestingly beautiful, capable of such images as ''the blaze at a porthole of stars raining like snow'' and the sighting of a city just before dawn ''with its stars waning, its buildings coming forward out of night into the stone of themselves.''

In the climactic pages, the inhabitants of Cabin Six learn that they have become ''the outside people'' to those left behind on earth. I mustn't reveal what has happened to them - and what yet may happen. But I will give a clue: It seems to me ''Mysteries of Motion'' is saying that the Citizen Courier's hopeful flight toward the territory ahead is emblematic of our needs as individuals to go outside and beyond what we are, to transcend. The spacecraft is burdened by that ''payload'' it must, yet cannot, jettison; so are we all weighted down by responsibilities and relationships that continually exert their claims on us just when we think we've broken free of them. Thus is ''motion,'' itself, inevitably a ''mystery.''

That may be a needlessly reductive way to approach a rich, troublesome novel, whose intricacies and challenges readers should explore for themselves.

I don't want to leave the impression that this is a swift, dramatic adventure story, or that any but the most diligent readers will find their efforts appropriately rewarded. But I do think that ''Mysteries of Motion'' is one of the most considerable novels that has come our way in recent years, and that anyone who means to keep up with contemporary American fiction must grapple with it.