Denis Donoghue at bay; The Arts Without Mystery, by Denis Donoghue. Boston: Little, Brown & Co. 151 pp. $15.95.

In his book, ''The Arts Without Mystery,'' Denis Donoghue proposes to reinstate mystery in art against the wishes of modern critics who ''want to deny the arts their mystery, and to degrade mystery into a succession of problems.''

The real mystery is why Donoghue's book is such an embarrassing failure. Is he just temporarily beneath his usual excellence? Or does his failure to defend his critical principles indicate a weakness in his general position?

Certainly the book itself has many faults. The backbone of the book is a series of six lectures Donoghue delivered on BBC radio in 1982. Six commentaries pad these lectures, in which Donoghue tediously elaborates tangential points and tendentiously answers criticism that his listeners, which included Terry Eagleton, put to him. A captious, defensive tone mars every argument. When he starts accusing English critics of prejudice against his Irish Catholicism, he touches bottom.

One of the reasons polemics fail Donoghue is that he takes no delight in them. In his last book about criticism, ''Ferocious Alphabets,'' Donoghue's was a voice of tolerance amid the strife of clashing ideologies. In ''The Sovereign Ghost'' he expressed ''hope that rival positions will not be regarded as a battlefield.'' His open-mindedness was a more effective rebuke to his dogmatic rivals than any equally strident attack of his own might have been.

But now he adopts the embattled spirit he previously criticized in others. He derides the psychoanalytic critics, like Lacan, for interpreting the artist, not the art. He is even harsher with the structur-alists, like Barthes, for discrediting the artist, for seeing the poet as a mere scribe and literature as an impersonal system of signs. But his angriest attacks fall upon the Marxists, for whom, as he sees it, criticism means identifying the political ideology that a particular work of art embodies and disseminates. Donoghue maintains ''the function of the arts is the critical interrogation of politics,'' and that by denying art this autonomy Marxists are forcing it to become part of the vulgar, technocratic world.

Judged on its own merits, this is an insignificant book. But when the position of a conservative critic like Donoghue becomes an opposition, something important has happened. A lot is at stake. Modern criticism assumes that the imagina-tion is a function of certain codes. If that assumption sounds like a calamity to you, then you are a Romantic. I am. I still say ''creativity'' when Lacan says ''displacement''; I say ''poet'' when Barthes says ''language''; and I say ''Plato'' when Derrida says ''Nietzsche;'' and I worry when Donoghue, who says these words too cannot make an acceptable case for the Romantic position.

I think I know why. Imagination is at the center of Donoghue's aesthetic theory. When he calls for mystery in art he is really asking us to respect the irrational, ineffable quality of artistic expression which has been, since Coleridge, the signature of the imagination. You can talk about diction, syntax, rhetoric, and structure, you can even mention politics and philology, but you must credit all to the greater power of the artist's imagination. The imagination itself is the one thing you can never articulate.

Coleridge, of course, was also a poet. Most great English critics were. It's easy enough for poets to say criticism halts at the frontiers of the imagination , because you can always read their poetry to discover what's missing. Romantic criticism serves poetry. But for the critic who is not a poet, like Donoghue, the Romantic heritage is in some respects a liability. Your starting point, the imagination, is also your limit. An argument for what Romantic criticism can do is also a confession of what it can't.

Modern critics don't labor under this built-in restraint. Since the imagination is ineffable, they choose to study the social and political laws which determine it. Their position rests on logic, Donoghue's on the illogical imagination and faith in its powers. His failure to defend himself is inevitable and a sign of just how vulnerable the Romantic position is today.

The future lies not in stubborn opposition. In ''The Sovereign Ghost'' Donoghue expressed this opinion himself. He looked to artists, not critics, for the final reconciliation and pointed to Wallace Stevens's ''The Comedian as the Letter C.'' as a step in this direction. That approach is a much more constructive one than the polemics he offers here.