Connecticut's Stratford offers a riveting 'Hamlet'; Hamlet Tragedy by William Shakespeare. Starring Christopher Walken, Roy Dotrice, Anne Baxter, Fred Gwynne. Directed by Peter Coe for the American Shakespeare Theater.

As ever, ''Hamlet'' remains the most glamorous test of an actor's mettle, marking a graduation into the theatrical big leagues. Christopher Walken, best known for such films as ''The Deer Hunter'' and ''Annie Hall,'' passes with honors.

In the generally riveting American Shakespeare Theater production, he is a properly fretful prince, positively reeling under the misgivings and forebodings the play piles onto him. Yet there's a touch of perversity about him, too - he seems to take a dour satisfaction from his dark project of reconciling royal duty, familial feeling, and grim revenge.

At times he seems to relish his own indecision, which prolongs the impossible task that has obsessively engaged his heart and mind. Add a natural energy and enthusiasm for life running just below his troubled response to present circumstances, and you have a complex and fascinating interpretation of this perennially popular role.

Thus the slim, scraggly-bearded Walken doesn't only pace the castle corridors like a caged cat, wrapped in a black cloak like a premature shroud. He whoops and hollers a bit, too, turning even a random ''How now, Horatio'' into a little dance of boyish high spirits. Equally boyish is his self-absorption, especially vivid during the electric ''Get thee to a nunnery'' scene with Ophelia, as Walken teeters on the edge of both violence and tenderness.

Many of his line readings are unconventional, and some are downright strange. Yet their idiosyncrasy suits Walken and his conception of the prince and fits the high energy level of the production as a whole. It's a sharp improvement over Walken's involving but half-formed work in the ''Henry IV, Part 1'' given at Stratford earlier this season.

The other players are similarly strong but in radically different ways. Roy Dotrice plays Polonius with a fierce piety that's cunningly undercut by the character's lovable pomposity. As in his recent Falstaff portrayal, Dotrice plays the audience like a master musician, milking even his pauses and hesitations with uncommon eloquence. And he gives a second virtuoso turn as the talky gravedigger.

For anyone who got to know Fred Gwynne through his hilarious work in TV's bygone ''Car 54, Where Are You?'' it's always a surprise and a revelation to remember how capable he is on the most lofty dramatic levels. His version of Claudius is touched with humor - like Hamlet, he seems to take a bemused satisfaction in his own predicaments - but manages withal to be as imposing as he is effective. Anne Baxter is a poignant Gertrude, Chris Sarandon a sturdy Laertes. Michael Allinson capably doubles as the Player King and the Ghost. Lisabeth Bartlett is a lovely Ophelia, as winning in the early scenes - girlishly mocking her father's portentous pronouncements - as she is heartbreaking in the end.

In all, a strong and sturdy ''Hamlet,'' with more than enough force to justify its quirks.