Agnes of God Drama by John Pielmeier. Starring Elizabeth Ashley, Geraldine Page. Directed by Michael Lindsay-Hogg.

A tense, anticlerical psychiatrist and a confidently determined mother superior struggle, each in her own way, to protect a mentally disturbed nun in John Pielmeier's ''Agnes of God,'' at the Music Box. Recalling the battle of wits and wills, Dr. Martha Livingstone (Elizabeth Ashley) tells how she was appointed by the court to determine whether Sister Agnes (Amanda Plummer) was competent to stand trial for manslaughter in the strangling of her newborn child. The quietly ecstatic nun denies any recollection of the crime.

Three remarkable actresses have a histrionic field day under Michael Lindsay-Hogg's direction as Dr. Livingstone employs her professional techniques, including hypnosis, to penetrate the maze of case histories and mystical intimations that make up the Pielmeier plot. Miss Ashley's vulnerable objectivity, Geraldine Page's no-nonsense aplomb as Mother Miriam Ruth, and Miss Plummer's childlike ingenuousness as Agnes combine in a vivid theatricalism to which Eugene Lee's austere, plain wooden backdrop and Roger Morgan's subtle lighting make their own contribution.

The play, however, overstrains believability. and self-destructs with hyperbole and hysteria. For all its pretensions to serious analysis of psychiatric vs. ecclesiastic dogmas, ''Agnes of God'' amounts to little more than old-fashioned sensational melodrama.