Landmark case upholds freedom of the press; Minnesota Rag, by Fred W. Friendly. New York: Random House. 379 pp. $12.95

The safeguards of liberty, Justice Felix Frankfurter once wrote, have frequently been forged in controversies involving ''not very nice people.''

Jay M. Near was one of those people. The editor and chief writer of the Minneapolis Saturday Press was as cantankerous as his publication - a scandal sheet - was despicable. But, through a chain of unlikely events, Near's perverseness led to a landmark Supreme Court ruling.

In 1925, to counteract allegedly libelous reporting by the appropriately named Duluth ''Rip-saw,'' Minnesota enacted a public-nuisance statute that permitted a single judge to treat a newspaper or magazine as he would a garbage dump: He could enjoin it from publication forever if he considered it to be ''obscene, lewd and lascivious . . . or malicious, scandalous and defamatory.''

Two years later, Near ran afoul of this gag law when he printed a virulently anti-Semitic diatribe as a way of urging the local prosecutor to investigate organized crime, which he believed was Jewish-controlled. Acting on the prosecutor's complaint that the Saturday Press had defamed city officials and newspapers in addition to the Jewish community, a local judge issued a temporary restraining order, which prevented anyone ''from producing, publishing, circulating, having in their possession, selling and giving away'' any copies of the publication. Lower courts upheld the order until the Supreme Court reversed it in 1931.

The issue of prior restraint against future publication was basic to a free society, but Near could not afford to pursue it. An unexpected ally then appeared. Col. Robert (Bertie) McCormick, the right-wing publisher of the Chicago Tribune, who was a champion of freedom of the press, entered the battle, and soon became, in Friendly's words, ''the Daddy Warbucks of the First Amendment.'' As his lawyer told him, ''Whether the articles are true or not, for a judge without jury to suppress a newswriter by writ of injunction is unthinkable.'' McCormick himself had also been the object of politically motivated gag attempts.

To the Supreme Court that decided Near v. Minnesota, the First Amendment, which guarantees freedom of speech and press, was largely uncharted territory. Chief Justice Charles Evans Hughes, aware that the states were viewing the court's resolution as a guide to possible press-curbing legislation, assigned the opinion to himself.

After a half century, his opinion on the case, which Friendly reprints, remains inspiring.

The ''operation and effect'' of the gag law, Hughes noted, was ''of the essence of censorship.'' Hughes concluded, ''The fact that liberty of the press may be abused does not make any the less necessary the immunity of the press from previous restraint in dealing with official misconduct. Subsequent punishment for such abuses as may exist is the appropriate remedy. . . .''

From this sweeping opinion has sprung a continuing constitutional tradition against prior restraints, and one of the many merits of Mr. Friendly's engrossing book is to remind us how vital this tradition is to our liberty.