Now who's telling us the truth about lie detectors?

When former Georgia Sen. Herman Talmadge's reputation and a possible jail sentence hung in the balance in 1979 over alleged financial misconduct in office , his chief accuser and the principal witness against him was given a lie detector test.

As a matter of fact, Daniel Minchew, Senator Talmadge's former administrative assistant, was given five lie detector (or polygraph) tests over a five-month period: three by the FBI and two by examiners hired by Minchew's own lawyer.

But the tests resulted in utterly conflicting testimony by ''expert polygraphers'' before the Senate Select Committee on Ethics, which was investigating Democrat Talmadge. At one point during the hearings, Sen. Jesse Helms (R) of North Carolina rumbled to one of the witnesses: ''You will forgive me, sir, if I observe that this is not the best day for the polygraph. Here we have the anomaly of two stipulated expert witnesses in direct contradiction. . . .''

Yet polygraphs were considered vital evidence as Talmadge was finally ''denounced'' by the full Senate for reprehensible conduct. He was not reelected.

They were also instrumental in Minchew's later receiving a four-month jail sentence for fraud.

Clearly, those squiggly little marks on paper in an exam based on physiological reactions, can momentously affect a person's life or reputation. But what is the truth about lie detectors?

Both their accuracy and validity are once again matters of national concern, reaching to the top of the US government. Even President Reagan has no objection if the FBI decides, as reported, to ''flutter'' a dozen high administration officials in its Carter briefing papers investigation.

James Hamilton, special counsel to the House human resources subcommittee investigating the Carter papers affair, initially said he doubted the usefulness of polygraphs in this probe. Mr. Hamilton was Talmadge's lawyer in the Senate Ethics Committee probe.

But now he has revised that position and said that the committee chairman has not decided whether to use lie detectors, adding that they can be useful in some cases.

Lie detector use is sharply on the rise in the United States. Many private companies - Kentucky Fried Chicken and Chase Manhattan Bank are notable examples - routinely use polygraphs. They do this to screen job applicants as well as to attempt to detect guilt in internal theft or fraud.

In government, polygraphs are also increasingly being used for job screening, to detect suspected leaks of classified information, and to spot probable spies. In fact, the Department of Defense views them as effective deterrents to leaks of classified information.

Conversely, the American Civil Liberties Union (ACLU), says its national legislative director, John Shattuck, sees them as ''a form of 20th-century witchcraft.''

Lie detectors even qualify as entertainment, as criminal lawyer F. Lee Bailey proved with his syndicated TV series on the subject.

How accurate are these controversial and increasingly prevalent tests? Two of the foremost polygraph experts in the country disagree.

Says Dr. David Raskin, professor of psychology at the University of Utah, ''In my opinion, when polygraphs are used in an appropriate situations by competently trained examiners, their accuracy rate is 90 percent or slightly higher than that.''

Dr. Raskin has been researching the subject for 13 years with funding from the US Department of Justice.

On the other hand, David Lykken, professor of psychology and psychiatry at the University of Minnesota, says:

''There isn't much good evidence of accuracy. Three major studies that have credibility tend to agree with each other . . . that the polygraph is wrong half of the time. In flipping a coin, the chance of accuracy is the same, 50 percent . . .''

The accuracy rate may go as high as 70 percent when the test is administered by well-trained people, Dr. Lykken adds. But he notes, ''Lie detection is biased against the truthful or honest person - 40 to 50 percent of the innocent or truthful are deceptive.''

This apparent bias against the innocent may be a matter of life or death in criminal cases, Lykken says.

''I've personally been involved in three different murder cases where the defendant was found guilty of murder and sentenced to life in prison on the basis of polygraphs,'' Lykken says. Two were exonerated when the real killers were found; the third escaped from jail, turned himself in and requested a new trial in which he was found not guilty.

Lykken is the author of ''A Tremor in the Blood,'' a book about lie detectors whose title is taken from a phrase of Daniel Defoe's for a truth test. Both he and Dr. Raskin are members of a congressional Office of Technology Assessment panel currently drafting a report on the accuracy and validity of polygraphs. That report, to be released in October, is eagerly awaited by a Congress faced with decisions on the unprecedented use of polygraphs in government under the Reagan administration.

ACLU spokesman Shattuck suggests ''there's an obvious irony in the administration which promulgated a broad polygraph order'' finding the President's men may have to take the tests. For it is not just White House chief of staff James Baker and CIA director William Casey who may be strapped into lie detectors because of their conflicting testimony on the Carter papers. But hundreds of thousands of other government employees also face possible polygraphs as a result of two recent administrative actions:

One of these is a presidential directive of March 11 ordering that all federal employees with access to classified government material may be required to submit to polygraphs when unauthorized disclosures of classified information are investigated. Those who don't submit, guilty or innocent, face ''adverse consequences.'' The directive, which originated in the office of the President's National Security Affairs adviser, William Clark, sprang from President Reagan's complaint that he was fed up with leaks to the press and wanted something done about it.

Both the presidential directive and an earlier one by the Defense Department are under a congressional freeze and cannot be implemented until next April 15. (Only the supersecret National Security Agency is exempt from the freeze).

In its own action, the Defense Department drafted a November 1982 directive vastly expanding the use of lie detectors to give random tests to an estimated 15,000 to 20,000 employees for the purpose of detecting spying and, according to critics, stopping the flow of budgetary, military, and political leaks to the news media.

The directive had its roots in Deputy Secretary of Defense Frank Carlucci's order of lie detector tests in January 1982, in an unsuccessful attempt to discover who leaked Pentagon cost estimates of the administration arms buildup to the Washington Post. Among the top brass of the Pentagon whose truthfulness and loyalty were questioned via lie detector were the chairman of the Joint Chiefs of Staff, the secretaries of the Army, Navy, and Air Force, the deputy secretary of defense, two undersecretaries, and half a dozen assistant secretaries.

The freeze until April 1984 on both the presidential and Pentagon directives allows time for Congress to hold hearings on polygraphs.

The late Henry Jackson (D) of Washington, in one of his last speeches in the Senate, proposed legislation (passed along with the freeze) that curbs both president and Pentagon in this area. It prevents either from taking ''adverse'' action against either military or civilian employees based only on lie detector tests or their refusal to submit to them.

The Defense Department policy, it was later estimated, would quadruple the number of polygraphs given there, involving up to 60,000 people if it goes through.

Senator Jackson called the polygraph ''inherently unreliable,'' said ''it could destroy any number of careers as well as the general morale'' of government employees, and noted its ''intimidating effect'' as a tool for prying out confessions. To underscore that, he quoted former President Nixon as saying, ''Listen, I don't know anything about polygraphs, and I don't know how accurate they are. But I do know that they'll scare the hell out of people.''

Few people who submit to lie-detector tests are aware of the premises on which they're based and how dissimilar their subjective findings are from scientific evidence, like fingerprints or dental records.

The test is based on physiological monitoring of a subject strapped into a series of instruments that measure changes in respiration, pulse, and blood pressure, galvanic skin changes resulting in perspiration, and muscular activity said to indicate tension.

Ideally, a polygraph examiner will put the subject at ease so that after his or her normal behavior is established with ordinary questions, the crucial ''detection'' questions can be interspersed with others and physiological changes noted.

Some experts describe lie detectors as ''fear detectors.'' The person being examined is much more likely to be at ease (or free of fear) if he has nothing to lose - such as his reputation, his job, or possibly even his life - or in a test in which a ''deception'' registered in the test will not be made public. Such a test might be administered by a polygrapher hired by the subject's defense counsel. (This is called the ''friendly user syndrome'').

''Control'' questions are used in exams frequently given by agencies like the FBI. These questions are designed to cause concern to a truthful person, and the physiological reactions to them are then compared to the reactions to other questions.

For instance, a middle-aged man being tested in the case of a specific and recent theft might be asked if he had ever stolen anything up to the age of 21. An innocent and honest man who answered ''no'' on this control question would be suspect because polygraph examiners usually assume that everyone has stolen something in his life.

When the crucial specific theft questions were asked, the response to this ''control'' question would be compared with the specific theft question. If polygraphers base their findings on the premise that man is inherently guilty of occasional small crimes or sins and base their control questions on that premise , the morally scrupulous man is at a disadvantage in the polygraph.

As Dr. Lykken says, there is a bias against the innocent. One inadvertent physical response, even a shallow breath on one of the crucial questions, may cause the polygrapher to describe the subject as deceptive on the whole test. Conversely, the Talmadge case polygraph testimony indicated that depressant drugs may be taken by the subject that can skew the emotional-physiological reactions on which the test is premised, creating the ''flat graph'' that suggests innocence to examiners.

During the Carlucci polygraphs at the Department of Defense, John Tillson IV, director of manpower management in the office of the assistant secretary for management, flunked three separate lie detector tests and was accused of disclosing official information to the Washington Post. At that point Post reporter George Wilson took the unusual step of denying that Mr. Tillson had been his source and wrote to Defense Secretary Caspar Weinburger that ''an honorable man stands falsely accused.''

Even at the Pentagon there is vigorous disagreement over the efficacy of lie detectors, as a later Post story by reporter Wilson indicates. He described the details of an internal Pentagon memo written by John Beary III, acting assistant secretary of defense (health affairs), in which Dr. Beary said the polygraph ''does not work . . . there is no physiologic response unique to the cognitive state of lying. . . . It is important to realize that the polygraph is an excitement detector, not a lie detector. . . .''

''The polygraph,'' Dr. Beary said, ''misclassifies 30 percent to 50 percent of innocent people as liars in field tests.'' He also indicated that polygraphs could actually endanger national security because Soviet spies may be trained in how to beat them.

Neither Dr. Beary nor his assistant were available or at liberty to comment on these statements, and a Pentagon spokesman said the matter was an internal memo not for public disclosure.

The increasing use of lie detectors, not just in the military but throughout government, suggests a need for a closer look at their legality and professionalism.

Stanley Abrams, a Portland, Ore., clinical psychologist and one of the leading polygraph advocates, notes that they are admissible as evidence in 37 state courts. In general, this is not the case in federal courts, but Dr. Abrams adds that they can be admissible sometimes as stipulated evidence, a fact many people are not aware of.

''Past research by various federal agencies has been very positive,'' Dr. Abrams says, but he admits that he ''would not want anyone's fate to hang on one thing alone, including polygraphs. There should be other evidence as well.''

The polygraph, Abrams says, ''is not infallible, but it is quite accurate, 90 percent accurate, in the hands of a good examiner.''

But he admits that even a good examiner can't distinguish between emotions on the polygraph: ''You can't differentiate between fear of being detected in a lie vs. anger (over the asking of the question).''

Although many of the government polygraphs are given in the District of Columbia, there are no licensing requirements here for polygraph examiners. In fact, there are no federal licensing requirements anywhere. In states where licensing is required, it involves attending a polygraph school whose instruction period is normally a fraction of that devoted to most professional sciences and much less than that required even at beautician schools. Many of the FBI's polygraph examiners are trained at the US Army Military Police School at Fort McClellan, Ala., which awards a degree after 14 weeks of instruction.

The ACLU objects to lie detectors on constitutional grounds and because of their lack of veracity, says John Shattuck. Its New York branch even opposes licensing polygraphs, suggesting that it would be ''like licensing tea leaf readers.''

One psychologist who has made several academic studies of lie detectors, Julian Szucko of the University of Illinois at Chicago, says: ''Anyone who's done work in it is either pro or anti. I began my research as a neutral and as a result of the information I've gathered have decided that in most situations it is not justified.''

In one study, he and a colleague, Dr. Benjamin Steinmetz, took polygraph test records for 100 different people and submitted them to six recognized experts. Half of the subjects had committed real crimes: in this case, thefts. The other half were innocent people who had submitted to polygraphs, but were later exonerated when the actual thieves confessed to the crimes.The accuracy of the six examiners ranged from a low of 63 percent to a high of 77 percent.

The accuracy rate on the 50 innocent people alone was much lower; one polygraph expert called 62 percent of them guilty.Dr. Szucko concludes, ''Until someone can come up with a technique that is much more accurate than what is around now, it's not a valid way of testing people to determine who is innocent or guilty.''