Is pendulum of justice swinging too far toward favoring victims of crime?

As the pendulum starts to swing in favor of protection for the victims of crime - as some legal observers say it is starting to do - the long-fought-for rights of the accused could be slowly chipped away.

If it swings too far, civil libertarians warn, the United States Constitution and the Bill of Rights would become the victims - along with the public.

Up to now, the US Supreme Court has generally protected the rights of criminals and those accused of crime under the Fourth through Eighth Amendments, which, among other things, ban unreasonable searches and seizures and excessive bail and fines as well as guarantee the right to a speedy and fair trial.

An increasingly tough-on-crime high court, however, and state and local judges, Congress, and law enforcement agencies - all sensitive to the public mood - are starting to heed President Reagan's call to ''tip the scales on the side of the victim, not the accused.'' Among the trends:

* The US Supreme Court has failed, so far, to modify the controversial exclusionary rule, which forbids admission in court of evidence obtained by ''unreasonable'' search and seizure. The justices have held out on this issue despite pressure by law enforcement officials who say many cases are being thrown out of court and criminals going free because of improper police procedures (often unintended).

But in a series of recent cases, the high court has, in effect, narrowed the use of this rule, which is central to the Fourth Amendment. For instance, last week it decided in a drug-related case that a police-obtained search warrant based on a tip from an anonymous informer was constitutionally valid. This overturned the Illinois Supreme Court and reversed two decades of its own precedents on this type of matter.

The court chose to sidestep deciding a ''good faith'' exemption to the exclusionary rule which the White House and many law enforcement advocates have pushed for several years. But observers say this issue could come up again in another case the court has agreed to decide.

* The President's Task Force on the Victims of Crime is getting widespread publicity and apparent public support. Federal proposals include tougher bail, parole, and evidence rules. And the Reagan administration continues to push for tighter rules on the insanity defense and a ban on federal court review of state criminal convictions.

* The Federal Bureau of Investigation and other law enforcement agencies are stepping up their argument that some heretofore illegal or questionable police practices, including wiretapping and the use of undercover agents and informants , are needed to check crime today.

''The simple, traditional, 'knock-on-doors' kinds of law enforcement are not adequate anymore,'' says FBI Director William H. Webster.

* More than a score of bills have been introduced in Congress to limit the power of federal courts, including the Supreme Court, to make rulings that might be considered permissive toward crime and criminals.

* A public outcry against lawlessness seems to be growing through grass-roots movements. For example, California's voter-passed referendum, dubbed the ''Crime Victims' Bill of Rights,'' mandates longer prison terms, makes bail more difficult to obtain, and restricts plea bargaining.

Meanwhile, citizen groups, such as Mothers Against Drunk Driving, are monitoring courts in several states to assess whether judges are imposing stiff enough sentences.

Civil rights groups view these moves with alarm, saying they endanger Fourth Amendment rights. Even some moderates and conservatives view attempts to limit the exclusionary rule as tampering with the Constitution.

Former Supreme Court Justice Potter Stewart recently said the ''good faith'' exemption is unworkable, because police may end up asking not ''What does the Fourth Amendment require?'' but ''What will the courts allow me to get away with?''

In writing for a 6-to-3 majority in the search-warrant case, Justice William H. Rehnquist held that those who issue warrants should use ''common sense'' when judging the credibility of an informer.

In dissent, Justice William J. Brennan stressed that ''everyone shares the court's concern towards the horrors of drug trafficking. But under our Constitution, only measures consistent with the Fourth Amendment may be employed by government to cure this evil.''