Home video market has Hollywood twisting and turning

Home video, that rambunctious new kid on the block, has stirred up a hornets' nest in Hollywood. Recent reports indicate these unresolved issues rankling the film and video communities:

* In their search for maximum profits from popular film cassettes, a la ''Star Wars'' and ''Superman II,'' Hollywood studios have lurched from one marketing strategy to another. For video users, the result is confusion - with some movies available for purchase, some for rental, and others not at all.

* War has broken out between studios, who make movies, and video retailers, who market them on cassettes. Video outlets complain of rising overhead and bookkeeping costs as they struggle with shifting demands from production and distribution companies.

* Video dealers are competing more vigorously for rights to market potential hits, to the dismay of theatrical distributors, who prefer to control home-cassette as well as big-screen options.

* Movie theater operators are striving to protect their own turf from a perceived home-video threat. In some cases, showmen reportedly worry that movies are being made specifically for home-video, and released to theaters only as a promotional device. In areas where distributors also operate theaters, friction may grow between film and video factions, as big-screen exhibitors try to hold on to hot properties.

Uncertainties and change are to be expected in a field as young as home video , which didn't begin in earnest until 1978. But each new quake in the video business has serious consequences for the 5,000-8,000 outlets dealing in video software. American Film magazine, published by the American Film Institute, estimates that as many as 90 percent of these are independent firms with limited cash flow. And about half may be out of business within a year.

This could muffle the video boom, which has mushroomed on the strength of numerous, highly competitive dealers and rental fees low enough to rival ticket prices. In order to survive, many outlets have sought to cut overhead and staffing costs, often by turning to outright sales of cassettes instead of time-consuming rental agreements. But this has proved difficult, if not impossible, as studios have encouraged rentals - rather than sales - in an apparent effort to keep control of the entertainment ''product'' they generate.

The problem stems from a Hollywood tradition, writes Seth Goldstein, managing editor of Video Week. With rare exceptions, the big studios have never sold their pictures, preferring to lease them to theaters and TV networks. Upon entering the cassette market, the studios tried to maintain control by dictating terms and to retailers.

As a result, relations between studios and video companies have been uneasy and sometimes turbulent, as both sides jockey for the right to handle cassettes as they choose.

Until recently, Mr. Goldstein reports, the studios were winning their battle for a rental-based system that gave them a continuing share in cassette revenues. Under this setup, a video dealer pays a ''leasing fee'' to the studio, which must be recouped before the retailer starts earning a profit on a given tape. In many cases, the tape is rented only for its first several months on the market, after which it may become available for sale.

Video stores have chafed under this system, which burdens them with paper work and delays profits. Adding to the problem is the fleeting nature of movie popularity. Dealers must stock many cassettes in order to keep up with current fads, and a film may cease to be fashionable before enough rentals have occurred to ring up a dealer profit. The average tape is considered ''exciting'' for about a month, says media specialist Howard Polskin in American Film's VideoFile section. Then it loses most of its luster.

Video dealers have tried alternatives to Hollywood-movie cassettes, with poor results. While there is a limited market for pornography, more respectable options - such as instructional tapes on dance and exercise - have shown little promise. Major dealers may use cassettes as deliberately unprofitable ''loss leaders'' to lure customers into buying expensive video cameras and recorders. But this is slim comfort to the 5,000 outlets dealing in cassettes alone. For them, reports Polskin, the outlook is devastating: Some 3,000 are expected to fail in the next year.

Lately, according to American Film's recent study of the video-movie scene, the whole phenomenon shows signs of fading. Fewer stores are stocking fewer cassettes for rental. Dealers are sticking to the biggest hits and enticing customers to reserve them in advance. In response, studios are now leaning away from rental plans.

The trend away from rentals is a victory for retailers, says Goldstein. But if profits don't pick up, the Hollywood studios could abandon the video business altogether, squelching the highly touted future of prerecorded cassettes. Viewers would be back to taping ''Star Wars'' sequels off their TVs, pushing buttons to dodge commercials.

This would be ironic, since the home-video market is just becoming a major element in the financing of new films. Variety, the entertainment newspaper, reports that some producers now regard the sale of advance video rights as a promising new source of funds.

Though theatrical distributors prefer to acquire those rights for themselves, partly to protect the traditional system of marketing films in theatrical settings, video companies are increasingly eager to ''buy in early,'' and producers are tempted by the prospect of ''up-front'' cash to pay production costs. Distributors and exhibitors, it is felt, will have to swallow their displeasure and buy the theatrical rights anyway, since big new movies are the foundation of their business.

The need for new financing sources, writes film analyst Roger Watkins, is being heightened internationally by many factors, including an upswing in costs, a decline in box-office revenues, a hardening dollar, a shortage of attractive new pictures, and the fact that retail video is outpacing theatrical film in many areas. In a dramatic example from Britain, the lurid ''Southern Comfort'' pulled in more profit during 10 days of video release than during 18 months on theater screens.

''The video tail is wagging the big-screen dog,'' says Mr. Watkins, quoting one producer who claims his company could now fund a low-budget movie from home-video presales revenue. Still, problems exist - including the debate over how long theaters should have exclusive rights to a movie before it becomes available for home use. If a film goes to cassette form too soon, theaters fear they will lose profits. If it is held away from the cassette market too long, though, home-video interests fear the picture could dwindle in appeal, and more time would be available for illegal pirating activity to occur.

Meanwhile, video customers and companies are becoming more selective, shifting their attention from "schlock" to prestige pictures. By taking a part in film financing from an early stage, video maguls could have more sway over what types of movies go into prodcution in the first place, and how they are eventually marketed.