Britain's mainstream parties drowning out challengers?

A momentous question hangs over Britain - and American hopes for a firm, nuclear, united Western Europe could well be affected by the answer.

The answer could determine whether the United States can continue to rely on Britain as its closest NATO ally in Europe, or whether this country will become a nonnuclear advocate of a nuclear-free Europe, and quit the European Community as well.

The question is this:

Is the traditional two-party political system here beginning to reassert itself after last year's boom of centrist middle-class support for the new Social Democratic Party?

If it is, then the Labour Party next year, under unilateral nuclear disarmer, Michael Foot, has a much greater chance of becoming Britain's next ruling party than many political analysts had thought.

In a swing of political fortunes, Labour is picking up support in public opinion polls.

Unemployment is at crisis levels of 14 percent (3.2 million) and rising. Despite falling inflation and interest rates, the issue is a powerful political one.

In addition, Labour's annual conference in Blackpool has handed Mr. Foot two public victories which moderates claim weaken the party's hard left wing and improve Labour's image around the country.

If Mr. Foot does win power, he is personally pledged to the position voted by the party conference Sept. 29: to give up British nuclear weapons and to reject US cruise missiles due to be stationed here for NATO over the next few years.

Mr. Foot also repeated in a major speech Sept. 28 that a Labour government would take Britain out of the European Community.

This would also run counter to US policy, which wants a strong and united Europe, containing Britain, as a NATO counterweight to the Warsaw Pact bloc.

The reason the question of a Labour resurgence arises now is largely due to one key development:

The march into the political limelight, banners fluttering, of a number of right-wing trade unions, determined to brush up the party's image and oust Prime Minister Margaret Thatcher.

The moderate unions include the Transport and General Workers' Union with 1.2 million members, the Amalgamated Union of Engineering Workers with about the same number, the electricians' union, some railroad men (but not train drivers, who are militant), white-collar workers of various kinds, and more.

Trade unions want Labour back in power, and Thatcher monetarism and restrain abandoned. They provide the cash and the voting discipline that are the basic strength of the Labour Party. Together, unions and the grass-roots Labour groups and the parliamentary Labour Party have won four of the last six general elections in this country.

Mrs. Thatcher's Conservatives wrested power back in mid-1979 because many voters grew tired of the Labour government's apparent incapacity to stem years of high-wage, low-productivity economic disarray.

Since then, Labour has been plunged into fierce civil war between the moderate left (Mr. Foot), and the hard left (former cabinet minister and, ironically, former peer, Tony Benn - still the most charismatic figure in the party) as well as a small but influential group of grass-roots Marxists known as the Militant Tendency.

The high-water mark of leftist influence came 18 months ago. Mr. Benn and his supporters forced every Labour member of Parliament to submit to a reselection process which grass-roots left-wingers are determined to dominate.

Left-wingers also took away the right of Labour MPs to elect the party leadership. Labour MPs now only have a 30 percent say in that election. Grass-roots groups have 30 percent and trade unions 40 percent, in a complex weighted-vote formula.

Soon afterward, former Labour deputy leader Roy Jenkins launched the Social Democratic Party and took it into alliance with the Liberal Party amid a surge of media coverage and middle-class euphoria.

At one point, opinion polls were showing to 50 percent support for the new party and the new alliance. Out of power since the l920s, the Liberals told each other last year, ''prepare to govern!''

Labour plummeted. Mr. Foot's personal standing in one respected poll slid to an unprecedented 15 percent. Mrs. Thatcher led the nation into the Falklands campaign, and eventual victory.

Labour was being written off.

But the scene looks a lot different now.

The right-wing unions have closed ranks to hand left-wingers like Mr. Benn, Miners' Union leader Arthur Scargill, and the Militant Tendency a series of headlined defeats.By a resounding majority, the Labour conference in Blackpool voted to set up a register of organizations which will exclude the Militant Tendency from affiliation with the party.

What this means is that Mr. Foot has won his battle to expel the militants - actually a small group of activists whose weekly newspaper sells some 30,000 copies. It has managed to obtain only eight nominees for Labour seats at the next election, which will see more than 600 seats contested.

The unions hope the party has begun to lose its chaotic, leftist public image in time to rise up and crush Mrs. Thatcher when the elections come, in all likelihood late next year.

In complex maneuvering, the moderate unions also slapped down Mr. Benn in Blackpool. They ensured that the new party National Executve Committee will have a firm right-wing majority.

Since the committee selects key party committee chairmen, it is being widely predicted that Mr. Benn will lose his key post of the Home Policy Committee, and his ally Eric Heffer will be deprived of the Organization Committee.

The previous week, the Liberal Party held a generally low-key conference in Bournemouth. They had to grapple with polls showing their alliance with the Social Democrats in third place to Tories and Labour, with support hovering between 14 and 23 percent.

They patched up a quarrel with the Social Democrats over which candidates would contest what parliamentary seats. Mr. Jenkins made such a soothing speech to the Liberals that one commentator likened it to ''molten nougat.''

Meanwhile, Mrs. Thatcher has been touring Asia, and is still hammering away at ''the Falklands spirit'' as a sign of Tory resolution and leadership.

Polls show Tories ahead with anything from 42 to 47 percent of public support.

But Labour has climbed back to 30 to 37 percent. Right-wing unions are on the move. Mr. Benn must now fight from a smaller power base. The Militant Tendency will battle on, but with less publicity.

Suddenly, the antinuclear, anti-European Community Mr. Foot looms larger than before.