A far-right party that blames Muslim immigrants for social ills won seats in Sweden's parliament for the first time on Sunday, marking the latest advance of anti-immigrant populism in Europe.

With votes counted from 99% of Swedish districts, the Sweden Democrats had won 5.7%, clearing the 4% threshold needed to enter Parliament and resulting in 20 seats in the 349-seat legislature.

The result is a shock for Sweden's political elites and many ordinary Swedes, who have long prided themselves on being one of the Western world's most tolerant and open societies.

"Today we have together written political history," said Jimmie Akesson, the 31-year-old leader of the Sweden Democrats.

The Sweden Democrats' populist campaign against immigration, particularly of Muslims, has underscored the spread of a pan-European backlash against liberal immigration policies, which is increasingly rattling the region's political establishment.

Swedish Prime Minister Fredrik Reinfeldt four-party Alliance won 172 seats, below the 175 needed to form a majority government and opening the possibility of minority rule after the left-leaning three-party Red-Green coalition won 157 seats.

Mr. Reinfeldt is likely to remain in office, but will need support from left-leaning opposition lawmakers to pass laws. Mr. Reinfeldt has said he would refuse to cooperate with the Sweden Democrats.

Across Europe, radical parties exploiting xenophobia are gaining votes and influence, profiting from economic uncertainty in the wake of a deep recession and tapping fears that Muslims and other minorities cause crime, terrorism and the erosion of national identity.

The trend risks deepening Europe's social tensions and the alienation of many minorities, while making it harder for policy makers to argue in favor of immigration—something Europe needs more of to help sustain its economy as its native populations age, according to economists.

"In the long run we need more labor force, and Europe is already not very successful at taking in the labor it needs," said Björn von Sydow, a lawmaker for Sweden's left-leaning Social Democrats and political scientist at Stockholm University.

Rising anti-immigrant sentiment across Europe "is related to the economic downturn, and to a sense of sharpened competition for jobs and for a fair share of public spending," he said.

"Sweden has been quite a homogeneous country. Previously immigrants were only from Europe, but now lots of immigrants have come from Africa and Asia, and there is fear of what one is unused to," said Alf Svensson, former leader of the center-right Christian Democrats.

The Sweden Democrats, like some other far-right groups in Europe, have cast off neo-fascist roots and open racism in favor of a more mainstream image, in a quest to emulate the success of anti-immigration populists in neighboring Denmark, the Netherlands, Austria and elsewhere. The party denies it is xenophobic and says it wants a "responsible immigration policy."

An online campaign advertisement sheds light on what the party means by that phrase. A horde of burqa-clad Muslim women are shown racing against a Swedish pensioner for government money. The stroller-pushing Muslims pull a brake marked "pensions" before the retiree can pull a brake marked "immigration."

A Swedish television broadcaster refused to air the spot, prompting the Sweden Democrats to cast themselves as victims of political correctness.

Many Swedish voters who reject the Sweden Democrats see their rise as a symptom of economic gloom and joblessness, especially in relatively poor parts of southern Sweden.

"It is normal people, average people are voting for them," said Emilia Borg, a 28-year-old production manager from Gothenburg. "When the economic situation is hard they blame it on immigration."

Gaining seats in Parliament will give the party a platform to "raise their questions in a whole new way," Ms. Borg fears.

Sweden's paternalistic welfare state is partly to blame for some immigrants' marginal status in the economy, said Mr. Svensson. "We had...a system which was 'taking care' of immigrants, which didn't give them a chance to flex their own wings and show what they could do, and this has made integation worse," he said.

Europe has had far-right fringe parties throughout the postwar era, but their clout has been growing steadily since the 1980s, when economic sclerosis created new opportunities for charismatic populists, such as Jean-Marie Le Pen in France and the late Jörg Haider in Austria, to blame immigrants for unemployment and crime.

"The new phenomenon, which goes beyond the classical far right, is that people with a more liberal background see immigrants from Muslim countries as a threat to their lifestyle, and to liberal achievements in Western countries," said Cem Özdemir, chairman of Germany's Green party and the son of Turkish immigrants.

Mr. Özdemir rejects the idea of a "Muslim population" that lumps millions of Europeans together with proponents of hate, saying: "The fact that I'm born into a Muslim family doesn't mean I have anything in common with some reactionary." But established parties have neglected voters' angst about the poor integration of some immigrants into European society, concerns that can't simply be dismissed as racist, Mr. Özdemir said.