Beji Caid Essebsi ran for president on a promise to eradicate terrorism and restore security.

The message resonated with voters, driving him to victory in the runoff on Sunday in Tunisia’s first presidential election since independence in 1956, a contest that international observers deemed free and fair.

But some supporters of his rival, Moncef Marzouki, and activists say the president-elect’s emphasis on security and state prestige is a sign that hard-won freedoms of the past few years will be rolled back.

“I’m expecting more repression in the name of fighting terrorism,” said Alaa Talbi, executive director of the Tunisian Forum for Economic and Social Rights.

Mr. Essebsi said during his campaign that his opponent had been mostly supported by Islamists, including radicals waging a low-level insurgency in the border regions. He didn’t offer evidence to back up his claim, and Mr. Marzouki denied the accusation.

The accusation was emblematic of his successful campaign, in which the 88-year-old politician called on his background as a minister under the authoritarian regimes of Tunisia’s first president, Habib Bourguiba, and longtime dictator Zine El Abidine Ben Ali.

Mustapha Benahmed, an executive board member and parliament member from Mr. Essebsi’s party Nida Tunis, said his party didn’t intend to revive the repressive policies of Mr. Ben Ali. He said Mr. Essebsi and Nida Tunis are uniquely positioned to manage Tunisia after years of political instability.

“We are a party of statesman, not a party of the old state,” he said.

Tunisia is the only country to emerge from the Arab Spring with a largely peaceful democratic transition. The country, however, faces a faltering economy and a lack of reforms of security and judicial authorities. It also faces a regional threat from radical Islamists that has preoccupied the public and generated more tolerance of autocratic practices in the Arab world.

In Egypt, a democratically elected Islamist president was ousted by the military in a popularly backed coup, largely driven by a media campaign that equated political Islam with radical insurgency. The country is now led by a Abdel Fattah Al Sisi, who served under the old, autocratic regime of Hosni Mubarak and promised to restore security and stability.

While Tunisia’s military has stayed out of politics, the media has linked the insurgency to the nation’s moderate Islamist party—an accusation the party denies.

Tunisia, a country of 11 million has exported the largest number of fighters to extremist groups in Iraq and Syria, about 3,000 young men who have joined the ranks of Islamic State and other radical groups, independent researchers and the country’s interior ministry say.

The fall of longtime dictator Ben Ali in 2011 unleashed the wave of radical Islamist unrest along the nation’s borders with Algeria and Libya, two nations that have struggled with powerful, violent extremists.

The Brussels-based International Crisis Group and other political analysts say the Tunisian radicals have allied with drug and weapons traffickers, leading to deadly attacks on security forces that have unnerved the public.

“We are concerned that Essebsi will duplicate the old regime in a gentler way with superficial reforms,” said Ghazi Mrabet, a prominent civil rights attorney who has represented artists and activists charged with crimes associated with their criticism of government and police.

Mr. Essebsi and Nida Tunis now dominate the emerging democracy. Amna Guellali, Tunisia director for Human Rights Watch, warns that grip on power could lead to the return of a police state in the name of fighting terrorism.

“[Nida Tunis] has at the heart of its rhetoric the re-establishment of the security of the state and we risk the attempt to re-establish the police state,” she said.

Radical Islamist terrorism played a significant role in the presidential vote. Many voters said their main concern was that fragile security and political instability would distract the government from focusing on improving the economy.

In campaign speeches, Mr. Essebsi linked security with restoring economic stability, a message that resonated with the nearly 56% of voters who elected him and expressed deep frustrations over Tunisia’s halting transition.

Nida Tunis politicians, including Mr. Essebsi, have said they favor liberalizing the economy and encouraging foreign investment and trade, particularly with Europe. So far, they have offered few details.

“All this is guaranteed through policies that ensure security and eradicate terrorism, which is the biggest threat to our ambitious project,” said Mr. Benahmed of Nida Tunis.

Nida Tunis won a plurality in parliament in October elections by emphasizing the same themes.

Ennahda, the moderate Islamist party unseated by Nida Tunis in October parliamentary elections, has shown little willingness to play the role of opposition. This is something civil society groups such as Mr. Talbi’s fear will make it harder for them to push for reforms to address the grievances that drove the popular uprising leading to Mr. Ben Ali’s fall.

With Mr. Essebsi’s win, those groups are preparing to wage battles to lobby for legislation that will fundamentally change the way the security sector operates, aiming to curb its powers.

Though there have been basic reforms in police training since 2011, Ms. Guellali of Human Rights Watch said there is evidence police are using outlawed interrogation practices against suspects who have been accused of low-level crimes.

“It is very clear in Tunisia that there is a popular endorsement for the use of torture for people who are allegedly linked with terrorism,” she said.

Two people died in police custody after being accused of petty crimes this year. The interior ministry has denied torture is systemic but acknowledged its use by individual officers in violation of the law.

An interior ministry spokesman didn't return calls seeking comment.

The International Crisis Group said in an October report that “sensationalist media” along with antiterrorism rhetoric by government officials “is recasting the anxieties of the educated middle class toward a fear of religious extremism.”

Ms. Guellali and other critics say the incoming government hasn't offered solutions for the social ills that they believe drive extremism and a willingness to rewrite laws that allow police to use low-level crimes against citizens to punish dissent.

Among those is a charge of “attacking a civil servant”—a catchall crime that has been used by police in dozens of cases against activists and people who have resisted routine detentions.