When Islamic State collapses in Iraq, a lot will ride on whether the Iraqi Shiite militias taking part in that campaign will stop at the international border or will cross into Syria and open a new phase of that country’s war.

The Hashed al-Shaabi or Popular Mobilization Forces, which unite several of Iraq’s powerful Shiite militias, were established to combat Islamic State in mid-2014. At the time, the regular Iraqi army collapsed as the extremist group seized the country’s second-largest city of Mosul and advanced all the way to the outskirts of Baghdad, the capital.

While technically under the control of the central government in Baghdad, most of these militias have been trained and armed by Iran and don’t hide their close links with Iran’s Revolutionary Guard Corps. On their tanks and armored vehicles, they often fly banners with portraits of the Islamic Republic’s Supreme Leader Ayatollah Ali Khamenei.

The current campaign to retake Mosul, the last major population center in Iraq still under Islamic State’s sway, is led by the rebuilt Iraqi army and the federal police. The Shiite militias have deployed to areas west of the city, cutting off Islamic State from its remaining territory in Syria and taking a string of towns along the Syrian border.

Some officials of the Popular Mobilization Forces have already said that they won’t stop at the border and that they received the blessing of Syrian President Bashar al-Assad to go after his enemies across the frontier.

Iranian officials, too, have openly spoken about using these battle-hardened Iraqi militias in the Syrian conflict once Mosul is finally freed from Islamic State. While Iraqi forces suffered serious casualties in Mosul, they have already reclaimed a significant part of the city.

“We say clearly, if there is a necessity to secure the Iraqi-Syrian border and to prevent terrorists from coming to Iraq, then we may go to Syria to do so,” said Moeen al-Kadhimi, a senior leader of Badr, the most prominent Iraqi Shiite militia, and former chairman of the Baghdad provincial council.

He cautioned, however, that any such cross-border operation would have to be approved by both the Syrian and Iraqi governments, and that the Iraqi militias’ target would be Islamic State and not the other opponents of the Assad regime or the Syrian Kurdish forces.

“They are not our business. The Syrian Arab Army has proved that it is able to defend Syria.”

In Syria, these militias would find a very different demographic makeup. While Iraq is majority Shiite, some 74% of Syria’s population is Sunni, according to Central Intelligence Agency estimates. Islamic State follows an extreme form of Sunni Islam and has driven non-Sunnis from the lands it controls. Mr. Assad and the core of his regime hail from the minority Alawite community, a distant offshoot of Shiite Islam.

“The entrance of tens of thousands of Popular Mobilization Forces fighters in Syria will have a deep impact and will cause a sectarian ignition,” said Riad Hijab, a former Syrian prime minister who now heads the High Negotiations Committee, an umbrella body representing the mostly Sunni opposition and rebel groups backed by Saudi Arabia, Turkey and Qatar.

“They will come under sectarian mottos and will be provocative as they fight Sunnis. This will affect the stability of the region, and not just the Arab region. It will create sectarian confrontation everywhere Sunnis and Shiites are living,” Mr. Hijab said.

Shiite militias from abroad already play a major role in the Syrian conflict. The most important of them is Lebanon’s Hezbollah, which has become indispensable for the survival of the Assad regime.

Thousands of fighters from Iraqi Shiite militias such as Al Nujaba and Asaib Ahl al Haq—both part of the Popular Mobilization Forces—have also been flown in recent months to regime-held parts of Syria. There, they have taken part in the takeover of the rebel half of the Syrian city of Aleppo, more than 300 miles from Iraq.

An independent overland entry of much larger Iraqi militia units would, however, be cardinally different from an expeditionary mission to assist Syrian regime forces in Aleppo.

While the Assad regime holds large territory encompassing major population centers of western Syria, there is no contiguity between that territory and Iraq. In eastern Syrian regions abutting Iraq, there are only isolated enclaves of regime control that have to be resupplied by air.

Most of eastern Syria is currently held either by Islamic State, also known as Daesh, or by the Kurdish-dominated alliance known as the Syrian Democratic Forces.

Sunni Syrian rebels backed by Turkey and Kurdish-led forces backed by the U.S. are separately advancing toward the biggest city in eastern Syria, Raqqa, Islamic State’s de facto capital.

Injecting Iraq’s Shiite militias into this overwhelmingly Sunni region would be a recipe for disaster, cautioned Monzer Akbik, a senior member of the Tayar al-Ghad Syrian opposition group which is allied with the Kurdish forces.

“Sectarian Shiite militias coming into a Sunni area will ruin everything,” he said. “If that happens, maybe people in the area will join Daesh so they can go and fight against the Hashed al-Shaabi.”