For years, as the party struggled to shape its modern identity, commentators spoke of the Democrats’ dilemma. Now, as the second decade of the 21st century arches to a close, it is increasingly clear the Democrats have a dilemma in two dimensions.

The first dimension is the drive Nancy Pelosi of California is mounting to return to the speakership of the House of Representatives now that the Democrats have control of the chamber. She won the endorsement of her party caucus recently but the defection of 32 Democrats means her fight won’t be over until the final balloting on Jan. 3.

At the same time, the Democrats are trying to determine whether they should select a 2020 presidential candidate who leans left or who hews to the centre. This question is central to the party’s effort to defeat President Donald Trump’s campaign to win a second term 23 months from now.

The party has an enormous amount to sort out.

The fight over the House leadership is a generational struggle that pits veteran political insiders – the top three likely Democratic leaders are aged 78, 79 and 78, a far cry from the party’s youthful image of the mid-1970s – against novice outsiders. By contrast, the fight over the party’s profile in the next presidential election is an ideological struggle. And taken together, they underscore a fundamental but almost always misunderstood or utterly ignored aspect of American politics: the deep divide between congressional politics and national politics.

The result is a party that – though it has reclaimed one half of Congress – remains in the political wilderness, confused about its mission and unsure of its direction.

“The congressional Democratic caucus has to deal with very different constituencies and very different pressures,” said Stephen Farnsworth, a political scientist who heads the Center for Leadership & Media Studies at the University of Mary Washington. “The result is a message that is not coherent enough to take to the campaign trail. The Democrats can pass legislation that can define the party, like Obamacare or the ‘cap-and-trade’ environmental bill, but that is much different from a presidential candidate going from a group of activists in one state to a group of activists in another state in pursuit of the presidential nomination.”

When American political parties try to square the circle – when they nominate lawmakers for president – the result is often disastrous. Senators Bob Dole of Kansas (GOP nominee, 1996), John Kerry of Massachusetts (Democratic nominee, 2004), and John McCain (Republican nominee, 2008) all failed to win the presidency. And in the two and a half centuries of American national life, only three senators have gone directly from the Senate to the White House: Warren G, Harding (1920), John F. Kennedy (1960), and Barack Obama (2008).

“It’s very hard to run from the Senate,” said Mr. Dole, regarded as one of the premier legislators of the 20th century but who was defeated by Bill Clinton. “You have real duties as a senator, making it particularly hard to campaign. And you have a big record for people to pick apart. It’s going to be hard for those Democratic senators to do that.”

A number of Democratic senators, including Bob Casey of Pennsylvania, Kamala Harris of California, Amy Klobuchar of Minnesota, Kirsten Gillibrand of New York, and Cory Booker of New Jersey are contemplating presidential campaigns, along with Senator Bernie Sanders of Vermont, who ran in 2016. Representative Beto O’Rourke, who lost a close Senate race last month, also is considering a White House campaign.

Meanwhile, Ms. Pelosi, who needs 218 votes to regain the speakership, won only 203 in her own caucus. No Republican will vote for her next month, even as a mischievous gambit to bring to power a politician whom the GOP used, often successfully, as a target of opprobrium.

Professional political figures are reluctant to attack Ms. Pelosi – if she becomes speaker she will have a forum for retribution that she may not hesitate to employ – but clearly there are established voices in the Democratic Party who want to take a sharp turn away from the past.

“Democrats will need new leaders with fresh ideas to preserve their fragile House majority and build a big tent coalition that can send Trump Republicans packing in 2020,” said Will Marshall, president of the Progressive Policy Institute, regarded as a moderate think tank. PPI has praised some of the Democratic rebels, including Representative Seth Moulton of Massachusetts, the leader of the anti-Pelosi strain in the party.

“Our request was, and has always been, simple,” Mr. Moulton, 40, said after the first round of voting. “Produce a meaningful plan for a leadership transition, as [Pelosi] promised in the summer, to allow a new generation of leadership to step forward.” He says he is still waiting.

Gary Hart, the two-time Democratic presidential contender whose 1988 campaign is the subject of the new film The Front Runner, was at the centre of an earlier generational, stylistic and policy struggle within the party. He said the Democrats’ greatest moments have been when they have been the party of inclusiveness and experimentation and he worries the potential class of White House contestants may not share that conviction or commitment.

“Times change and policies must adapt to those changing times,” Mr. Hart said. “But those changing policies must still benefit the maximum number of Americans, including especially those in need.”

How to do that, and with whom, is the question that bedevils the Democrats, and the answer could shape the next decade of American politics.