Keir Starmer is hoping to soften the impact of his government’s changes to the immigration system after a backlash from Labour MPs and a dramatic intervention from his former deputy Angela Rayner.

The prime minister is considering exempting large numbers of people from the proposed changes, which would make it harder to achieve settled status in the UK, as he attempts to keep his restive party onboard.

Under the plans, most people would have to wait 10 years to qualify for settled status, rather than the existing five-year period. But proposals included in a government consultation could involve migrants working in the public sector excluded from the changes, as well as those who are on the verge of being settled.

Ministers are now debating how far they want to extend those exemptions but Downing Street said on Wednesday they would not cover everyone who had already arrived in the country, as demanded by Rayner and others.

“In the four years before the election, we saw record levels of immigration,” a spokesperson for the prime minister said on Wednesday.

“In the manifesto, we promised to deliver a fair and properly managed immigration system. We are considering responses to Home Office consultation, and we respond in line with our principles and values.”

Shabana Mahmood, the home secretary, announced the proposals earlier this month as part of a package of measures designed to limit the number of people entering the country.

The plans would make refugee status temporary rather than permanent and the qualification period for indefinite leave to remain doubled to 10 years in most cases.

Mahmood also announced a pilot scheme to pay families whose asylum claims have failed up to £40,000 to leave the country. If they refuse, she said, they would be ejected forcefully, even if that meant handcuffing children.

The home secretary said a key part of the changes to indefinite leave to remain was making sure they applied retrospectively to those who were already in the country, especially to hundreds of thousands of people who entered when Boris Johnson was prime minister.

She said in a speech two weeks ago: “Between 2021 and the 2024 general election, [the previous government] oversaw net migration of 2.5 million.

“Absent action, over the next five years, some 350,000 low-skilled workers and their dependents will qualify for settlement. At that point, they will gain access to welfare, free healthcare and social housing.”

Mahmood has led the government’s attempts to toughen its approach on immigration as it responds to the rise of Nigel Farage’s Reform UK.

But her announcement angered many of her colleagues, who argued that the Green party’s victory at the Gorton and Denton byelection shows Labour faces as much of a threat on its left as its right.

A group of 100 Labour MPs signed a letter opposing the measures when they were announced, arguing: “You don’t win back public confidence in the asylum system by threatening to forcibly remove refugees who have lived here lawfully for 15 or 20 years.”

Sarah Owen, a leader of the centre-left Tribune group, compared the threat of force against children to Donald Trump’s Immigration and Customs Enforcement (ICE).

Rayner echoed many of those criticisms in a speech on Tuesday night to the Mainstream campaign group, during which she criticised the changes to settled status.

“We cannot talk about earning a settlement if we keep moving the goalposts,” she said. “Because moving the goalposts undermines our sense of fair play. It’s un-British.”

Downing Street insisted on Wednesday that the proposals were fair. But government officials said ministers were now poring over the consultation responses to see how widespread the exemptions should be.

As part of the consultation, Mahmood is proposing that some people be given faster passage through the system.

Those earning £125,140 for three years, for example, would be able to qualify in only three years, while those working in the public sector would do so after five.

But people who have claimed benefits for less than a year would have to wait 15 years in total, while those who have done so for more than a year would need to wait 20.

On Wednesday afternoon Starmer met black and minority ethnic members of the parliamentary Labour party in Downing Street after MPs pushed for talks amid rising anger over the indefinite leave policy.

The meeting was also attended by the justice secretary, David Lammy. One MP said there had been “a lot of frustration” with Lammy before the talks, with some feeling senior figures were failing to properly listen to concerns being raised privately and publicly.

“There’s a sense the centre just isn’t hearing us, not even on the tone or framing,” they said, adding it “serves no purpose” for Downing Street to alienate its own MPs over an issue so politically and personally sensitive to many in the party.

Another MP was more blunt, saying the approach was flawed from the outset. “It’s always been a poor policy,” they said.