Four heritage-listed defence sites, including Sydney’s Victoria Barracks, will remain in public hands after the Albanese government yielded to public backlash against their sale.

The sites, along with 63 others around the country, were identified in an independent audit as being underutilised by defence, not fit for purpose and costly to maintain, and were earmarked for sale to recoup $3bn in the federal budget.

But the plan to sell off the heritage-listed sites drew fierce opposition from public campaign group Save Victoria Barracks Sydney, veterans and high-profile politicians including former prime ministers John Howard and Paul Keating.

Ahead of an important cabinet meeting on the budget on Monday, the Albanese government confirmed on Sunday that four sites – Victoria Barracks in Sydney and Melbourne, and Fort Queenscliff and RAAF Base Point Cook in Victoria – will remain publicly owned and their historic elements preserved for community use after they are vacated by defence personnel.

The prime minister, Anthony Albanese, said Victoria Barracks Sydney held a “special place in our nation’s history” while the defence minister, Richard Marles, said the military needed to focus its spending on keeping people safe.

“Given the strategic landscape we face, we cannot afford to be wasting money. We need to look at how we best use defence money for defence capability,” Marles said.

“By moving personnel off these sites, and working to open them up for the public we are ensuring that the history of Victoria Barracks is one that can be protected and returned to all Australians.”

The government will work with the Australian War Memorial once the sites are free to transform the sites into public spaces that tell the history of Australia’s military.

Sydney’s Victoria Barracks was built between 1840 and 1848 and was originally occupied by the British army. It is also the site of the city’s oldest wine cellar.

The shadow defence minister, James Paterson, said questions remain about whether the military should have a presence and capability in Melbourne and Sydney.

“While this is a win for Australia’s defence heritage, it is only the beginning. The Coalition will continue to fight to save other critical defence sites, including HMAS Penguin, Lancer Barracks, Irwin Barracks, and more,” he said.

The NSW opposition leader, Kellie Sloane, said the site was an “irreplaceable part of our military heritage that deserves to be protected for future generations”.

“Our military heritage and defence capability must come before a short-term cash grab,” she said.

The multi-year audit, completed in 2023, was tasked with identifying surplus and costly property holdings, often costing millions to maintain.

The audit found underutilised sites are “draining resources from higher priority needs” for defence.

“Today’s estate footprint comprises numerous legacy sites without a clear ongoing link to current or future capabilities. Urgent interventions are needed to correct the unsustainable trajectory that has resulted from decades of deferred decisions on contentious estate issues,” it said.

The sales of the Sydney and Melbourne Victoria Barracks were expected to raise $1.3bn, given their prime locations in the biggest capital cities.

Up to $2.4bn was expected to be raised from the sale of 26 major metropolitan sites, saving about $3bn in upkeep and security costs over 10 years. These include sites in Randwick, Sydney; Sandringham and St Kilda in Melbourne; Petrie Terrace in Brisbane; and Fremantle in Western Australia.

The government hopes the sales will free up land for new housing developments with a long-term target of 6,000 new dwellings.

Defence expects personnel will have vacated six sites by the end of the year.

The announcement comes as cabinet’s expenditure review committee, which determines budget decisions, meets on Monday after the official interest rate was lifted to an 18-year high of 4.85% in a bid to combat persistent inflation this week.

The treasurer, Jim Chalmers, said this week higher fuel prices due to the Middle East conflict were the “primary driver” of inflation rather than the government’s budget.