The federal government has decided against an 11th-hour intervention to halt construction of an Olympic stadium and aquatic centre in the heart of Brisbane, in a park that traditional owners say is a First Nations sacred site.

The environment minister, Murray Watt, issued a statement on Sunday afternoon to say he had considered applications made under the Aboriginal and Torres Strait Islander Heritage Protection Act for him to stop construction in Victoria Park.

The application said a “significant Aboriginal area was under serious and imminent threat of injury or desecration”.

Watt said he had decided against making that emergency declaration, but he had appointed a reporter to review further applications and to “determine if longer term protections are required”.

“Today’s decisions follow consultation with interested parties, and I acknowledge the importance of the area to the Turrbal and Yagara Peoples,” Watt said.

As the statement was issued, hundreds of protesters were gathered in Victoria Park on the final day before the 64-hectare site was transferred from trust land held by Brisbane city council to the Games Independent Infrastructure and Coordination Authority (GIICA).

GIICA is responsible for building the 63,000-seat stadium, which will afterwards be used for Australian rules football and cricket.

Sue Bremner, the Save Victoria Park president, said she expected fences to enclose the site would be completed by Monday morning.

When the first fences went up on Friday, five people were arrested from a First Nations protest camp: the Goori Camp Embassy.

“The world was appalled, on Friday, by what they saw,” she told the crowd on Sunday.

Nurri Theresa Williams, who lodged protection applications for the site, said it was “the last sacred site of the gathering of our people in the entire Brisbane area”.

Williams said her family had lived in the area for hundreds of years.

“I’m now in my 80s and this place does have birthing sites – my family’s birthing sites,” she said.

“This park is a beautiful, natural park – our motherland – has burial sites. My family’s burial sites.”

The Greens councillor Seal Chong Wah described “police descending on this Country” as “evil”.

From the other side of the political spectrum, the former Liberal National party premier Campbell Newman said he was “behind the Aunties” and their claims “100%”.

Newman pointed to a sign that said “I preferred Joh!” in reference to his predecessor, Joh Bjelke-Petersen – Queensland’s longest serving premier who was notorious for the “dozing” of heritage buildings in Brisbane.

Newman said the activists who demonstrated against those demolitions were on the right side of history.

“Do you feel like you might be on the right side of history, ladies and gentlemen?” he asked the crowd.

“Because those people were back then, and over the years since we’ve all heard how terrible it was that Brisbane lost such heritage. That is what’s happening here: we are about to lose something precious to an act of barbarism and expedient political calculation.”

The state government and Brisbane city council issued a joined statement saying Victoria Park would “become a world-class destination with incredible new facilities” as well as “revitalised parkland for people to enjoy for generations to come”.

“While we respect the right to peaceful protest, the Victoria Park camp has now become a safety issue, and no one can be in any doubt that these protestors have had ample opportunity to leave the site for their own safety,” the statement read.

“GIICA, Council and Police have repeatedly engaged with protesters at Victoria Park and made clear it will be unsafe for people to remain once the site becomes an active construction zone.”

There was a light police presence at Victoria Park on Sunday and a spokesperson said Queensland police had no immediate plans to evict protesters, referring questions to city council.