Jacinta Allan has dismissed leadership speculation and says she will continue to call out “sexist, misogynist, hateful” commentary directed at her, despite the One Nation leader Pauline Hanson’s advice to “suck it up, sweetheart”.

On Sunday, the Victorian premier criticised a truck-mounted billboard, which has been travelling around Melbourne for several weeks, featuring AI-generated images of her wearing a black pointed hat alongside the phrase “Ditch the witch”.

Julia Gillard – who was subjected to the phrase during her tenure as prime minister, Anthony Albanese and the Victoria opposition leader, Jess Wilson, also condemned the billboard.

But Hanson told Sky News on Monday morning she had been called a witch “long before” Allan.

Later that evening, she doubled down on her criticism of the premier, saying it was “no wonder they called her a witch”, given growing crime and debt and allegations of corruption in the construction union in Victoria.

“Probably stronger words might be used for her, but they feel the sooner she’s gone, the better,” Hanson told Sky News. “Jacinta – if the shoe fits, then wear it.”

Speaking at a press conference in Melbourne on Tuesday, Allan said: “Pauline Hanson chooses to barrack for the bullies. I choose to fight them.”

Allan said the billboards deliberately chose “language and visual imagery that speaks to centuries old sexism against women” and she was grateful to see Gillard and others calling it out.

“It made my heart sing to see Julia’s response, and also to see the response from so many other women and men in our community, because we’ve got to draw the line,” Allan said.

“There’s too much division, too much hate. I don’t have a problem if people have a different view – that’s democracy – but what is wrong is when difference is used, is weaponised in a sexist, misogynist, hateful way, and I’ll always call that out.”

Allan said she had seen reports that Labor MPs had revived conversations about a possible leadership spill against her but dismissed it as “navel gazing” by anonymous sources at a time most Victorians were focused on cost-of-living pressures.

“I am absolutely determined to see this through, because when the going gets tough, the tough get going, and I am all in for this challenge,” she said of her leadership.

She added that she didn’t need polls to tell her the “old rules of politics have changed”.

A Freshwater Strategy poll conducted for the Herald Sun, published on Tuesday, showed Victorian Labor’s primary vote has slumped to 23%, compared with the Coalition’s 27% and One Nation’s 25%.

“One Nation is cannibalising the Liberal party vote, the National party vote, and I will also acknowledge it is taking a slice out of the Labor vote as well,” Allan said.

“We have to see that, hear that, understand that, keep listening to Victorians, but more importantly, take action to help people right now with those pressures that are real.”

The poll also showed Allan’s personal popularity had dropped five points for a net favourability rating of minus 37 points – well behind Wilson, who has a positive net rating of 15. Sixty-two per cent of respondents said Allan should be replaced as Labor leader before the election.