Officials from the Construction Forestry Mining and Energy Union (CFMEU) found asbestos at the site on Monday.
A stop-work meeting held by workers on Tuesday resolved to stop excavation until an independent hygienist declared the area to be safe.
The 40 construction workers who have been digging foundations on the southern end of the site will instead work on the northern end.
NSW Premier Barry O’Farrell said the site was expected to have problems.
‘What’s happening at Barangaroo is remediation work,’ he said.
‘We know that for decades this was an industrial site that wasn’t cared for in the way that industrial sites are cared for today.
‘So yes, there are some bad things there.’
Mr O’Farrell said workers at the site were being taken care of.
‘I’m convinced though that all of the actions by contractors and the companies to date are in accordance with WorkCover practices and procedures,’ he said.
‘WorkCover is on site today, they’ll make the final arbitration on that.’
Earlier, CFMEU assistant state secretary Rob Kera said developer Lend Lease had previously declared the site to be safe.
‘I believe they handled the situation disgracefully … from a company that says they follow world’s best practice,’ he told AAP on Tuesday.
It’s the second time work has been halted at the high-rise development because of fears about asbestos.
About 150 workers were removed in April after the CFMEU said testing revealed the area was contaminated by toxic chemicals, including asbestos and high levels of lead.