Mr Harvey and his family had been fighting the federal government’s attempts to deport him to New Zealand, where he had no friends, family or prospect of work.
Federal Court of Australia Justice Susan Kenny adjourned the matter in March, and reserved her judgment on Mr Harvey’s case.
On Monday, hours after his death, Mr Tudge told the ABC that Mr Harvey was “a person with a very serious criminal background”.
“That is why he was in a detention facility, because he was being evicted from the country, because he was not an Australian national,” Mr Tudge said.
“He has passed away, sadly, and obviously our condolences to his family.”
Mr Harvey’s elder sister, Rae Harvey, said she was appalled to read of Mr Tudge’s comments.
“I was highly offended and upset by it,” she said. “My brother is a victim of the system that failed him.”
Ms Harvey said her brother had not left Australia since 1980, and maintained the Australian system had let him down.
A fellow detainee said Mr Harvey had appeared grey in the days leading up to his death and was unresponsive earlier on Monday morning, two hours before guards entered his room.
He said Mr Harvey was worried about the prospect of being transferred to Christmas Island detention centre after Border Force and Serco guards told detainees that some may be transferred to the remote island.
Mr Harvey’s mother, Pamela, told The Age she was devastated at the loss of her son, but took small comfort in the thought he would no longer be deported.
Her daughter was making preparations for Mr Harvey’s funeral, including purchasing a double plot for Mr Harvey and his mother.
Through tears, Ms Harvey said of her son: “He gets to stay home now, they can’t take him away.”
Police will prepare a report for the coroner. Mr Harvey’s death is not considered suspicious, and investigators are not treating it as self-harm.
A Border Force spokesperson said the man’s death had been referred to authorities for investigation.
“As this matter will be subject to ongoing investigation, it would not be appropriate to comment further.”