Ivory was the son of unknown parents. The date and place of his birth have not been found.
His wife was Rochelle Anderson Hall. They were married, but the date and place have not been found. Their only known child was Summer Patrice Hall ().