His father, Patrick Brown, appears to have been Irish, or born in the colony of Irish descent, and his mother was an Aboriginal mother.
In 1858, when he was 19, he married Hannah (Annie) Nicholson, under the Roman Catholic rite at Braidwood.
They had 11 children, six of whom were living at the time of their father's death: Patrick (b.1860), Charles (b. 1862), Thomas (b.1865), Richard (b.1873), John (b. 1875), and Frank (b.1878).
The two deceased girls were Myrtle and Margaret.
When the couple married, Thomas was living at Currawang on the Clyde River at Bateman's Bay and and Annie at Broulee.
He was a servant and she a house servant.