Jimmy's parents were no ordinary blacks.
His mother, Annie, was the daughter of a house servant working in the homestead of a wealthy squatter, Andrew Brown, Esq., JP.
She was born on his pastoral property, Caigan, on the Castlereagh River, about 10 miles north-east from Mendooran.
In June 1847, Caigan was one of eight pastoral runs in the Bligh county for which Brown had despasturing licences.
Another wealthy squatter, Richard Rouse, held nine runs in the district, including Breelong which was also on the Castlereagh River, 20 miles west of Mendooran.
Brown and Rouse were both know for their benevolence towards indigenous people, although the latter had whipped his convict workers.
They let them live on their properties and gave them food, shelter and work.
This was the very least they could do when they had stolen the land from the Aboriginals and left them without any way to survive on their own.
People who work for important people generally feel important themselves by way of association.
This could have been a reason why both of Jimmy's parents did not like mixing with other Aboriginal people.
When Annie's Irish father had died before her birth, her mother Polly had raised her with a male house servant at Caigan.
The Rouse family claim that Jimmy's father, Tommy, was born on one of their pastoral properties on the Barwon River.
In later life he asked Mr Rouse to act as intermediary for him in his dealings with white officialdom to assist him to get food rations.
When Tommy discovered silver in 1887, he was not allowed to make a claim for it, but instead was given the use of 10 acres of a common at Wyaldra Creek on the northern outskirts of Gulgong.
He built a house for his family, but it appears they only lived there together briefly.
When Tommy was not working, they were forced to move onto Aboriginal reserves where they could get food rations.
There also might be something in the fact that the English surname they adopted was that of the top white official in the land, the governor.
It is said that when Tommy had encountered the real governor of New South Wales, Lord Jersey, at the Mudgee show, he had introduced himself to him as 'the other Governor'.
Tommy had chutzpah!