"Col. William" Brown was completely cut-off by his brother Tom.  Their brother William's daughters remember that it was not permitted to speak of Jim in Tom's presence. Even Jim's nephew Stewart Brown never knew of his Uncle Jim's existence.

It is speculated that the rift between Jim and Tom may have begun over Jim's dubious military service. Jim claims to have been a member of the Canadian Mounted Rifles and present at the coronation of King Edward VII.  He claims to have fought in the Boer War, having served the entire campaign. All of this is possible. Tom certainly served in the military, but Tom never saw any action.  One theory is that Tom was jealous because Jim saw action and Tom did not. The more likely theory is that Jim's military record is purely fictional and that he combined the identities of his brothers William and Tom to create his sideshow persona.

It is known that Tom was present at the coronation of Edward VII. Was Jim present also, or did he "borrow" that story?  If he did indeed "borrow" part or all of his military record from his brothers William and Tom, then this would certainly have offended his proper public figure of a brother, Tom, the Chief of Police. The real truth may never be known and may be a combination of both theories.

Jim never married, nor had children (that we know of!).