Math Puzzle: Father and Son

My interpretation of this word puzzle is that "the man" in reference is the speaker's son.

The way I looked at this is by parsing the sentence into two, with the conjunction used as a separator. The first part says the speaker (let's say A) has no brothers or sisters. The second part says there is a man, and that man's father equals my father's son.

I then parsed this into two phrases ("That man's father" and "My father's son")  with the "is" acting as an equal sign. From the second phrase, I deduced that it means the same as A's father's son, which equates to A. Hence, the first phrase equals. A, and so that man's father is the speaker himself.

What I found interesting in the end is that the first sentence has no effect on the second sentence. Removing either would not change the relationship deduced in the second part.