Originally built as flats, with one flat upstairs and one flat downstairs in each house. An outside staircase gave access to the upper flat. The original name of the building was Bremen Square, which perhaps reflected the ancestory of the owner, Dietrich Martin. It was changed to the more patriotic Nelson Square a year later, at the outbreak of the First World War.

The houses were built for Jabez Bryant, who in 1864 was an employee outfitter. Shortly after, he attempted to set up his own business but was brought before the insolvency court in 1867, and discharged in 1869. Yet he was able to build these two fine houses only four years later. He owned them until 1886 when Henry Martin bought them. Bryant lived in No. 15 from about 1881 until 1886.