If user sets addresses it supercedes anything that was set in host
or port (same as in the native ConnectionFactory).

Fixes gh-524