Note: This is the current behavior, but not a guaranteed result
Not sure if this is an anomaly or not as the sets are unordered but doing this seems to put it back in the order they were added. Also removes a loop.
//Build the set of values
Set<String> tmp = New Set<String>{'C','B',null,'A',null,'Z','D'};
System.debug(tmp);
19:19:42.23 (23673744)|USER_DEBUG|[2]|DEBUG|{null, A, B, C, D, Z}
Notice how they are not in the order they were put in
//remove the null values
tmp.remove(null);
Now we join the remaining values back
System.debug(string.join(New List<String>(tmp),','));
19:19:42.23 (23854080)|USER_DEBUG|[4]|DEBUG|C,B,A,Z,D
Whala - They are back in the order they were initially added and the nulls are removed
So in your case
Set<String> tmp = New Set<String>{
record.Street, record.City, record.State, record.PostalCode, record.Country
};
tmp.remove(null);
String theAddr = string.join(New List<String>(tmp),',');