Email2Case will automatically populate a field on the case record called Web Email (From memory the developername for this is SuppliedEmail)
When you send an email from the Case using the Service Console Email Publisher, Salesforce first looks to see if a contact record is associated and if so will auto populate the To field with the contacts email address (as stored in Contact.Email).
If there is no Contact associated, then the case will automatically populate with the Web Email address.
If this isn't happening, first place to check should be field level security and page layouts to make sure the Web Email field is accessible.
Unfortunately I have found the web email behaviour isn't 100% foolproof, the best approach is to make sure that there is an associated contact with an email.