To access the REST api from an external app you need to configure the account for remote access, get a consumer id etc.
Why does the Chatter Desktop app not require this?
Presumably it is just hard-coded somewhere or is it doing something special?
All Remote Access Apps (or, as they are known from Spring '13 on, Connected Apps) must indeed be configured, but only once, by the app developer. You can use an app configured in one (typically Developer Edition) org in any other ('target') org.
It's a common misconception that you need to create a Remote Access App record in the target org, and copy that client id into the app, but this is not the case. Configure once, use anywhere.
From Spring '13 creating a Connected App gives you an install URL, as well as the client id and secret. Connected Apps can still be used in any target org as before, but, if you do use the install URL to add an app to a target org, you can get more control. For example, you can authorize access to the app via profiles and permission sets, widen the IP-range for that app, or, for a mobile app, specify that the app uses PIN-locking.
Given that any app can be used in any org, there is also the question of API access. Apps access Salesforce via the APIs, and can only be used by users with API access. So, how does Chatter Desktop work even for users without API access? Well, API access can be enabled for approved apps, Chatter Desktop being one. I believe that ISVs can have apps in the AppExchange similarly approved.