Possible Flaw in Metadata API and Source Control. I think you may have found a flaw in the development cycle your using and the Metadata API when it comes to Connected Apps. You quite rightly want all your components in source control, yet clearly some attributes of those components cannot be captured and reapplied.
The Need to Package Connected Apps. Yet you do want the benefit of having the connected app definition in your package for admins to control access within in the subscriber org. The only workaround I can think is to remove from the XML the information (consumer key etc) you cannot write to, which will permit the Connected App to deploy in the packaging org, this will generate a new consumer key which you should ensure becomes the new one you use in your client apps.
Trouble Ahead... That said, I do think that it will probably block other developers from deploying to their orgs from trunk, as the Connected App name has to be unique throughout the Salesforce service and you will have already created one in your packaging org.
Final conclusion... Bottom line is sadly I think the way in which the Connected App's scope spreads beyond the single org it is created in, in terms its name and the unique consumer key. Its not really an artefact that fits well within a source control system. Thus you may have to concede to having this managed solely in your packaging org and not in trunk at all.