So I don't understand a few things, so I'll restate some things I know or believe and will move from there.
- You don't need a package for a Connected App to be used with OAUTH flow/REST API access. I know this, I've done it.
- You do need a package if you want IP Address restrictions to be relaxed (the admin can install it, and then change the settings). Changes to IP address restrictions settings made on the "original" DE Connected App are only propagated via package (and probably that's best left to the admin instead of the package.
- You do need a package if you're gonna package custom objects, visual force/apex stuff, widgets, layouts, and permission sets. Or if you want the admin to be able to flip between self-authorization and permission set acccess to the app
- You don't need a package for Canvas, but on the other hand using a package to distribute some widgets like tabs, buttons, etc, makes it easier for the admin, maybe.
- You need a package to get whitelisted for Group/Professional edition rest api access.
Do I have the right of it?