I have a custom page with controller in my Salesforce1 application on a mobile device. And I'd like to open another application on the same device and pass some parameters to this application. Is there an option to design this scenario? Maybe it's very complicated, so any ideas/approaches are welcome.
Yes, you can put a link on your VF page using the target app's URI scheme. For example,
skype://call?415-555-1212 will open the Skype app and start a call.
Not every app has a URI scheme exposed, and those that do may not accept arguments. All of this depends very much on which app you're trying to open.
However, because you're running this in a VF page inside SF1, you won't have access to the native device APIs that would let you detect if the target app is even installed in the first place. If the user doesn't have the target app installed and taps your link, nothing will happen.