Well, the answer is yes, and no.
There are two ways to connect heroku apps (regardless of the language) and salesforce.
1: the Salesforce Apis. There are a number of them but key api's are probably the Soap Enterprise and the Rest api. These allow for the exchange of data between your salesforce org and external applications (which can be hosted on heroku). You can find out more information on the use of these apis here: http://www.salesforce.com/us/developer/docs/api/index.htm (Soap) and http://www.salesforce.com/us/developer/docs/api_rest/ (Rest)
Canvas would probably require more re-working of your application, than the APIs.
Without knowing more about the java applications you want to deploy to heroku and integrate with salesforce, I can't speak to the specifics of how that would work.