Salesforce App subscription license gives users the ability to access and create up to 10 custom objects (from what I understand). If they have a custom object which has a lookup field to another custom object (which they were not given access to) would this prevent them from using that field?
They won't be able to access that lookup field. You might know already that you can expand custom object limit to 110 by upgrading to Apps Plus (former App Bundle), and further to 2000 with Expansion Pack
I also found a blog post that says you can stack a few Force.com App subscription license to expend the number of Custom objects limit by 10 for each extra license assigned to the user, but I'm not sure how easily this can be done in practice.