I created a manage package which contained a few components, one of them being a permission set on Object Settings for Leads, Accounts, and Contacts. On each I object I made certain fields read/edit, and then added the permission set to my unmanaged package.
Then I imported the package to a secondary dev org for testing, and everything seemed to import successfully. However when I clicked into the permission set that came from the package, the read/edit access I gave for specific fields were unchecked. I made sure this was saved on the org where the package was created.