What kind of update of app on appexchange does trigger security review of app? For instance, if I change field of custom object to be unique, will this require new security review?
Please check the FAQ guide below
As per the FAQ guide published by SFDC
Security review is a periodic, point-in-time review at an interval determined by salesforce.com (typically anywhere between 6 months to 2 years).
When you upload a new package version to the AppExchange and attempt to associate it with your listing, we automatically run a source code analysis against your Force.com code to identify potential security vulnerabilities.
If issues are identified, you will receive a report via email and will be requested to address issues immediately.
We reserve the right to conduct random security penetration tests on your application throughout the year.
This is covered within the listing fee that you pay annually and there is no extra charge. However, if we find that you have deviated from our security standards and best practices we may remove your application from the AppExchange.