I have a button called Sync on Order object. I want to sync only those orderlineitems whose IsSynced field has value as 'No'. If there are no such orderlineitems then i want to display a message on Order detail page say 'No more orderlineitems to Sync'.
If you want to send an error message from your controller, you need something like this:
ApexPages.addMessage(new ApexPages.Message(ApexPages.Severity.ERROR, 'my error msg'));
Plus, you need to include
<apex:pageMessages id="messages"/> on your Visualforce page and make sure you rerender it after your action runs.