Option 1. You can achieve most of the functionality with only point and click configuration as follows:
- You can create a Unique External Id field which is the concatenation of the AccountId + Rank.
- This external id you need to generate with a Workflow/Flow/Process builder.
How this would work: In the example you provided assuming the Accound Id for Account A= '001abc' (not adding 18 digits for simplicity) the External Id field for Case B will be '001abc1'.
When somebody tries to insert Case D, the External Id will be eventually the same and the system will give you an error like the following
This option does most of the job but doesn't give you many details regarding the conflicting record.
Hint: the user can use this id, paste it accordingly to the url and navigate to the conflicting record
Option 2. Build a custom solution with triggers and an error message which will provide more details to the user
I would go for Option 1 which does not require custom code, is simple to configure and delivers the main requirement.