I have custom object X which is look up with Account and custom object Y which is detail of X(master). Now I have to make Y appear in related list section of only those Accounts, whose reference present in X.
Can someone tell me if that is feasible or I should change my approach with a customized Visualforce page?
Unfortunately you can't 'roll-up' related lists like this.
In order to achieve what you want you will either need to override the standard Account detail page or use an inline Visualforce page.
Depending on what you want to do you may be able to achieve what you need using just the standard controller, although as you need to traverse two levels of relationship I don't think it's possible. Otherwise you'll need a Controller Extension as you'll have to use the standard controller to allow you to use your page as an override or inline page.
