I've created public groups and sharing rules in order to give access rights to the "Y" profile to A, B and C case record types. The user that has profile Y assigned cannot find, access and view "D" cases record types. However, the profile Y still can search, find and access "D" cases record type, he just cannot edit it. How can I restrict Y profile to find and view "D" cases record type?
You can't really restrict access to view records based on record types from the profile level. if you think of Salesforce as a filing cabinet, each drawer is an object, within the drawer there are folders that group the records. Record Types are those folder. On the profile when you say user has record type x, y or z available to them it only means their ability to create records into that type. It does not change their access.
My friend Justin wrote a nice blog post about it here: http://www.arkusinc.com/archive/2013/deep-diving-into-salesforce-record-types
To answer your question it seems like you would want the object security settings to be private and then do criteria based sharing to share records with type x to public group 123. However, if someone below the user has access to the record with type y and your user is above them in the role hierarchy then they would see that type of record as well.