I am using metadata API to retrieve the LastModifiedBy of an Object. But, it returns me only one User.
I wanted to know if we can track, the Users whoever modified the Object. Is it possible by any means?
I think you're looking for the Setup Audit Trail. This setup screen should display the last 20 metadata modifications, the last 6 months can be downloaded as csv.
Update: Apparently this data is not exposed/supported by the Metadata API (or any other). There is an idea on the IdeaExchange on which you can vote: Make Audit Setup Trail History available via the API