I am wondering on how the framework understand that the data is stale when we use the annotation cacheable=true. Will the framework looks for a change in the timestamp of each record?
This is explained in the documentation.
If the response has been cached for longer than the refresh time, the storage entry is refreshed. When an application enables storable actions, a refresh time is configured. The refresh time is the duration in seconds before an entry is refreshed in storage. The refresh time is automatically configured in Lightning Experience and the Salesforce mobile app.
Basically, if the response is in the cache, and the refresh time hasn't elapsed, it uses the cached value without a server call. Otherwise, the server is invoked and a new, second callback is called if the values have changed.
This is not developer configurable, and the platform does not know about changes to records that would otherwise have been retrieved. There has been talk about making the platform more aware of changes that occur because of automations, but that's probably some ways out yet. For now, just know that calls to the same method in a short period (I believe it's a few minutes) will result in possibly stale results.
Basically, you should only use
cacheable=true for data you know won't change frequently, because it'll otherwise be unreliable in returning correct information before the refresh timer expires.
For example, custom labels and custom metadata are probably good candidates for
cacheable=true, but Account or Opportunity records probably should probably be returned with
cacheable=false (assuming you're not using Lightning Data Service).
That's another side note: you should prefer Lightning Data Service for retrieving and updating records whenever possible, because it guarantees you'll have the latest data automatically.