As mentioned in order of execution what are "layout-specific rules"? Is it the required values at layout level?
If the request came from a standard UI edit page, Salesforce runs system validation to check the record for:
Compliance with layout-specific rules
Required values at the layout level and field-definition level
Valid field formats
Maximum field length
What this means is that if the request was run from a VF page as opposed to code inside of an APEX trigger or class, any validation rules that would apply to the standard UI page are run against the record. UI pages can have validation rules that go beyond FLS and are specific to the page layout. These rules are run at the point in the Order Of Execution list your question refers to when in the context of a specific page layout.