There isn't a way to modify the behavior of those Detail Page links other than what you can see in the selections when defining the options you provided.
You could create a Visualforce or Lightning Component action instead that does the navigation call and include it in the Action Bar for Salesforce1 to make sure it does not open in a new window.
For example using Visualforce, make a page and assign it to an action that simply does:
When a user launches the action it will flash the Visualforce page briefly as an action modal but then direct the user to the report and stay in the same window. It is always best to use the navigation events designed for Salesforce1 specifically: