When I add a custom button for use in lightning and set the Behavior to 'Display in New Window', I was expecting it to open in a new tab/window. But when I click it it opens in the current window. This happens for both VisualForce and ULR content sources. Am I doing something wrong?
No, it's a known issue. Until it's fixed you'll need to try alternative routes.
Custom Buttons for VisualForce pages set to "Display in New Window" should open in a New Window
Summary [Lightning Experience] Custom Buttons for VisualForce pages set to "Display in New Window" open within the same tab. Expected: The custom button for a VisualForce page should open in a new window.
Repro This can be repro in any object detail page in Lightning Experience. Here I use Opp as an example.
- Create a Custom Button in an Opp to a VisualForce page and set its behavior to display in a new window
- Add the Custom Button to the Object Page Layout
- Open an opportunity record in SFX and click on DETAILS
- Find the custom button you created (should be in the page level actions bar/drop-down) and click on it.