I posted an idea here: Lightning Component Styling on UI Components

Also unpalatable for its own reasons but it works. Will paste it below.

For what it's worth you can see a comprehensive list of all the styles added to the ui:button component here: https://github.com/forcedotcom/aura/blob/master/aura-components/src/main/components/ui/button/buttonFlavors.css

There's one sort of brute-force solution I used to just wipe out these styles, not with ui:button but with another component... I made my own component that does nothing but inherit from the ui element but I set all its custom CSS attributes to "inherit". So for ui:button it would look like this:

neutralButton.cmp:

<aura:component extends="ui:button">
    {!v.body}
</aura:component>

neutralButton.css:

.THIS{
    font-weight: inherit;
    font-size: inherit;
    margin: inherit;
    padding: inherit;
    text-decoration:inherit;
    text-align:inherit;
    border-radius:inherit;
    border:inherit;
    border-top:inherit;
    background:inherit;
    background:inherit;
    box-shadow:inherit;
    text-shadow:inherit;
}

.THIS:hover,
.THIS:focus,
.THIS.is-selected{
    background:inherit;
    background:inherit;
    text-shadow:inherit;
}

.THIS .label{
    white-space:inherit;
    color: inherit;
}
.THIS:hover .label,
.THIS:focus .label{
    color: inherit;
}
.THIS:disabled{
    cursor:inherit;
    background:inherit;
    box-shadow:inherit;
    text-shadow:inherit;
}
.THIS:disabled .label{
    color:inherit;
}
.THIS:disabled .label:hover{
    color:inherit;
}

And then use a c:neutralButton instead of a ui:button and it no longer hijacks the SLDS styles. Still it's a little silly that this ends up being the only reliable way to shut off hard-coded component styles.

I posted an idea here: Lightning Component Styling on UI Components

Also unpalatable for its own reasons but it works. Will paste it below.

For what it's worth you can see a comprehensive list of all the styles added to the ui:button component here: https://github.com/forcedotcom/aura/blob/master/aura-components/src/main/components/ui/button/buttonFlavors.css

There's one sort of brute-force solution I used to just wipe out these styles, not with ui:button but with another component... I made my own component that does nothing but inherit from the ui element but I set all its custom CSS attributes to "inherit". So for ui:button it would look like this:

neutralButton.cmp:

<aura:component extends="ui:button">
    {!v.body}
</aura:component>

neutralButton.css:

.THIS{
    font-weight: inherit;
    font-size: inherit;
    margin: inherit;
    padding: inherit;
    text-decoration:inherit;
    text-align:inherit;
    border-radius:inherit;
    border:inherit;
    border-top:inherit;
    background:inherit;
    background:inherit;
    box-shadow:inherit;
    text-shadow:inherit;
}

.THIS:hover,
.THIS:focus,
.THIS.is-selected{
    background:inherit;
    background:inherit;
    text-shadow:inherit;
}

.THIS .label{
    white-space:inherit;
    color: inherit;
}
.THIS:hover .label,
.THIS:focus .label{
    color: inherit;
}
.THIS:disabled{
    cursor:inherit;
    background:inherit;
    box-shadow:inherit;
    text-shadow:inherit;
}
.THIS:disabled .label{
    color:inherit;
}
.THIS:disabled .label:hover{
    color:inherit;
}

And then use a c:neutralButton instead of a ui:button and it no longer hijacks the SLDS styles. Still it's a little silly that this ends up being the only reliable way to shut off hard-coded component styles.