I don't think you would be able to achieve the apex:selectRadio tag. Standard Visualforce tags and SLDS don't play well together, because Visualforce won't generate the proper markup structure or apply the proper classes. My suggestion is to use Visualforce to render pure HTML elements, wrapped in the proper SLDS containment structure with appropriate classes. Something like this:
<apex:page id="Test" controller="Test">
<apex:stylesheet value="{!URLFOR($Resource.SLDS, 'assets/styles/salesforce-lightning-design-system.css')}" />
<div class="slds">
<apex:form>
<fieldset class="slds-form-element">
<legend class="slds-form-element__legend slds-form-element__label">Colors</legend>
<div class="slds-form-element__control">
<apex:repeat value="{!colors}" var="color" >
<span class="slds-radio">
<input type="radio" id="color-{!color.value}" name="myFavoriteColor" value="{!color.value}" />
<label class="slds-radio__label" for="color-{!color.value}" onclick="document.getElementById(this.htmlFor).checked=true;">
<span class="slds-radio--faux"></span>
<span class="slds-form-element__label">{!color.label}</span>
</label>
</span>
</apex:repeat>
</div>
</fieldset>
<apex:commandButton value="Submit" action="{!submit}"/>
</apex:form>
<span>{!myFavoriteColor}</span>
</div>
</apex:page>
You should then be able to get the value of your radio group inside your controller's submit handler as a page parameter using ApexPages.currentPage().getParameters('myFavoriteColor'), like so:
public with sharing class Test {
public String myFavoriteColor { get; set; }
public List<SelectOption> colors {
get {
if (colors == null) {
colors = new List<SelectOption>();
colors.add(new SelectOption('red', 'Red'));
colors.add(new SelectOption('green', 'Green'));
colors.add(new SelectOption('blue', 'Blue'));
}
return colors;
}
private set;
}
public PageReference submit() {
myFavoriteColor = ApexPages.currentPage().getParameters().get('myFavoriteColor');
return null;
}
}
Note the onclick handler on the label element. I found this was necessary to get the option to actually be selected when you click on it, because the styles applied to the input radio actually make it unclickable, because it has a height and width of 1px.