One reason may be that Set<SObject> is a risky mechanism to use: equality is based on all the fields (so is expensive) and if fields are changed logic can easily break. Same problem using SObject as a Map key.

But perhaps just because also supporting Set<SObject> adds a bunch of extra methods that need documenting and supporting. (Given that there isn't any common superclass.) And the conversion to list is trivial:

Set<SObject> s = ...;
someDmlOperation(new List<SObject>(s));

One reason may be that Set<SObject> is a risky mechanism to use: equality is based on all the fields (so is expensive) and if fields are changed logic can easily break. Same problem using SObject as a Map key. So not a pattern to be encouraged.

But perhaps just because also supporting Set<SObject> adds a bunch of extra methods that need documenting and supporting. (Given that there isn't any common superclass.) And the conversion to list is trivial:

Set<SObject> s = ...;
someDmlOperation(new List<SObject>(s));