I stumbled upon an interesting limitation, it seems that SOAP API endpoint handles requests larger than 16Kb differently --- whenever body is larger than that request just times out.
We're using 'create' SOAP request to create sObjects. The main reason we're using SOAP is its ability to batch up-to 200 sObjects into one call while still being synchronous (comparing to Bulk API).
When I tried to increase HTTP read/open timeout, then after waiting for 10 minutes or so (a '<= 16KB request' takes just a second), I've received once a response about invalid XML: "Content is not allowed in trailing section.".
Interestingly, I've noticed this behavior for 'Notes' sObject, for 'Attachments' requests larger than 16Kb were handled correctly!
Has someone seen something like that? How does this works together with a 200 objects API limit?
Update: I've tried this on 2 different instances: one is Developer Edition and the other is Professional Edition with enabled API. Could that be the issue?
Just to eliminate any library specifics (I am on Ruby with Savon SOAP library) I tried making a simple post request with body as the same XML/SOAP message and with body size less than 16Kb --- I received some warning about ContentType, but as soon as I hit 16Kb + 1 byte, I started receiving Timeouts.
Update 2: Tried posting big response ~64Kb (several Note objects) with cURL and it actually worked. I will dig deeper, seems like it might be a Ruby specific issue (or something stupid I am (not) doing).
Update 3: This is strange and stupid, but now everything works like a charm. And what worries me, that I haven't changed anything.
Thanks, everybody for help: you gave me ideas how to try making those requests differently.
Update 4: Next day I started to get Timeouts again. I immediately checked if the same request works with cURL and it did. So, I just replaced usage of Ruby stdlib net/http to curb and voila --- no more timeouts.
Quick googling didn't yield any results if this is a known issue. Probably, it is a combination of the way Salesforce SOAP API endpoint responds and some bug in Ruby stdlib (or misconfiguration on my side).