Are sandboxes slower than production Orgs?

Question

I've noticed quite often that sandbox instances are showing a slower performance during Metadata API and Tooling API access. Today this happened on cs14.

The force.com IDE froze completely during a "Metadata Refresh" and CodeFusion took about 100 to 400 seconds to load a single apex class. In CodeFusion I can monitor exactly what happens: the Metadata retrieval is queued and therefore it will be checked after a threshold again and again until it is finally retrieved. Typically that happens within less than 3 seconds. As far as I know, on force.com IDE (and other IDEs) something very similar happens. Switching between Tooling and Metadata API doesn't change much - if one is slow, the other one is slow either and vice versa.

I know these performance issues from other Sandbox instances (mostly cs17 and cs18 where I struggled into it last year several times). On the other hand I'm working a lot on developer orgs running on production instances (mostly on eu0-eu5 and na9 in my case) where it's nearly always fast or very fast.

The issues last year where sometimes visible on https://trust.salesforce.com/trust/status/ as yellow exclamation mark - but not always.

Now I can't see anything on cs14 - however the issue is still present.

I have several questions:

• How this transparency page works exactly? Will the entries be generated by an automatic reporting and monitoring mechanism or are they based on customer reports about issues?
  • If so, is there an easy way to report performance issues?
• How high are the thresholds until performance issues will be reported there?
• Is there the same quality of service level for sandboxes and for production orgs? Or should I simply expect a somehow lower performance on Sandboxes?
• How much slower should I consider as too slow? (In my opinion > 200 seconds to retrieve Metadata which typically comes in 3 seconds it too slow)

Also for huge SOQL queries it feels way slower to work on sandboxes than production orgs, however I did not track this detailed in the logs.

• Is there any documentation on performance differences between sandboxes and production orgs?
• has anyone of you shared the experience of slow sandboxes?
• any idea on how to help to get this improved or reflected on trust.salesforce.com?

Answer 1

Also, use the staging environment to perform stress and performance testing. Note that because the hardware for sandbox in the Force.com platform differs from the production organization hardware, the results of performance testing on sandbox might not match those in production. Nonetheless, performance testing on the staging sandbox can still help catch performance issues and errors that arise from Apex governor limits.

Source: https://www.salesforce.com/us/developer/docs/dev_lifecycle/salesforce_development_lifecycle.pdf

Answer 2

I think we have to split my mixup from above again.

a) General Performance Differences

The general and SOQL issues seem to be explained very good by the answers of @Jagular and @jpmonette. Salesforce is using different hardware and a there is a different usage degree.

So as a conclusion if we need to fine-tune and measure the "real" performance, we could not relay Sandboxes but would require an additional non-sandbox staging environment in form of a DE or a second EE having same metadata and data and which resides ideally on the same pod. That could be possible but come with huge effort and costs.

b) Temporary Tooling/Metadata API Performance Issues

• This seems to be very specific to certain Pods and times
• Looks like bugs
• Some of them are reported on https://trust.salesforce.com/trust/status/ - but not all of them
• The risk of getting these on sandboxes seems to be higher than on production (but it's hard to proof)
• if we could get more insights on this category, especially on how to report or handle them efficiently without premier support, it would be great! I got for hours a performance like this while it should be more like that right now it's randomly fast and slow...

Answer 3

I've had the opposite experience. It was about a year ago. I have a full data sandbox where a particular set of SOQL queries were running much faster in the sandbox(cs4) than in production(ssl/na0). The problem occurred in a Visualforce page backed by Apex code that potentially accesses a large amount of data.

The Visualforce page and associated Apex code were created for us by a Salesforce MVP. He worked with several levels of Salesforce support to restructure the queries. This reduced the query time, but did not change the performance difference between the sandbox and production. He was told during this process that Saleforce makes no specific performance promises with regard to production versus sandbox performance. Sandboxes may generally be slower as crmprogdev notes, but not always.

Fortunately, the people with the worst delays (admins with access to all data) are not usually the people using this new VF page. The usual users see much less data and thus don't see a large delay. I have not re-checked the performance of the query in production vs sandbox because performance is adequate for the normal user.