17

I've noticed quite often that sandbox instances are showing a slower performance during Metadata API and Tooling API access. Today this happened on cs14.

The force.com IDE froze completely during a "Metadata Refresh" and CodeFusion took about 100 to 400 seconds to load a single apex class. In CodeFusion I can monitor exactly what happens: the Metadata retrieval is queued and therefore it will be checked after a threshold again and again until it is finally retrieved. Typically that happens within less than 3 seconds. As far as I know, on force.com IDE (and other IDEs) something very similar happens. Switching between Tooling and Metadata API doesn't change much - if one is slow, the other one is slow either and vice versa.

I know these performance issues from other Sandbox instances (mostly cs17 and cs18 where I struggled into it last year several times). On the other hand I'm working a lot on developer orgs running on production instances (mostly on eu0-eu5 and na9 in my case) where it's nearly always fast or very fast.

The issues last year where sometimes visible on https://trust.salesforce.com/trust/status/ as yellow exclamation mark - but not always.

Now I can't see anything on cs14 - however the issue is still present.

I have several questions:

  • How this transparency page works exactly? Will the entries be generated by an automatic reporting and monitoring mechanism or are they based on customer reports about issues?
    • If so, is there an easy way to report performance issues?
  • How high are the thresholds until performance issues will be reported there?
  • Is there the same quality of service level for sandboxes and for production orgs? Or should I simply expect a somehow lower performance on Sandboxes?
  • How much slower should I consider as too slow? (In my opinion > 200 seconds to retrieve Metadata which typically comes in 3 seconds it too slow)

Also for huge SOQL queries it feels way slower to work on sandboxes than production orgs, however I did not track this detailed in the logs.

  • Is there any documentation on performance differences between sandboxes and production orgs?
  • has anyone of you shared the experience of slow sandboxes?
  • any idea on how to help to get this improved or reflected on trust.salesforce.com?
  • 1
    I get premier support email alerts and from my experience alerts in there often are not on the trust site, or at least significantly later than the email alert. – Girbot Mar 20 '15 at 14:00
  • @Girbot - that's what I thought. But if the page is about trust, IMO there should not be more trust for Premier than for the mere mortals. Everyone should be in the scope of trust. Unfortunately most of our clients have EE without Premier. On my recommendation with sandboxes which perform now a bit disappointing. Due to that issues, now one of our Developers went into an early weekend, since it wasn't productive to work on cs14 anymore. I can rule out connection problems or IDE-problems. It clearly drills down to a Sandbox performance issues. – Uwe Heim Mar 20 '15 at 14:16
  • I think you right, I forgot to add to my original comment but I had to wait 48hours for a dev sandbox to be created on CS14. Based on previous experience they normally take a couple of hours max. – Girbot Mar 20 '15 at 14:18
  • In my experience, Production Orgs are always given priority, which makes perfect sense to me. As a consequence, Sandboxes tend to run slower. Sometimes they simply crawl. – crmprogdev Mar 20 '15 at 15:06
  • @crmprogdev - yes, it makes sense to a degree. >100 seconds to save single file is a showstopper. The risk of crawling sandboxes seems to be present. CS14 is still slow now after hours and I'm not sure that the issue has popped up at the right team to handle it yet. A clear report path and more transparency would be of great help. Logging a ticket at the standard support seems not very helpful to me... – Uwe Heim Mar 20 '15 at 15:56
12

Also, use the staging environment to perform stress and performance testing. Note that because the hardware for sandbox in the Force.com platform differs from the production organization hardware, the results of performance testing on sandbox might not match those in production. Nonetheless, performance testing on the staging sandbox can still help catch performance issues and errors that arise from Apex governor limits.

Source: https://www.salesforce.com/us/developer/docs/dev_lifecycle/salesforce_development_lifecycle.pdf

  • thanks @jpmonette your link isn't the one you intended to paste as source I guess. It's the link of this page... – Uwe Heim Mar 20 '15 at 16:35
  • @UweHeim I was quoting myself recursively :)! (Now fixed) – jpmonette Mar 20 '15 at 16:37
5

I think we have to split my mixup from above again.

a) General Performance Differences

The general and SOQL issues seem to be explained very good by the answers of @Jagular and @jpmonette. Salesforce is using different hardware and a there is a different usage degree.

So as a conclusion if we need to fine-tune and measure the "real" performance, we could not relay Sandboxes but would require an additional non-sandbox staging environment in form of a DE or a second EE having same metadata and data and which resides ideally on the same pod. That could be possible but come with huge effort and costs.

b) Temporary Tooling/Metadata API Performance Issues

  • This seems to be very specific to certain Pods and times
  • Looks like bugs
  • Some of them are reported on https://trust.salesforce.com/trust/status/ - but not all of them
  • The risk of getting these on sandboxes seems to be higher than on production (but it's hard to proof)
  • if we could get more insights on this category, especially on how to report or handle them efficiently without premier support, it would be great! I got for hours a performance like this enter image description here while it should be more like that enter image description here right now it's randomly fast and slow...
1

I've had the opposite experience. It was about a year ago. I have a full data sandbox where a particular set of SOQL queries were running much faster in the sandbox(cs4) than in production(ssl/na0). The problem occurred in a Visualforce page backed by Apex code that potentially accesses a large amount of data.

The Visualforce page and associated Apex code were created for us by a Salesforce MVP. He worked with several levels of Salesforce support to restructure the queries. This reduced the query time, but did not change the performance difference between the sandbox and production. He was told during this process that Saleforce makes no specific performance promises with regard to production versus sandbox performance. Sandboxes may generally be slower as crmprogdev notes, but not always.

Fortunately, the people with the worst delays (admins with access to all data) are not usually the people using this new VF page. The usual users see much less data and thus don't see a large delay. I have not re-checked the performance of the query in production vs sandbox because performance is adequate for the normal user.

  • Interesting. Biggest issue I have right now is that it takes 80 to 200 seconds to save even very short files (APEX classes and VF pages). Same files saved yesterday within 3 seconds and probably (hopefully) will save again fast tomorrow. For me this one on CS14 clearly repeatable and obviously a blunt bug. When it comes to general or query performance, I think you are right and it might be very different form pod to pod and case to case... – Uwe Heim Mar 20 '15 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.