56

Over my time as a Salesforce developer, I've written some code which on occasion managed to surface Oracle plsql exceptions and curiously the error text contains the name of one of Disney's Seven Dwarfs.

I've teased out Doc, Grumpy, Happy, Bashful and Sneezy. I'm sure the other two are not far away.

  1. What is the underlying "thing" on the database platform that is named after a Dwarf?
  2. Why were these named after Dwarves?

java.sql.SQLException: ORA-30006: resource busy; acquire with WAIT timeout expiredORA-06512: at "DOC.CAPEX", line 2200ORA-06512: at line 1: SQLException while executing plsql statement: {call cApex.update_sysmodstamp(?)}([

java.sql.SQLException: ORA-30006: resource busy; acquire with WAIT timeout expiredORA-06512: at "GRUMPY.CAPEX", line 2563ORA-06512: at line 1: SQLException while executing plsql statement: {call cApex.update_sysmodstamp(?)}([

java.sql.SQLException: ORA-30006: resource busy; acquire with WAIT timeout expiredORA-06512: at "HAPPY.CAPEX", line 2715ORA-06512: at line 1: SQLException while executing plsql statement: {call cApex.update_sysmodstamp(?)}([

java.sql.SQLException: ORA-30006: resource busy; acquire with WAIT timeout expiredORA-06512: at "BASHFUL.CAPEX", line 2715ORA-06512: at line 1: SQLException while executing plsql statement: {call cApex.update_sysmodstamp(?)}([

java.sql.SQLException: ORA-30006: resource busy; acquire with WAIT timeout expiredORA-06512: at "SNEEZY.CAPEX", line 2718ORA-06512: at line 1: SQLException while executing plsql statement: {call cApex.invalidate_agc(?,?)}([

Error: ORA-20000: ORA-06512: at "SNEEZY.SIMPORTLOOKUP", line 850 ORA-06512: at "SNEEZY.SIMPORTLOOKUP", line 823 ORA-06512: at line 1 : SQLException while executing plsql statement: {call sImportLookup.get_by_ext_id_text(?,?,?,?,?,?,?)}([

  • 6
    Related: salesforce.stackexchange.com/questions/41142/… – sfdcfox May 15 '15 at 17:41
  • Another one of the 7 dwarves sighting: salesforce.stackexchange.com/questions/17913/… – Jenny B May 15 '15 at 19:05
  • 1
    shd this be a discussion in meta or chat room.. ?? – Vamsi Krishna May 16 '15 at 0:03
  • I got the sleepy one today "System.DmlException: Insert failed. First exception on row 0; first error: CANNOT_INSERT_UPDATE_ACTIVATE_ENTITY, A workflow or approval field update caused an error when saving this record. Contact your administrator to resolve it. common.exception.SfdcSqlException: ORA-20067: OPPORTUNITY_LINEITEM.TOTAL_PRICE ORA-06512: at "SLEEPY.SOPPLINEITEM", line 1182 ORA-06512: at line 1 {call sOppLineItem.update_opplineitems(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)} {call sOppLineItem.update_opplineitems(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)}: []" – RajeshShah May 21 '15 at 7:03
  • I found grumpy one time working with Quotes: salesforce.stackexchange.com/questions/68243/… – Programmable Medley May 21 '15 at 16:58
36
+100

This seems timely as Grumpy just popped out of the developer console at me.

Grumpy Dwarf in Salesforce

As per @sfdcfox's comment and answer to the linked question.

The Seven Dwarfs live in the platform code. You should never see DOPEY, SLEEPY, DOC, GRUMPY, SNEEZY, BASHFUL, or HAPPY, but occasionally they break out and appear to a user. If you spot one of them, you need to contact support so they can be put back in their dwarfy walled garden.

There are references to them back in 2009. See Salesforce, Snow White and the Seven Dwarfs and The Silver Lining - Meaningful Error Messages #94.

I suspect anyone who can tell you exactly where they are in relation to the database has probably signed a non-disclosure agreement. That doesn't mean we can't make some wild conjectures.

The presence of Oracle error codes like ORA-00001 and ORA-20191 plus the java.sql.SQLException would indicate they are some sort of layer immediately above the database running Java.

Its also interesting that they tend to bypass the GACK. Either they aren't being caught by the GACK mechanism or they are being rethrown. The latter seems unlikely. Why surface these errors rather than the generic GACK message. So the dwarfs live outside the standard application GACK error handling.

Why are there only 7? I'd assume it has something to do with the way Salesforce scales. Each instance/pod, e.g. na7, may have exactly 7 dwarfs. There are never any more or less and they would stand up a new pod rather than scale out the dwarfs. If you've got seven fixed servers I guess you are either naming them after dwarfs or deadly sins. Getting errors from Happy are probably better than errors from Lust or Gluttony.


Here's slide 8 from a Salesforce.com -architecture talk that I've defaced with where I think the dwarfs roughly reside in relation to the servers that makeup a pod and it's database cluster. enter image description here


Here's another diagram from The Force.com Multitenant Architecture pdf.

The Query Servers might be in about the right place and performing the expected dwarf functions.

enter image description here


In Performance Monitoring and Testing in the Salesforce Cloud the Pod is shown as "30 plus servers" being used for Application,API, and Search servers over a "8 Node RAC cluster". I've never done infrastructure at that scale, but I suspect there is something working between all those servers and the database cluster.

enter image description here


See also: * Quora: What does Salesforce's infrastructure look like?

  • 5
    Haha. "Getting errors from Happy are probably better than errors from Lust or Gluttony." Maybe not Rage though. – Adrian Larson May 19 '15 at 23:31
  • 2
    Thanks for the solid answer Daniel. I find it oddly amusing that I crafted nearly the same title for this question as that forum post from 2009, despite not having seen it prior to your answer. I bet GoodGroove and I are sympatico in our quest to seek the great unknown! – Mark Pond May 19 '15 at 23:39
  • Thanks for the great answer Daniel. this is useful. :) – sanchit Nov 7 '18 at 7:56
2

While I cannot provide insight to the naming conventions of the salesforce devs, these errors seem to imply that the database is timing out when attempting to acquire concurrency control locks on subsets of the database that are already locked/currently being accessed by another transaction. Perhaps you have conflicting transactions in your code.

  • Your guess is close to the root cause of those noted exceptions. I'm especially curious to hear about the fairy tale with this inquiry. – Mark Pond May 20 '15 at 1:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.