45

Apparently, Salesforce has released a patch that implicated "Modify all" to be enabled for every single profile in some orgs. This includes standard profiles and custom as well. Yes, even standard profiles!

Very serious stuff. Please make sure you weren't affected.

Link to Salesforce Trust: https://status.salesforce.com/products/all

Please make sure you follow this discussion:

https://www.reddit.com/r/salesforce/comments/bpq336/salesforce_enables_modify_all_in_all_user_profiles/

Guidance on how admins can manually restore user permissions can be found in this Known Issue article

  • 6
    That's not even the worst that is going on - apparently in an attempt to fix this, they removed the Modify/Access All Data from all Admin Profiles in some instances. Including Standard and Custom Profiles.. – Rain May 17 at 14:47
  • 2
    Impact to Salesforce customer orgs 9:46 am CDT, May 17 The Salesforce Technology team is investigating an issue impacting Salesforce customer orgs that have Pardot provisioned, or had Pardot provisioned, in those orgs. A subset of customers may experience intermittent errors, slow performance or an inability to access the Salesforce application. Additionally, all customers on CS3 may see availability impact. Customers should continue to check Trust for updates. status.salesforce.com/products/all – Rodrigo May 17 at 15:26
  • 2
    Wish we could sticky this somehow, I would have appreciated knowing this before my non- parodot orgs went down. Maybe a meta post since this effects a huge number of users & it will show in the sidebar? – battery.cord May 17 at 17:47
  • 3
    This post would be just as off topic on Meta as it is here. Given the prominence of this issue, I have elected to leave the post up for the time being, but my inclination is to delete it in the future as it never conformed to the guidelines here. I don't think moving it to Meta would be any improvement. – Adrian Larson May 18 at 1:55
  • 4
    @AdrianLarson Can't an edit turn it into a question? Like "what can i do to mitigate it, see if i'm affected, etc", "what effects does this have" etc. Post it in a way to keep existing answers on-topic. Perhaps it might be useful as a question even after it's fixed. (PS: I have no idea what salesforce is) – Fermi paradox May 18 at 9:17
30

You can easily check all of your profiles by clicking on "Your Name" > Developer Console > Query Editor, and using the following query:

SELECT Profile.Name, PermissionSet.Label FROM PermissionSet WHERE PermissionsModifyAllData = true

If you see any unexpected profiles there, you can try copying the profiles back from a Sandbox, if you have one available. Keep in mind that Modify All Data will also enable a ton of other object and system permissions, so simply unchecking the box is not sufficient to undo the damage that would be caused.

15

From Salesforce Trust @ 5/17 12:56 pm:

The Salesforce Technology team is investigating an issue impacting Salesforce customers who use Pardot, or have used Pardot in the past. The deployment of a database script resulted in granting users broader data access than intended. To protect our customers, we have blocked access to all instances that contain impacted customers until we can complete the removal of the inadvertent permissions in the impacted customer orgs. As a result, customers who were not impacted may experience service disruption. In parallel, we are working to restore the original permissions as quickly as possible. Customers should continue to check Trust for updates.

Instance List Effected:

NA42, NA44, NA51, NA33, CS21, CS25, CS26, CS30, CS50, CS51, CS53, CS60, CS13, NA63, CS12, CS23, CS52, CS54, CS59, CS138, CS99, NA146, NA92, NA56, NA50, NA57, NA49, CS97, CS93, CS79, CS78, CS67, CS66, CS47, NA58, CS18, CS64, CS65, CS77, CS98, CS11, CS10, CS9, NA32, NA68, NA62, NA37, CS70, CS71, CS90, CS20, CS19, NA77, NA72, NA65, NA46, NA66, CS84, CS85, EU7, CS91, CS61, CS62, CS63, CS68, CS69, NA155, NA196, NA99, CS17, CS16, CS15, CS14, NA87, NA86, NA78, NA74, NA73, EU8, EU9, EU12, EU13, EU14, EU15, CS7, NA39, NA40, NA76, NA88, NA45, NA47, NA52, NA53, NA54, NA59, NA60, NA61, NA64, NA67, NA79, CS8, CS94, CS95, CS96

Its safe to assume ALL instances are effected by this in some way. Comments below point to some orgs being unaffected.

Incident Link :

https://status.salesforce.com/incidents/3815


As of 5/17 ~3:00 pm est. I was able to regain access to na62 & cs65. The issue is still open however so some orgs may come back online before others. My permission sets & profiles all look untouched. These orgs did not use Pardot.


The Trust Incident is still listed as on-going as of 5/20. A new known issue has been created with details on how to restore modified user profiles & permissions.

Can I restore my profiles and user permissions?

Two options exist to restore production profiles and permissions from a Sandbox Copy

  • To determine if your Sandbox Copy contains a valid backup of the data, check the Profiles and Permission Sets in Setup under “Administration/Users”.

  • If your non-admin profiles are configured such that all of the “Standard Object Permissions” (Read, Create, Edit, Delete) are unchecked, then the sandbox org was impacted and is not a valid source for recovery.

  • Permission Sets and User Profiles can be deployed from Sandbox to Production orgs.

Option 2: Sandbox containing production profiles and permission sets does not exist

If a Sandbox containing production profiles and permission sets does not exist and there is an organizational need for you to restore, Admins can manually modify Profile and Permission Set configurations to grant appropriate access to their users.


Additional information is listed on Multi-Instance Core and Communities Service Disruption

(Text as of 5/28)

Incident Update as of May 24, 2019: The following describes facts as we currently understand them. Our investigation is ongoing, and we may provide customers with updated information in the future.

At approximately 9:56 Universal Coordinated Time (UTC) on May 17, 2019, Salesforce’s Technology team became aware of a user permission issue that was impacting customers across certain NA and EU production and sandbox environments.

As the team investigated, they related the issue to the deployment of an application database script that was launched at 01:45 UTC on May 17, 2019. The script was only intended to be applied to a subset of organizations that use Pardot. That change, however, was inadvertently applied to all users across Salesforce orgs that have, or previously had a Pardot license, giving those users elevated permissions for entities to which they had access within their orgs. These permissions may have been broader than the permissions those users were intended to have. Only users who already had permissions in an org received elevated privileges as a result of the script.

To protect our customers and expeditiously disable the elevated permissions given to users, the Technology team made a decision to block access to the Salesforce service for all customers on the affected NA and EU production and sandbox environments. As a result, customers who were not affected by the database script may have also experienced a service disruption.

By 01:29 UTC on May 18, 2019, access for users with a System Administrator profile had generally been restored to customers affected by the database script issue, and full access had been restored to customers unaffected by the database script issue. In parallel, the Salesforce Technology team worked to restore user permissions to the pre-incident state. That remediation to restore permissions has now been executed on all production instances. A subset of customers may still be experiencing issues with user permissions and the Technology team continues to work to restore those.

For the subset of customers who may still be experiencing issues with user permissions, administrators can manually restore permissions for users so those users can regain access to Salesforce. Instructions for administrators whose customers continue to experience issues are available in the Known Issue article. The article will be further updated as more information becomes available. Administrators who cannot resolve issues by the information in the Known Issue article should log a case with Support.

Additional remediation on NA53 / NA57 / NA59 on May 20, 2019 At approximately 11:00 UTC on May 20, 2019, the Salesforce Technology team was made aware that a subset of orgs on NA53, NA57, and NA59 instances were not fully remediated. An investigation showed that the remediation efforts undertaken by the Salesforce Technology team on those instances had not fully restored the permissions to their prior state.

To protect our customers and expeditiously disable the elevated permissions given to users, the Technology team made a decision to block access to the Salesforce service for all customers on the NA53 and NA57 production environments. As a result, all customers on NA53 and NA57 experienced a service disruption, as they were unable to access the Salesforce services. This disruption lasted from 13:50 UTC to 14:29 UTC on May 20, 2019. Once access was restored to the service, a subset of customers may continue to experience a service degradation, which includes the inability to read, update, and/or create objects within the Salesforce service. The Salesforce Technology team was able to remove the elevated permissions on NA59 before access to that instance was blocked, so NA59 did not experience the same service disruption. However, a subset of NA59 customers may continue to experience a service degradation as the Technology team is working through remediations.

Additional remediation on NA49 / NA72 on May 21, 2019 At 07:16 UTC on May 21, 2019, the Salesforce Technology team was made aware that a subset of orgs and users on NA49 and NA72 continued to have incorrect privileges. The majority of affected users in those orgs with standard default profiles now have the current default permission levels, and Salesforce is working to restore these users to their prior permission levels.

Next Steps: An active investigation of the incident is taking place at the moment. Once the analysis is complete and the root cause has been validated, the information will be made available to affected customers. When remediations and preventative actions have been scheduled and implemented, an update will be made available that describes the remediations and the actions that will prevent the incident from happening in the future, including potential technology enhancements, process improvements, and training and education efforts.

  • 5
    The list keeps increasing. It’s already bigger than the one above. Please make sure you keep following the trust site. – Rodrigo May 17 at 17:54
  • 2
    I would assume all instances are down & effected at this point. Every time I load the trust site the page itself doesn't load, just the header. – battery.cord May 17 at 18:12
  • all instances are not down; our PROD org is working fine – cropredy May 17 at 18:56
  • It seems hit-or-miss as to whether orgs unaffected by the database script on these instances actually went down. But I'm assuming that any org on a listed instance could go down at any moment until the issue is marked as fully resolved. – Thomas Taylor May 17 at 19:42
8

As of 3 PM CDT.

This is something I got to hear around from my colleagues who were part of the webinar that Salesforce had around this issue.

  1. The issue seemed to have affected DR sites as well, so rollback seems difficult
  2. Access to other Orgs were restricted as a precaution for the initial issue which impacted Sandboxes too (even if they never had Pardot installed)
  3. The issue is still not completely resolved and is ongoing as listed on trust site, there is no ETA as of now.
  4. Advice is NOT TO TROUBLESHOOT THIS ISSUE ON OWN and to keep monitoring the incident on trust.
  • 1
    The KI suggests that admins affected may need to do troubleshooting to fix (at least as of the time of last update/this comment). – sfdcfox May 20 at 3:50
  • Seems there's no easy way to restore this for sure. – Jayant Das May 20 at 13:05
  • Salesforce has done tons of work to clean up the mess they made. Unfortunately, the fix from Salesforce (I believe) will affect the standard profiles. If you have custom profiles (which we all have)... I'm not sure if they can do anything about it. Lucky are the ones that have some kind of VCS in place, so they can compare what has changed, or a recently refreshed Sandbox. Even though, there will be a ton of work on the admin side to fix it. And I believe this will keep rolling for a very long time. Not an easy fix for sure. What about those without Sandboxes or VCS in place? – Rodrigo May 21 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.