We're a relatively small team, so solutions that are costly to
implement aren't viable for us.
It's up to your team to decide if the cost is worth removing the risk that accidently doing something in production could have. You seem to suggest it's happening often so you need to implement a solution, but I'd question why it isn't getting better on its own.
I certainly have done this accidently, but can't say it has happened more than a couple times. In those incidences, I do think the working norms of the team contributed to it occuring
- Sometimes doing changes directly in production (ex. oh, it's a simple change)
- Not having another person test every single change before it goes to production
- Not documenting all changes or what is expected to be changed
Based on your example, I'd press why it took weeks to realize a validation rule was deactivated in production instead of, presumably, a testing environment? If the validation rule was deactivated in production (instead of a QA org), then someone checking the work should have caught it pretty quickly. You may think it's overkill for making simple changes, but your specific issue is the exact reason you always have someone else other than the dev check their work...because of human error. It's not atypical to see mistakes occur on the simplest of changes
- Misspelling label changes
- Wrong order of picklist values
- Page Layout order
No matter how simple or complex the change, someone else should be checking the work against what the expected change should be (acceptance criteria).
Otherwise, changing how you work could help to minimize how someone may have production open or accidently click on the wrong domain link:
- Working in more scratch orgs (if possible)
- Using IDE as one-stop shop (ex. open org from VS Code)
- Removing solutions that may accidently redirect to production (links/references in hyperlinks, emails, etc in a sandbox).
That last one can be quite tricky if you, while working or testing something, get redirected to production without realizing it. That doesn't sound like what happened for you, but something to consider.
For Admins, you could also take the following approaches:
- Set one person as rotating "production support" and have that person only focus on production for that day, week, etc. Limit the occurrence of them having to be in production and doing work in a sandbox.
- What permissions do they actually need? What changes are they expected to make? Could they get away with limited permissions that would still allow them to make dashboards and reports?
- Going off the above and assuming they do need to make setup changes in sandboxes, you could take your final solution (post-sandbox refresh script) to assign a permission set to your Admin users (add them to a public group) that will give them the "setup" permissions needed just for sandboxes. It's kind of a half-measure, but this is assuming Admins are more at risk for this type of error because they have to be in production at times and they can't adopt all the benefits of the developer workflow mentioned above.
