New Apex Batch Jobs always end up in the flex queue in holding

Question

I've noticed some strange behaviour in an org which I've not seen before regarding Apex Jobs.

It seems all batch jobs, which are started by a scheduled apex job, never run straight away and are always placed into the Apex Flex Queue in a "Holding" state and are only started once another job is added to the queue.

This behaviour means that if we have 2 scheduled batch apex jobs, one set to run daily at 03:00 (Batch A) and another daily at 04:00 (Batch B), Batch A is held from 03:00 until 04:00 when it then runs (because Batch B is added), but the Batch B then won't run until 03:00 the next day, 23 hours later, because Batch A has started again.

To get around this, as a temporary measure, I've created 4 new hourly batch jobs (running at ?:01, ?:16, ?:31 & ?:46) to ensure the jobs get processed more frequently.

From the Flex Queue Docs:

You can place up to 100 batch jobs in a holding status for future execution. When system resources become available, the jobs are taken from the top of the Apex flex queue and moved to the batch job queue. Up to five queued or active jobs can be processed simultaneously for each org.

This would imply that system resource isn't available, but how could that be if when the next job is started, it seems to force/allow the last job to start running, even though no other jobs are currently running in the background.

All the jobs are being executed using:

Database.executeBatch(new ThisIsMyApexBatchClass());

Any insight into why this is happening would be appreciated, since I believe this to be a problem on the platform, and previous efforts to get help from Salesforce Support are proving fruitless, and just directing us to the forums.

Thanks

Answer 1

In case anyone else ever has this problem in the future, you have to contact Salesforce support and (if you don't have Premier Support because you can't submit Apex issues) convince the agent to escalate this for you, explaining you need to remove any ApexToken jobs which are stuck in the queue. These are visible through a query, but not in the UI, and we (users) can't remove them.

SELECT Id, JobType, CreatedDate, Status FROM AsyncApexJob WHERE JobType = 'ApexToken'

Once this was done, the jobs started as expected.

There was a developer forums post which helped identify this issue: https://developer.salesforce.com/forums/?id=9060G000000I4EzQAK

Answer 2

I am not sure If I am facing same issue. I am not an SF developer but I am trying to identify root of performance issue. We have Third party integrated with Salesforce and it brings in data every day nightly process and also real time. We had 1 month lag in processing. When I try to dig into the backend, I found out there are batch processing happening every few seconds with batch size 1. Every batch running 4-5 secs Which is lot of time for size 1. I always see job always quequed in Flex Queque though only one job processing. There are 49 other scheduled Apex job quequed too.

when I queried using your above Soql . I saw one "ApexToken" in "queque" status since 10 months. Do you thing "ApexToken" is causing performance issue and we need to delete it?