@Antonio_Cucciniello's comment solved this for me, sharing as an answer to make this more noticable.
If your bot has any Apex actions, make sure you don't have any errors. This will automatically try transferring you to an agent.
Inside your Bot Builder's top left dropdown go to:
Performance
➡ Event Logs
Visually check the Errors column for any "✓" marks on any rows.
If you see any, double click the logs Session ID
, and explore what Event's have Errors. (if it was a recent log, you might need to wait a couple minutes for the log to load its data)
To see the actual error, click the "🔘" on the row showing the Error column with "⚠". (It's kind of confusing to click the radio input to show you the error)
When I checked, I found that my bot had error:
[{"message":"You don't have access to the Apex class: Bot_SearchFAQ","errorCode":"INSUFFICIENT_ACCESS"}]
It was easy to fix once I saw this error. (add the permission set to my bot)