Answering to your questions:
What is the behavior of a formula field that is part of a SOQL query?:
They behave as a normal field
Does the running user/site user need to have access to all fields
referenced in the formula? Or is access to the formula field enough?
No, they do not need to, access to formula field is enough. You can check this
What is the ...