If I'm understanding this correctly, you need something like this. If your company uses custom fiscal quarters, you'll need to adjust to account for that. If all conditions are true, the error would be thrown.
Keep in mind also that this will only check when the opportunity is edited and will not be a real-time indicator of opportunities that would otherwise meet the criteria.
AND(
/*Check that the year of the close date is the same as today's year*/
YEAR(TODAY()) = YEAR(CloseDate),
/*Check that calendar quarter of close date is the same as today's calendar quarter */
CEILING(MONTH(TODAY())/3) = CEILING(MONTH(CloseDate)/3),
/*Check that Stage is not Closed Won or Closed Lost*/
NOT(
OR(
ISPICKVAL(StageName,"Closed Won"),
ISPICKVAL(StageName,"Closed Lost")
)
),
/*Check that Probability_To_Close_This_Q__c is blank*/
ISBLANK(Probability_To_Close_This_Q__c)
)
****EDIT****
If Probability_To_Close_This_Q__c
is a picklist, and you want to use a contains operator on StageName, you can try the following.
AND(
/*Check that the year of the close date is the same as today's year*/
YEAR(TODAY()) = YEAR(CloseDate),
/*Check that calendar quarter of close date is the same as today's calendar quarter */
CEILING(MONTH(TODAY())/3) = CEILING(MONTH(CloseDate)/3),
/*Check that Stage is Closed*/
CONTAINS(TEXT(StageName),"Closed"),
/*Check that Probability_To_Close_This_Q__c is blank*/
ISBLANK(TEXT(Probability_To_Close_This_Q__c))
)