So for completeness I'm going to answer my question. Agentwork only works if you are using OmniChannel for chat. LiveAgent chat will not populate Agentwork object.

Now one thing to call out there is a nasty bug currently in Salesforce that if the Chat is transferred to another agent. What happens is the RequestDateTime field is null Salesforce treats this as a 0 for a date/time field which they store as a Epoch. Which then gives you SpeedToAnswer of 47 years. As they calculate it the following way AcceptDatetime - RequestDateTime = SpeedToAnswer.

More information can be found here. So I added a condition in my where clause to throw away anything with a greater then 10000 seconds wait time (Which should never happen in my Company Anyway).

Public String generateWaitTimeMessage(string queueToQuery)
{
    Final Integer MINS_TO_QUERY = -60;
    DateTime currentTime = system.now();
    DateTime currentTimeOffset = currentTime.addMinutes(MINS_TO_QUERY);
    List<AgentWork> agentWorkList = [SELECT id, SpeedToAnswer, OriginalQueue.DeveloperName, createddate 
                                     FROM AgentWork
                                     WHERE OriginalQueue.DeveloperName = :queueToQuery
                                     AND createddate > :currentTimeOffset
                                     AND SpeedToAnswer != NULL
                                     AND SpeedToAnswer < 10000
                                     ORDER BY createddate desc
                                     LIMIT 10];

    Integer count = 0;
    Integer waitTimeTotal = 0;
    for(AgentWork aw : agentWorkList)
    {
        waitTimeTotal += aw.SpeedToAnswer;
        count++;
    }

    Integer avgWaitTime = 0;
    if( count > 0 )
    {
        avgWaitTime = waitTimeTotal/count;
    }
    Integer minutes = avgWaitTime / 60;
    Integer seconds = math.mod(avgWaitTime,60);

    String newTimeMessage = 'Chat is Currently Open';
    if(avgWaitTime > 30) 
    {
        String second = 'seconds';
        String minute = 'minutes';
        if(minutes == 1) minute = 'minute';
        if(seconds == 1) second = 'second';
        if(avgWaitTime >= 60)
        {
            newTimeMessage = 'An agent should be available in '+minutes+' '+minute+' and '+seconds+' '+second+'.';
        }
        else
        {
            newTimeMessage = 'An agent should be available in '+seconds+' '+second+'.';
        }
    }
    return newTimeMessage;
}