It is unfortunate that Salesforce officially does not support this. But you can restrict the file types through JavaScript.
If you do a view-source of the client chat window you can see Salesforce generate the below HTMl code for <liveAgent:clientChatFileTransfer />
Visualforce component.
<input id="fileSelectInput" name="file" type="file" />
This is a normal file type input element with id="fileSelectInput"
. So in JS you can get this element and implement your own logic to restric the file types.
Here is a working version of the JS code, you can put in the chat vf page. Update the file types as per your requirement.
<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.7.2/jquery.min.js"></script>
<script type="text/javascript">
var jq = $.noConflict();
jq(document).ready(function(){
alert("1");
jq('#fileSelectInput').change(function () {
var ext = jq('#fileSelectInput').val().split('.').pop().toLowerCase();
if(jq.inArray(ext, ['gif','png','jpg','jpeg']) == -1) {
alert('This is not an allowed file type.');
this.value = "";
}
});
});
</script>