Partner API Query() : query string with large number filter criteria

Question

We are using force java partner api for playing with the Salesforce data

Salesforce Partner api provides a neat way to filter sobjecttype by ids as a String[]

String[] ids = {"id1","id2",.....,"idn"};
connection.retrieve("ID, Name, Website", "Account", ids);

If the filter criteria is Name, then the query() call is being used at the moment

String queryString = "SELECT id, name from Contact WHERE name in ('name1','name2',...,'namen')";
QueryResult res = connection.query (queryString);

We are still playing around with the records fetch. But I see that when the number of filter increases to 100's or 1000's the queryString length increases crazy and (a) exceeds limit, and (b) becomes ugly

1. Is there a better way to filter sobject by a field other than id ?
2. If (1) is not possible, which I believe so: There is a limit on the characters for the SOQL - how would you get around that - create multiple queries ?

Maybe (2) is what we may be looking here in case of large number of filters - but I am trying my luck here to see if someone has a better solution and I might have missed some hidden method that has been overlooked

I am looking for some method of format:

String[] recordNames = {"name1","name2",...,"namen"};
connection.query ( "ID, Name, Website", "Account", recordNames , "Name" ) ;

and this doesnt exist

Answer 1

Number 2 is the correct solution. You need to manage that length and perform multiple queries while building a result set in your client app. Really sounds like you need some type of external id \ indexed field to query against.

Answer 2

We ended up exposing a webservice to taken in the object name and filter criteria to fetch us the required results. This seemed to be the most efficient for the time being. We may end up with some humongous dataset at some point in time and may exceed the 10MB limit for webservice, which would need to be addressed as multiple ws calls! But this way we noticed that our api calls are reduced!