comments

The Community API from The New York Times allows to retrieve user comments from articles. I want to collect all comments from a given date and analyze the content, by looking at the most common words and display them in a tag cloud. Now the API allows me to search comments by date, but the problem is that it only returns 25 comments. The problem is that on some days roughly 5000 comments are submitted and the API doesn't give me the option to fetch them all at once. The only way to get more comments is by using the parameter "offset". This parameter allows me to set the starting point of the result set and is a multiple of 25, e.g. offset=25 displays me comments 26-50. Fetching all comments in this way needs too long, because each request needs ~1.5 seconds. I am using a web interface to present the results and collect the data on a server using java.