What i ended up doing is assuming that a channel wont have more than 100 moderators. So if i get a list of names less than 100 from IRC i double check with the API. This isnt really the safest solution but for any streams with 100 to 1000 viewers i can get list of viewers without an API call.