VIPs/mods isn’t found on the GetUsers endpoint.
And each can be loaded in one call to Get Mods/Get VIPS which yes (at time of writing) needs a broadcaster token. So to split out into groups. You need additional calls.
But VIPS/Mods don’t change that often so you can precache those.
In the case of Mods you can use EventSub to keep up with Mod add/remove (VIPs doesn’t have an eventsub but I just made a uservoice to request it Topic Notify when VIPs are added/removed – Twitch UserVoice)