Correct.
There is no correlation betwen the stream schedule and VODs.
So you’d have to assume the most recent vod is for the stream that just ended. (And compare start times, and lengths)
Alterantively, if you are trying to show past broadcasts, just show everything from Get Videos - Reference | Twitch Developers using a type param of archive to filter the results.
And this only works if the target channel has Vod Retention enabled