To put it another way, whispers are like PM’s.
They are user to user, with no reference to a channel at all.