If you look at the Luke-Vader-Emperor scenes, pre-duel, Vader is always about a head taller than Luke.
This reminds me of the Man of Steel fig debates, and my feeling is the same: it doesn't really matter how tall the actors are in real life, what matters is how tall they are portrayed to be in...