There's no worse feeling than realizing the backed up traffic in the right lane that you've been buzzing by was waiting for the same exit you need. I hate being "that guy" but sometimes ignorance catches you unaware.
Totally, that's why I let people in at the start of the queue and get more aggressive about not letting in people the further I get into the exit lane queue. I still let in people but the further into the queue I get, the less permissive I am because if you fucked up, then accept the fact that you will have to miss your exit and reroute back to your destination and hopefully you won't be that asshole next time and enter the queue early.