You have to expect users will click everthing, any time, while holding any key and if only by accident. If your UI cannot handle it, your states were not clearly defined.