The difference in performance will likely be very very small. Don't worry about it.
There is another way to organize your code - put events for each control type into separate groups. This way you can activate/deactivate each group, and events from deactivated groups will simply not run.