does the callback function need to the exact same reference of the function that was added?
Yes. Two functions with the same code still count as different functions. So in your project the attempt to remove the event listener will do nothing as you have not passed the original function.
is there a way to remove the event without passing the same reference?
There isn't built-in support, but you could write a wrapper "disposable" class that accepts a list of events to add, and has a release method that removes all of them. This wrapper class would store the function references internally so the rest of your code doesn't have to worry about keeping function references. C3's code uses this pattern.