I think at a certain number of points it will break. The game I'm working on has some large objects that need a bunch of points, so right now I have to break them up.
Could be. I made a mini-golf prototype a few builds ago that used one large custom collision mask for the whole level. I should load it up in 0.99.7 and see how it handles it.
Edit:
Seems fine. There's a whole crapload of points on the object and it appears to be working normally:
http://dl.dropbox.com/u/529356/pingolf_v0.2.cap
The Box object named "collisionmask" is the one with the custom mask on it. Click and drag the ball to shoot it. If it becomes unresponsive (a bug in my events that I never bothered to fix) then hit R on the keyboard to reload.
Anyway, it could be that your mask was bugged for another reason...? Maybe when you hit Close it just didn't save the mask properly or something. Anyway, I'm sure the devs can figure it out.