Machine flex; How much is to much?

Group,

I have just completed converting a mill/drill to cnc. It has a belt
driven spindle with a round column. For the z-axis I replaced the
riser casting with THK LM slides that now allow the mill to have
about 11" of travel. There are a couple of reasons why I did this.
First is that 5" travel is too short most jobs that I run, second is
that when boring using the quill there is too much flex resulting in
elliptical holes (same applies for side milling).
Currently I have not measured the amount deflection but I will
attempt to very soon.
My question is to the group what amount of cutter/spindle deflection
is acceptable on this type of mill that still provides reasonable
accuracy and good repeatability?

Trevor