Speeds and Feeds and Torque and Power

I've finally got my gantry mill together and I am having problems
getting the speeds and feeds right.

I've been cutting aluminium with small HSS slotdrills, 3mm and 4mm.
I've already broken 3 cutters, and I can't work out why. (one plus is
my x/y/z motors have more than enough torque)

I've been using Visual Mill to generate my gcode, and using its
internal speed/feed calculator. For 4mm HSS slotdrill, cutting
aluminium, it recommends...

surface speed - 183 m/min
spindle speed - 14562 rpm
feed/tooth - 0.13mm
cut feed - 3786 mm/min

These numbers are way too scary for my first cuts, so I tried to wind
them down a fair bit (which may be my problem) - sorry can't remember
exactly what I used.

I'm using WD40 as my coolant/lubricant manually applied from the can.

I am using die grinders as my spindles. I have two: one has a speed
range of 2000-5500 rpm, 3Nm torque, 400 Watts output power, the other
is 12000-28000rpm, 350 Watts.

I've only had one successful cut so far, running at 2500 rpm, and a
feed of about 125 mm/min. (Oh, and no problem with wood).

So, I'm wondering what torque/power is required to cut at the above
speeds. I'm thinking maybe the spindles can't keep up. My other
theory is I turned the feed down too low, and the cutters weren't
cutting properly.

Any ideas? Is there a Feeds and Speeds 101 anywhere?