How To Mill Slots
Having that leg of the angle iron that you are milling unsupported really isn't the best setup. I'm glad that it worked for you but you should really hold pieces completely.
I would first use a 5/16 end mill to cut a central slot clear through the part. From there, it depends on your tolerance for the slot width. Most 3/8 2 and 4 flute end mills will cut oversize slots. A 3 flute end mill will cut closer to the nominal size. Milling machines are designed tokeep the piece being machined steady and stable so I don't think it can jump and break the bit (if you secured it well enough). But the slot you want to mill will nto be very deep. Milling cutters have cutting depth not all that much more than about 2 times the diameter. Milling a Keyway in a Steel Shaft: A keyway is a small slot approximatly 1/4' in width and 1 1/2' long. You need it when you are putting a pulley on a shaft. It prevents the pulley from rotating on the shaft. I'm building a Thickness Sander and I spent a great deal of time makin. Groove or slot milling is an operation in which side and face milling is often preferred to end milling. Slots or grooves can be short or long, closed or open, straight or non-straight, deep or shallow, wide or narrow Tool selection is normally determined by the width and depth of the groove and, to some extent, length.
That formula for calculating spindle speed is wrong too. This is the real formula
(PI * DIA X RPM) / 12 = SFPM
How To Mill Slots In Steel
Using your formula it says to run a 5/16 inch cutter at 550 RPM which is only 45 SFPM, less than half the speed you should run. You should mill steel at 100 SFPM or more.
How To Mill Slots In Mild Steel
I'll mock up a better setup. Ignore the burn holes in the steel. It is the first piece of scrap angle I found laying around. Anyhow, the hold down on the leg keeps it from fluttering around. Chatter is a bad thing when machining.