The problems with LDOs vary between device types (and standard regulators aren't immune). It sounds like you have come across a common issue, which is that the output capacitor must have an ESR above a minimum value. If the ESR of the output capacitor is too low then the regulator will oscillate at high frequencies. This is not usually a problem unless you try and use high value (e.g. above 4.7uF) ceramic capacitors as output filter caps. As always, reading the data sheet carefully is the first step to enlightenment.