Personally I prefer flangers in the loop - otherwise the effect gets buried by heavier distortion, as you've found - it may just be that this is the wrong flanger for this amp or that the loop itself is causing problems.
There are three possible issues:
Level, phase, and feedback. (And with some - but not all - digital pedals, latency, which does a similar thing to phase.)
Level: if the signal level in the FX loop is above the headroom of the pedal - which it can be, especially with older pedal designs that were never designed for use in loops, and some have such low headroom to the point they will even overdrive with a loud guitar, in front of the amp - then the pedal will distort and limit the signal at the lower level of the pedal headroom, causing a muddy or bad tone and a volume drop. The signal level may be higher on your distortion channel, although I would still expect to hear it breaking up a bit on clean sounds. If it's this, the solution is to reduce the Send level and/or the channel masters, and raise the Output level.
Phase: some pedal designs reverse the phase of the dry signal (not the same as 'phasers', which cycle the phase of the effect part only) - this will cause some cancellation of the signal when used in a parallel loop (like the Mesa one) where you can't completely turn off the dry signal. The result is volume loss and poor tone, but it should affect both clean and distortion sounds equally - although it can still be more 'obvious' on distortion, for some reason. If it is this, the only real solution is to modify the amp to a series loop, unless you have some sort of pedal loop-switching box which allows phase correction (complex and expensive solution).
Feedback: the Mesa loop is very crude and the dry path is sent to the Mix pot without being buffered, which also means that signal can go back the other way, from the pedal's output via the mix pot to its input, which can then make the pedal unstable or actually feed back on itself - although it's not likely to happen with a non-level-boosting effect like a flanger. If it does happen you may not get an actual feedback howl though, if there is an oscillation present outside the audio range you can get a loss of volume and muddy tone too, so I wouldn't rule it out totally. The cure for this one is also to convert the loop to series.
I don't have an Electric Mistress any more (not since before I owned a Mesa amp), but I run quite a lot of analog pedals in the loop and I've come across all these problems with many vintage pedals, and reissues of them. The biggest issue of all is the crude parallel loop Mesa use, which has apparently been removed from the newest series of Rectifiers for 2010, presumably due to the volume of complaints and frustration it causes...
You can convert the parallel loop to series by removing just one wire, though (and moving another, to be really thorough, but not entirely necessary) - you don't need a new amp! There are a couple of threads here on how to do it.
