I know exactly the sound you are talking about, it's like it's hyper-distorted when you play with a fair amount of power on your strings, and not in a good hyper-distorted way... It's almost a scratchy/harsh sound around the edges.
It's your eq settings. Read the manual about the eq and figure out what you are doing wrong. If you have any settings above 2pm, then that is where I would start. Your gain control is also only really useful up to about 2pm, after that it gets a bit harsh sounding IMO. If you MUST have the gain above 2pm, roll off the volume control on your guitar slightly, and expect shorter lifetimes out of your preamp tubes.
I typically roll off my volume, reduce the treble and bass to compensate when the amp is in this state. I find it is really dependent on the environment when you get this phenomenon. It's usually after the amp has warmed up a bit, sounds great, then tends to get harsh sounding after it's been played for a while.
Question: are you gauging this at bedroom levels, or is this at live/jamming levels?? At higher volumes is where the Dual is supposed to be run. You may find you aren't satisfied with the sound at lower volumes, but once you get a nice amount of output volume behind the amp, it will appeal to you more. These things can f@#king roar at the right volume!!!
You should really read the manual, start with their settings, and explore a LOT!!!
That being said.... you aren't going to find any brutal distortion sounds like Dime in there really. Although Mesa Dual Recs have been known as the de-facto standard for metal, things have changed significantly in the past few years. All of the manufacturers have decent metal offerings right now, and a lot of them have also changed the standards for "high gain" and "brutal distortion". Mesa has not yet followed suit. I personally don't find that those newer brutal metal sounds really appeal to me, and I am content in the realm of mesa.
The simple fact is that to get that "brutal over the top" distortion from a mesa, you really need to boost it. Just what you boost it is up to you, but it's going to be a long process of trial and error. If you can find an amp that "already" does it for you, sell the mesa, and buy the "so and so" amp. (or keep both)
Remember, Mesa amps are designed for ruggedness, appeal to the masses, having a ton of usable tones and, of course, their signature sound.... if that does appeal to you, just move on...
Hope this helps, and I hope that Mesa either bonds with you, or bonds with whoever is lucky enough to buy it from you!!