Ok, I'll digress further on but I'd like to offer you a practical solution: Take your amp to a guitar store and a/b it with other single rectos. You can try different guitars just for arguments sake. This will help you avoid an unnecessary technician's fee and help to discern if the head simply sucks for what you need, if it is the guitar, or if there is actually a problem. Once you get your answer, then you can proceed. Spend no more money than you have to. I'd suggest for guitars you use a Gibson Les Paul Standard or a PRS of some sort. These guitars are basically guaranteed to match a production equipped dual out of the box with their pickups. It won't necessarily be YOUR tone but you can discern some problems pretty quickly if there are any. Run the heads with a Mesa Recto or Stiletto 4 x 12 if possible. If you get enough known components, it should be pretty easy to narrow down the problem. Also, bring your guitar(s). You can try them with other amps of the same brand to see if that is what you are not happy with.
This should help you troubleshoot and cost you nothing. You just have to tell the guitar guys you need to try some gear but you aren't ready to buy quite yet.
***********digression*********
I typed out a long post and realized that it is half irrelevant and doesn't really help you solve your problem but I left it in for informations sake. It will basically let you do some amp tweaking once you're done with troubleshooting.
Ok, I've never heard a recto compared to a boss distortion pedal before. Something is wrong here. A Single Rectifier solo head should blow a Traynor out of the water entirely in terms of massive distortion sound. It should be no contest. With my Dual + Gibson LP Standard, I run the Red channel with the volume just a hair over a quarter, the treble at 6, mids at 2 or 3, bass at 6, gain at 7 or 8, presence at 3, (out of 12) and the amp sounds huge like a bloody chainsaw. I've never heard any other head put out a huge sound quite like it. I've played Single Recs in the store and the tone is quite similar, if not more aggressive since the tubes run harder at an equivalent volume. I suggest emailing Mesa Boogie's customer support with the serial number to find out which design of amp is it. Is it the head? If it looks like the Dual and Triple for the chassis, then it is the newer version. (http://www.mesaboogie.com/Product_Info/Rectifier_Series/The_Single_Rectifiers/enlarge/Group-LG.htm) I checked on the site and it doesn't look like the back plate has the bold and spongy power setting like the higher wattage heads do. I know it doesn't have the tube rectifier already.
http://www.mesaboogie.com/Product_Info/Rectifier_Series/The_Single_Rectifiers/the_single_rectifiers.html
Can you check out the tubes again? I've tried running tubes that weren't tested by mesa in my dual and they were pretty terrible. (tonnes of microphonic squeal. I promptly returned them) The only other tube testing label I've heard in a dual is groove tubes which seem to be pretty awesome. When I re-tubed my Dual, I swapped out each of the Mesa badged 12ax7s in the first spot to see which were the quietest and the noisiest to make sure I was getting the least amount of tube noise possible from the amp. I actually use two SP-12ax7a (high performance) tubes for the first and second gain stage to minimize the woosh and flub. Because the gain is cascading, I suggest that for 12ax7s, you put the least noisy tube in position one and then the next least noisy in position 2 and so on. I went so far as to test preamp tubes at the local L&MQ to make sure I didn't get a dud) There should be a users manual for your amp that you can download from Mesa and if you ask nicely, they'll even ship a print version out to you.
( least noisy 1 - 2 - 3 - 4 - 5 most noisy)
Mesa also should have a tube chart somewhere talking about the different type of tubes they badge and the different sound qualities. This is very handy in tailoring/ tweaking your tone, provided you know everything else is working.
http://www.mesaboogie.com/Product_Info/TubeReference/TubeSalesReferenceGuide-11-10-08.pdf
Here we go. All the tubes have colours which indicate their operating parameters. You can match the power tubes with the same model number of tube that is the same colour. I recall they have Green, grey, Yellow, and Red. Can't remember if there are any other colours. The Yellow and Red run much hotter. I find I prefer those colours.
You can use any tube that is a 6L6, 5881, and el-34. The 6L6 and 5881 must both run on the 6L6 bias setting and the El-34 must run on the EL 34 setting. Basically, the 6L6 / 5881 have a really creamy and almost blue sound to them. They purr and have a smooth high end. They also have a longer tube life on average. In contrast, the EL-34s have a much more aggressive and throaty snarl for gain and the clean is 'brown' and much warmer. It still sounds like a dual, but the tone is more marshall-ish. With tubes, you really get to tweak your sound and 'pick your poison' so to speak. I've tried the 5881s (STR-425) and the 6L6 440s. The 5881s aren't as aggressive. I notice there are some tube types they aren't offering anymore. I haven't yet tried the STR 447s although I'd like to. The El34s I ran I don't see on that chart. They were said to have a sound midway between a 6L6 and an EL34. Right now, I run Yellowjackets with EL84s which is ideal for volume sensitive situations. I haven't had my halfstack out in ages.
As for speakers, I find the tighter the speaker cone, the more gain they can handle before getting muddy. If you have a grindy speaker then you have to back off the gain to clean the sound up. I also find that in live situations, I prefer not to saturate my sound too much because I want people to hear what I am playing. Especially with two guitars, you have to blend your sound. Back in my band days, my brother and I chose our gear on the advice of our guitar tech: I played a Les Paul and he played Tele so that our sounds would be distinct and separate. We found that the distortion tones really jived well together and resulted in a very thick crunch that was also well defined.