25W is 3dB less than 50W. 5W is 10dB less than 50W.
3dB is mildly noticeable. 10dB is noticeable.
Since hearing is logarithmic, it takes a huge change in power to make much difference in perceived volume. One rule of thumb is that you have to square power to double volume (100W is 2X louder than 10W).
Also, most people do not run their amps wide open. So you're not using all 50W from your amp (probably).
The biggest effect that most people will get from using a lower power amp is that it will sound more compressed. This happens when you run the lower power amp closer to its power limit to compensate for loss in volume. That means that the transient peaks will clip a bit more, reducing your dynamics. This is not really noticeable if 90% of your clipping happens in the preamp, like it does in a rectifier.
It's much more noticeable in an amp like the Electradyne or Marshall, where the power amp is more dominant.