Well i know some would disagree but check out the Line 6 TonePort UX2:
http://www.musiciansfriend.com/product/Line-6-TonePort-UX2-USB-RecordingModeling-Interface?sku=249701
It is a decent little recording interface. It supplies phantom power so if you want to use a mixing board in the future or a condensor mic it is possible. You can also mic a guitar just normally with it, you don't have to use it's modeling since most hate the digital sound. It is user friendly, nothing too complicated.
For mixing you would probably like some monitors for the best sound but could get by with some decent headphones, depending on what quality your after. Here are the monitors i have and really like them:
http://www.musiciansfriend.com/product/MAudio-BX5a-70-Watt-Active-Studio-Monitors?sku=603707
If you want the cheaper route and want to just get some headphones i have these also and really enjoy them:
http://www.musiciansfriend.com/product/Sennheiser-HD280-Pro-ClosedBack-Headphones?sku=242411
Of course you will want a decent microphone if you don't have one and we all know some good ones like the SM57
http://www.musiciansfriend.com/product/Shure-SM57-InstrumentVocal-Mic?sku=270102
Now the toneport comes with a sample of abletron recording software but i can use for quick projects a program made my Magix, i will see if i can find a place to download it and update you on that. It isn't anything too special but get's the job done for little projects. Easy to use also and free.
Now as far as drum machines i use for simple stuff a computer based program called Acoustica Beatcraft. Pretty simple to use, again maybe nothing too special but with a little time with it you can make some decent sounding drums. Or you can look into drum machines as hardware like the boss:
http://www.musiciansfriend.com/product/Boss-DR670-Dr.-Rhythm-Drum-Machine?sku=700289
This of course is all assuming you have a computer at hand all the time. Again it is all fairly simple to use and with little time spent reading you can start recording, which is what i like when trying to just get idea's down or fiddle. Of course there are many other things to look into that can cut some price i am sure, if you have any questions let me know. Also the toneport has bass amps and stuff modeled into it, so if you get your hands on a bass you can put down decent bass tracks too.
I will look into finding a link for that recording program and get back to you in this post on it. Hope this helped some.