I can't really help on the merchant account (i.e. payment system) but can recommend OSCOMMERCE as your shopping cart. It seems to have the best developer base and is relatively easy to customize. I tried MIVA and was frustrated by the lack of options and the price.
I have oscommerce (which is free, in case that wasn't mentioned) up and running on my radio site:
http://www.penguinradio.com/catalog/
If you are taking subscriptions, I think paypal has a really good system out and there APIs and modules available that will work the subscription payments into a mysql database and email type notifications.