sorry, just thought of another idea, it's a little bit of a far shot for someone new to Hong Kong, but may be similar to what you need in some ways. Worth at least a quick thought about.
The former airport hotel in Kowloon City. It is right on a main road heading towards the Choi Hung area/Kwun Tong etc. The hotel itself I have never been inside, but it looks decent enough and used to be one of the main hotels for the airport. (It had a footbridge from the building across to the airport) So I am sure it is not a dump. It is most likely low priced as now it is not a special spot. Kowloon City is famous for having so many different countries foods. It can be really tasty. There are some parks there too. Shopping isn't the greatest, but for day to day needs it is in Kowloon City, for your fun shopping it's a quick bus/mini bus ride to Mong Kok or even Diamond Hill.
The thing that has me thinking here is the fact it is really not far road wise from Choi Hung. You can take a taxi or bus on the main road heading for the Choi Hung area. (Make sure the MTR station and not the regular bus terminus) Then change to bus 92 which is at Choi Hung about to go up and over the hills to Sai Kung. So it's perfect spot to get on.
Let me know if you need more info on North Point, Kowloon City, or Tseung Kwan O.
BTW for your map
http://www.centamap.com/cent/index.htm