On your way out of the castle, stop by the Argyll House. Argyll House was the 17th Century residence of the Earl of Stirling. It is decorated in true Renaissance fashion. This beautiful piece of history is managed and maintained by Historic Scotland, and makes a brilliant addition to a side trip to Stirling.
A side trip to Stirling doesn't necessarily have to cost you more than a half day, though you could stretch it out with an exploration of the quaint shops and restaurants in the surrounding neighborhoods.