Greg,
I have the 35mm-150mm that came with my D80 and I would say go for the 300mm. If I had any kind of a comment about the D80, which I truelly love, it would be that I wish I had just a bit more reach in the zoom area which I think a 300mm would be GREAT! What you might do is take the camera to a camera shop and see if they will let you try the lens you might be interested in on your camera and just take a few shots in the store...sort of like a test run. They might let you do that. Then you can make a better decision of what you might want.

Good luck!