Here is a really good blog posting about a photographer who was hassled by some security guards while he was photographing the building they work in.
They tried to tell him that it was illegal for him to photograph the building.
This is complete bullshit. And he stood his ground.
Worth a read.
And here is a PDF by a lawyer listing your rights as a photographer. Useful to carry around. I’m going to put a copy in my camera bag.