Hmmm... I'm not sure I'd call them taboo, exactly...
More like odd. Since we use the word 'toilet' for the actual... well, toilet,

it seems weird to say you're going to it. Going to the room where the toilet
is, however...
Now, if you're talking about the actual toilet - 'Make sure you scrub the toilet when you clean the bathroom' or 'Can you tell me which aisle has the toilet bowl cleaner?' - then (obviously!

) we say 'toilet'.
'Loo' is just sort of odd in the US. If you're somebody with a strong english accent, people who know the word probably wouldn't react much except to tell you where the bathroom is. If, however, you speak with a down-home twang, New York accent, midwestern drawl (etc.) then I imagine you'd get some looks

.
And yes, 'restroom' is used a lot when you're out in public, and 'bathroom' is what it's called at home. 'Ladies' room' and 'men's room' are pretty common, as already mentioned - but I don't think I've ever heard those in a home. Of course, home bathrooms are unisex, so...
Then you've got your less "polite" term - 'Where's the john?'
And your standard armed forces term - "I need to use the head.'
Huh. Thought I outgrew "bathroom talk"

Guess not.
~Toc