Did you toy with the thought of using a swastika for buddhist temples? (like on Japanese maps)
I have a vague recollection of doing a fairly simple b & w map of Japan for a client when I was a grad student. The client wanted the traditional swastika for temples so that's what I did. I think it was ok in that case because the intended audience (and the legend) would've kept confusion with other meanings to a minimum. But for the general public I would probably avoid it just because other meanings would likely come more readily to mind to most people.