When answering a question on Physics, should answers that make reference to non-peer reviewed material that is found on, for example, arXiv, be discouraged, or even removed?
No. A non-peer reviewed reference is better than no reference. The reader can judge the legitimacy of a reference him/herself. In general, Phys.SE encourages answerers to back up claims with references.