"You did the measurement wrong" is not an answer

Question

I've seen two hot network questions in the past two days where one of the highly upvoted answers was a variation on "you did your measurement wrong". In both cases, this answer was incorrect--the measurement was right, the simplest physical model was wrong, and there was better physical model that matched the data very well.

• The first such question is Why did I measure the magnetic field to vary exponentially with distance?. The measurement showed that the magnetic field of certain magnets decayed exponentially. This differs from the simplest model--that of a dipole--and one of the answers claimed that the OP got the measurement wrong. It turns out that there was a better model for the magnet in question, namely a Halbach array, which correctly accounts for the exponential field decay.
• The second such question is Pan flute physics. The measurement showed that the frequency of a half-open pipe was lower than the naive formula $\frac{c}{4L}$ would predict. One answer claimed that this was due to imprecise knowledge of the speed of sound $c$. However, this turns out to be wrong (for reasons I explained in a comment on said answer), and the right explanation is that there is a better model for the frequency of a half-open pipe which accounts for non-zero diameter of the pipe, and does a better job of predicting the measured frequency.

A key point in both of these questions is that the OP knew the simplest model for the phenomena in question (respectively, field of a magnetic dipole and simple formula for frequency of a half-open pipe), and their questions were essentially "why does my measurement differ from the model?"

I'm going to make a bold assertion and say that trying to explain away surprising measurements as "bad data" is itself bad physics, and I think answers along these lines should be discouraged. The reason is that literally anything can be "explained" in this way, and it prevents you from seeing the "right" physics if it doesn't coincide with whatever simple models the answerer happens to favor. I think both of the preceding examples prove this point.

Having said this, I acknowledge that there are certainly cases where experiments are performed improperly. Notably, neither of the aforementioned questions had any estimate of the uncertainty of their measurements (an aspect often tragically overlooked in non-research-level experiments). However, for the purposes of this site, I think we ought to generally consider experimental error as a legitimate explanation only as a last resort. We should prioritize answers which explain an observation with physics, rather than fudge factors.

Answer 1

I think we ought to generally consider experimental error as a legitimate explanation only as a last resort.

Practically by definition physics is defined by experimental verification. It's not even remotely a "last resort" to suggest an experiment has a fault. It's exactly what happens anyone who publishes results they cannot explain - the first things you have to do is show you are :

• using an accurate theory for that scenario
• using reliable technique and equipment to measure accuratly

On Physics SE we have two linked rules for closing questions : "homework" type questions and "check my work" type questions. These are almost invariably closed by the community. It would be very easy to argue (and I would) that "explain why my experimental results differ from my theory" questions are off topic under that check my work rule.

We should prioritize answers which explain an observation with physics, rather than fudge factors.

I'm going to make a bold assertion and say that trying to explain away surprising measurements as "bad data" is itself bad physics

Saying "you most likely made an experimental error" is not a fudge factor, it's a legitimate observation - it usually is the reason for problems when a reasonable theoretical model has been used. Having spent an indecent amount of time in labs doing experiments I can tell you it is also the most likely explanation. As more than one (in)famous example shows, even the most highly regarded of scientists can get it wrong very publicly.

So for me, "bad data" is the default assumption until it is shown to be "good data". This is why repeating experiments and other people repeating your experiments is so important. It is why peer review is so important - other eyes see what you don't because of (unconscious) confirmatory bias or just mistakes. But peer-review is not a function of Physics SE, so we do not do these type of questions.

There is no practical way (on a site like Physics SE) for members to know that the OP did (or did not) make accurate measurements. The default position for most physicists is that if experiment does not match theory then either you used the wrong theory or (more likely and the Occam's Razor first option) that measurement was not done properly or did not account for some unknown factors.

Physics experiments are extremely difficult to do well. Models are normally going to model ideal conditions whereas experiments never usually happen that way. Errors are normal, expected and if I saw no errors or unusually low errors I would actually consider the experimental results very suspect (including my own).

So, for me, experimental error is the first explanation you look for if you think the theory ought to be accurate enough. Whether it's a failure to design a good experiment that matches the conditions of the theoretical model or a failure to be diligent enough measuring or an equipment failure, it's the experimenter's job to demonstrate they made no errors. Not a thing that's typically practical here.

their questions were essentially "why does my measurement differ from the model?"

We need to differentiate between "why does this happen ?" questions (as we get them in everyday physics type situations) which are not experimental but ask for qualitative/intuitive explanations of observations and "check my experiment" type scenarios. The former is fine, but the later is not practical to answer and arguably off-topic under the "check my work" rule.

The questions you ask about seem to fall under the "check my work" category (IMO), making them off-topic.

"My experiment doesn't work" is something Physics SE is just not (in general) designed to deal with.

That's my ten cents anyway.

Answer 2

Notably, neither of the aforementioned questions had any estimate of the uncertainty of their measurements

... which means that the explanations provided in the answers to those threads that you objected to, which are basically of the form "the observations are consistent with experimental noise at the level to which you have reported the latter", are correct.

There are plenty of experimental observations that don't match the simplistically-expected result and which are instead due to more complicated physics than were anticipated. However, the key aspect of those observations that allows them to make that distinction is the fact that they take seriously the quantification of the experimental uncertainty.

In those two threads, it does seem that there were for a while top-voted answers with incorrect responses to the specific details, but even then (particularly for the magnetic-field one) those answers still provided valuable insights about physics to the wider HNQ SE audience.

Insofar as they were wrong, there's a strong argument to be made that there's nothing specific to experimental physics about it -- it was just simple entry-level answers which turned out to be incomplete or inaccurate upon closer examination, which is a generic liability on this platform. You're right to point out that we should keep an eye out for when OPs' observations might be more correct than we often give them credit for, but does this not also apply to analogous situations on the theoretical side?

Just some thoughts -- I have no strong opinions about this.