If the current system of closing questions functions well, why are there particularly well-received questions that are closed? ... And how can such a system possibly be justified when questions are this well-received.
This is like asking why, if the current system of parents explaining nutrition to their children works well, candy is still popular and children would still make poor choices if left to their own devices. Or why, if Wikipedia's system of editing and reversion works well, the admins don't seem to care, when pornography is posted on arbitrary pages, how many people liked seeing it. (Here I am glossing over the fact that I don't in fact think Wikipedia's governance works very well overall - they have many problems, but this isn't one - and that Wikipedia is "not censored" and would allow sexual content that was on topic and appropriate to explaining the subject matter.)
A well-recieved question that recieves a large amount of upvotes means that the physics community at large likes the question and thinks that it is a meaningful question
It means that users of physics.stackexchange.com like the question.
That doesn't mean they're in the right.
Sites have rules, guidelines, principles and mission statements for a reason.
Even Reddit has understood this for at least 12 years.
Q&A sites, like physics.stackexchange.com, have a much clearer and more focused objective than social media like Reddit or discussion forums like Quora. The goal is to produce reference questions and answers that meet guidelines and are helpful to people who use a search engine to look up information.
Site standards are also allowed to change over time. This has happened to some extent globally across Stack Exchange. The "Opinion-based" close reason was added globally more than 10 years ago, and there was a lot of careful thought and deliberation behind it. The user-facing rationale is also described in the help center, which is more or less copied and pasted to every SE site.
For example, consider this highly-viewed, highly upvoted question, with answers that received massive amounts of upvotes.
Putting aside the previous points - that none of this actually matters - this is nothing. The top questions and answers on Physics sit at around +800. The most viewed question is over 850,000 views.
I do not think that this question is "opinion-based,"
I don't think this is your true objection, given how you have phrased the rest of the post here to complain about the policy generally. If it is, then you should have posted about the specific question, given concrete reasoning opposed to the closure, and not ranted about established network-wide policy.
and also the criteria for what is and isn't closed is incredibly vague – so much so that virtually any off-beat question that has some degree of popularity gets closed
I think "opinion-based" is easy to understand as a closure reason. The question clearly asks "should I trust it", which is blatantly a matter of opinion. Whether you "should" trust any given thing at all is a matter of personal values. Further, the answers clearly offer opinionated advice.
However, I would have voted to close this question instead as off topic. It is either about philosophy ("how do you know that true things are true?") or politics ("why should we be able to trust published research without attempting to replicate it ourselves?").
It is also clearly not focused; there's no way to answer "what is the process to get science knowledge by ourselves?" that isn't either a completely unhelpful "uh, do the experiments yourself anyway I guess?" or else a complete treatise on the scientific method.
A properly focused, on-topic question in this sphere would look more like "How can I do an experiment at home to verify [insert here: one, specific, clearly understood scientific principle]?" - and OP would have to be prepared to be told that it isn't feasible.
What is the point of locking people out of participating in these kinds of questions?
The point of locking people out of participating in these kinds of questions, is to lock them out of participation. This is intentional, and a good thing.
The reason for doing so, is because it is beneficial to the goals of the site to do so. Considered in a consequentialist framework: not locking people out of participation encourages more such questions; and a site full of such questions is ipso facto not useful as a searchable reference on an objective topic. Such questions dilute search results and attract other unwanted behaviours, such as treating the site like a discussion forum. Physics is about as "hard science" as it gets, so these principles are especially important here.