Search Ideas

Bulk

Related Popper reading: ‘Preference for Theories and the Search for Truth’

The rules should say whether AI-generated content is okay to submit or not.

For posterity regarding the most recent bounty on this idea, ChatGPT made me this awesome illustration a while back for my post about the bounty on X:

I’ve just posted the following update on my article linked in the parent idea. The bounty can be closed, at least for now.

Update 2026-09-06

TL;DR: Computational underspecification may have been solved; HTV is still not a description of rationality or of the cause of progress.

Dirk Meulenbelt, Benjamin Davies, and I ran a Veritula bounty worth USD 3,600 to rehabilitate HTV with regard to its computational underspecification. Jad Elmourad came up with a simple idea:

1. Prompt the user for two explanations.
2. Prompt the user for variants of each.
3. Compare the count of variants and rank accordingly (lower count = better).

Unfortunately, the integrity of the discussion of this idea can’t be warranted because of problematic participation in the bounty. As a result, it’s currently not known whether the idea has pending criticisms, ie, whether its advocacy is rational.

That said, the program runs. I’ve tried it out. It offers a simple alternative to the sliders above. It does seem to partially rehabilitate HTV specifically with regard to its previous computational underspecification. In other words, where proponents of HTV used to be unable to explain exactly how to determine how HTV an explanation is, they can now follow the steps above and answer accordingly. They can now rank explanations this way. That helps.

What the program also shows, maybe ironically, is that nobody actually follows those steps in real life. For example, scientists don’t count variants of theories like that. Deutsch is still right about the following: “Whenever a wide range of variant theories can account equally well for the phenomenon they are trying to explain, there is no reason to prefer one of them over the others, so advocating a particular one in preference to the others is irrational.” (BoI chapter 1; also, the ‘grass cure’ from FoR comes to mind as an example.) So when a theory is easy to vary in this sense, that does indeed mean we can’t form a rational preference for one of the variants, at least not without additional information. But that doesn’t mean all preference formation involves theories being hard to vary.

What scientists actually do is criticize a theory for being ‘easy to vary’ (they’d just call it ‘arbitrary’), reject it, and look for a different one. They wouldn’t care to rank theories this way. And they’d use HTV as only one mode of criticism among many. It’s not the core of rationality and progress; again, not all problems of preference formation can be reduced to having too many variants.

Philosophers don’t rank theories this way either. I don’t recall a single instance in BoI where Deutsch counts variants and justifies a preference that way. Instead, he lists and addresses criticisms. He follows the Veritula method.

Also, one would only enter variants into the program that have already survived criticism. So then the question is how scientists tell whether an idea has survived criticism. And there the answer is still the Veritula method. So the program could maybe be incorporated into Veritula as an additional step. But again, it’s not needed if ETV is simply one kind of criticism among many, with no ranking.

To be clear, that isn’t a criticism of the program – the bounty specifically asked for the ranking. Here, I’m only exploring the wider implications if the program can be advocated rationally. And the answer is: the computational underspecification is then solved, but HTV is still not a description of rationality or of the cause of progress. That idea, as I just phrased it, can be adopted rationally at the time of writing, and we can move forward accordingly.

The integrity of the discussion branches off of this idea and its siblings is questionable due to problematic participation in a bounty. Labels around pending criticisms and battle-tested may be not be accurate.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved, see #5624. Submitting this as a fix. Jad agrees to terms, breaks them, then wants to argue the purpose and meaning of them. That’s arbitrary, ie the opposite of reason and Veritula. He will no doubt claim he was just trying to follow the Veritula method of addressing criticisms, but the effect is that he’s turning Veritula on its head in the process. There’s no point discussing with people like him.

I gave you an opportunity to be eligible for the remaining funds if you just stopped arguing. Now you’re doubling down and arguing more. You’ve forfeited the remaining funds.

Veritula discussion trees can’t accurately reflect problematic vs unproblematic ideas when unreasonable people are involved.

I’m banning you from Veritula in perpetuity. Don’t contact me again. I wish you well and hope you find another forum more to your liking.

When thousands of dollars are at stake, I think it’s fair to ask people to play by the rules. Still, I’m allowing flexibility by giving Jad a chance to collect the remaining funds.

I’m always happy to reward good work; I’ve paid out bounties in the past. And before I found out about Jad’s use of AI, I gave him tips increasing his odds of beating the bounty. I also submitted criticisms of some of my own ideas, further increasing his odds.

But people need to earn the reward. At some point, I noticed Jad had submitted his replies very quickly, with unusually verbose text, and in fairly rapid succession. AI is notoriously verbose, so that would have been the second-best time for him to mention he was using AI. He did not. But some people do genuinely type fast, so I gave him the benefit of the doubt.

When I later asked him directly whether he was using AI, he admitted to it but claimed it was merely to “help edit some of [his] writings”.

Pangram, an AI detector, paints a different picture: it says 100% (!) of the code and 22% of the prose in his first idea are AI-generated. I’ve also sampled some of his replies since, which Pangram says are also 100% AI-generated.

To play devil’s advocate once again, I wanted to make sure Pangram wasn’t giving me false positives. So I ran several tests against some of my own prose and code, and compared the results with known AI-generated prose and code. Pangram was almost always right – the few times it did make mistakes, they were almost always false negatives, so if anything, Pangram is too permissive. It does have lower confidence for shorter texts, so I also made sure to use only texts that are long enough. I showed Jad these results privately and he didn’t have a good answer, so I’m surprised he now wants to continue the discussion publicly.

In terms of content, handwriting the code matters because of Deutsch’s yardstick for having understood a computational task: if you can’t program it, you haven’t understood it. Jad says he still understands the code regardless, so it shouldn’t matter. But why not raise that before agreeing to a rule he took issue with? Why accept the risk of disqualification instead?

In terms of fairness, as Jad knows, payout for a bounty depends on addressing all known criticisms inside a review deadline. When a participant uses AI to help with that, but the funders don’t know about it, it creates an unfair advantage. That’s a violation of not just the letter but also the spirit of the rules. It’d be like secretly using a chess computer for help in a chess competition. The amount of text Jad has submitted also creates an uphill battle for the remaining funders to review his submissions before the deadline. Jad says he knows how bounties work, so he knows payouts are automatic at the deadline. Still, the remaining funders are willing to accommodate him.

Before I made the decision to withdraw my funds, I reached out to Jad privately, as I said, to make sure there wasn’t anything I had missed and that there were no hard feelings. We discussed the issue thoroughly and I couldn’t find anything. I also discussed the matter with the co-funders for a second and third opinion, again to make sure I hadn’t left any stones unturned. (To be clear, the responsibility for the decision to withdraw my funds is mine alone.)

I understand Jad was hoping for a bigger payout, so this must be a disappointing outcome. I empathize with that. I’d be bummed too. But he could have easily avoided this outcome. One can’t agree to a rule, immediately break it, get an unfair advantage, hope I won’t notice, and then complain when I do. But with any luck, he’ll get the remaining funds, provided that the arguing ends now.

Unfortunately, Jad has admitted to using AI for his submissions, in violation of the bounty terms. As a result, he’s not eligible for a payout. After discussing the issue with him and separately with cofunders @dirk-meulenbelt and @davies, I’ve decided to withdraw my funds.

However, the co-funders say they don’t mind the use of AI. For that reason, and as a token of appreciation for Jad’s idea, I’m making an exception and he’ll remain eligible for their funds, pending further development of this bounty.

This is out of scope for the bounty. The bounty merely says the program must “Return a ranking indicating which explanation is better, worse, or equal.” It doesn’t say the program has to specify which variation.

The program can't accept my judgment that these are genuine variations and then independently override that judgment because it “knows” they're actually irrelevant. Doing that would require the program itself to understand what is doing explanatory work. That's precisely the part I'm outsourcing to the user.

Fair enough.

As of 2026-09-02, I’ve removed that part from the blog post.

It sounds like you’re agreeing, in which case this shouldn’t be marked as a criticism.

@jadelmourad be sure to submit separate criticisms separately. You benefit because it means I need to address each criticism individually, not in ‘bulk’. (See #5546, keyword ‘bulk’.)

One could run the program again with corrected inputs but as I noted in #5533, that approach won’t scale.

Jad originally replied in #5559. I’ve extracted his reply to avoid bulk criticism:

If the user later changes their mind about one of those judgments, then yes, the current minimal CLI requires them to run it again with the corrected input. I agree that wouldn't be good UX for a full application, but that's not what I'm trying to build here. Adding the ability to add, edit, or delete variations and immediately recalculate the ranking is straightforward and doesn't change the decision procedure being demonstrated.

I think the program already gets this through human input.

The prompt isn't asking the user to enter arbitrary changes. It specifically asks:

“Enter variations of this explanation that would still work to explain the same thing.”

So by entering a variation, the user is making the judgment that it still accounts for what the original explanation was supposed to account for. That's intentionally a human judgment, just like the other judgments I've discussed in #5553.

Revisions aren’t for counter-critizing criticisms. Revisions are for changes you make because you agree with a criticism. I’ve restored the original. I’ve also updated #5519 to make the purpose of revisions clearer. It’s now #5546. Please read it in its entirety before you continue working on the bounty. Then continue with #5544.