For this to work, you need to have 51% of miners that are “rational”. If there are 51% of miners that are not maximizing profits in the short term, the censorship I’m describing is unlikely to be sustainable.
By “rational” I’m assuming that miners are not worried about how their small participation in something against the ethos in Bitcoin is going to destroy Bitcoin forever. This may or may not be true, that discussion is for another day. “Rational” simply means miners are hoping to maximize their expected revenue in the next day or so. Let’s avoid discussions of tragedy of the commons, market fragility hypothesis, etc.
First you have some blacklisted UTXO. Suppose you have a specific UTXO you would like to blacklist: Say some evil terrorist organization has consolidated all of their proceeds from selling child porn videos on the dark web into one big UTXO holding some Bitcoin.
Now you work for the some US agency, and have a budget, and are tasked with freezing this transaction forever. So you make a declaration: If there is a transaction sourcing from this UTXO, and it gets included in a block, we will pay the miners of any blocks who participate in a reorg that successfully kicks the transaction back to the mempool.
First, of all, people will believe you. You have an very clear interest in establishing your credibility, and people will know this, and so miners who are aware of this declaration will trust you. They then have a decision to make.
Suppose a transaction comes along from that UTXO. A rational miner can either A) include it, or B) ignore it. By ignoring it, the miner takes on no risk, and is at most forgoing a transaction fee (which if large enough, could make a difference.) By including it, the miner risks losing the entire 6.25 subsidy.
The instant the block mining the transaction is posted, “rational” miners then have a decision — they can create a parallel block at the same height, which would put them in position to receive a bribe. In order to make this decision they must consider how many miners they believe are also rational. Rational miners would most certainly choose to append to the censorial block. So then the miner does a quick computation. They multiply the fraction of the miners they believe to by rational by the amount of bribe. Then compare that number with the fraction of miners who would not support the reorg and multiply that by the block subsidy.
For example, if 60% of miners are rational, a bribe of 5 Bitcoin should be sufficient. Someone who mines the equalizing block gets 5+6.25 with probability 0.6 and 0 with probability 0.4. Add these up, the expected value is 6.75. This exceeds the reward for default mining.
The key to this working is a reasonable chunk of miners are rational.
Now, you say, this is very expensive. But if many miners are rational, they will simply ignore the transaction, so you won’t have to pay the bribe all the time. If 80% of the miners are rational, there is 64% a block mining the transaction will be supplanted. Thus the UTXO would have to include a large fee of around 4 BTC to make this appealing to any rational miner.
So most rational miners will simply not mine the transaction in the first place.
However, as many have pointed out, eventually, the transaction is likely to make it through, depending on when you decide to give up the fight. A non-strategic miner will include the transaction, and another non-strategic miner will append to their block.
Now if there are more than 51% of miner who are rational, you can keep upping and extending the bribe, eventually the transaction gets booted back off the chain.
Why though? Depending on what they want to do with their Bitcoin, the terrorist may lose many options. Most counterparties don’t want to be caught up in this and risk having their funds vanishing due to a reorg. The point is mostly to send a message, establish dominance, etc.