scrabble engine flaws ii

Previously on Apriiori, I expressed some skepticism of the common stance that engines aren’t yet cleanly superior to human players. I absolutely buy the case that Scrabble is nowhere close to computers having the same superiority that they do in chess or Go — but I’m still not sure anyone can point to a particular position where Macondo makes a move that can be decisively proven to be worse than one a human would play. This does, however, come down partially to my having pretty high standards for what constitutes a “decisive proof”: commenters on the previous article gave some pretty credible arguments. I am at this point pretty convinced that a rigorous proof could be put together if someone tried, but I think making the argument fully mathematically rigorous is difficult.

anon: Macondo has clear deficiencies. For most of the game it has two modes: static and sim. Sim is generally preferred, and I think empirically 5-ply sim has had best results. But 5-ply sim is Monte Carlo using an empirical winrate lookup table at the end of a 5-turn rollout assuming both players play like a simple static evaluation (that is, playing solely based on points + rack leave value). So it models its future self poorly, and potentially its opponent as well. This is basically unchanged from Quackle, see http://www.breakingthegame.net/computers5¹¹ They actually linked http://www.breakingthegame.net/computers3, but then corrected themselves and said they meant 5. .

I agree that Macondo has many of the same questionable structural traits that lead to arguable Quackle issues. I had Opus 5.5 look through Macondo’s code and check whether each of the complaints made about Quackle are still true of Macondo:

Macondo does not have all the same issues Quackle does — it does some inference, as discussed in the previous article, and it also can solve end-games if ran long enough (though the Woogles BestBot doesn’t always fully solve the end-games, and doesn’t use Macondo’s inference system). But it has a lot of them.

The thing is, while I agree these are all legitimate imperfections, I sort of suspect a lot of them are still there just because they end up pretty much being good enough approximations in practice, most of the time? It seems to me at least possible to argue that Macondo doesn’t make any outright errors on the positions Matsumoto²² The author of Breaking the Game. brings up. I think some of the positions could pretty plausibly be firmly established to be Macondo errors, but I wasn’t myself able to do this by spending some time hacking away at it with Claude.

Our checks patch each mechanism inside Macondo, so they test the mechanism rather than the move. Scores weren’t visible in the article’s screenshots; 0–0 is assumed except BENZOATE (-150, inferred).

Complicating the picture is that Macondo plays Matsumoto’s preferred moves on some settings. Macondo not being sufficiently certain that ILK is better is only arguably an error, since on deeper sims it does at least play the move Matsumoto thinks is correct. Macondo only playing Matsumoto’s preferred move of SHAKO(S) on shallow sims is something I’m less certain what to make of.

DU: There are cases where engine-preferred move is worse, but it’s mostly because of how current engines work, doing a Monte Carlo simulation and assuming your opponent will play his top equity play.

On closed boards, macondo sometimes struggles to find the best moves (tho it’s often arguable which of the moves is the best, when you cannot trust the engine), especially when it’s trailing, it tends to assume that opponent will open up a bingo spot, which is very unlikely to happen.

https://woogles.io/game/9uDRKeYKf8?turn=22

For example VITAE here is a clear error to any human player. It’s hard to say what’s best for sure, but this move basically concedes the game. I would strongly prefer something like INTI or NIT (hard to say their real win %, it should be lower than what the analysis says).

The engines give VITAE about 30% win rate, in reality i believe it is 0%, because the only bingos possible to draw are through the AG, and any sane player will simply block the spot (as Mack did).

The endgame and pre-endgame (1 in the bag) are played perfectly by the current engines if they look deep enough. On woogles website the bots sometimes make errors because they are solving the endgame at insufficient depth.

As DU observes, VITAE absolutely does not have a 30% win rate. But DU’s analysis here isn’t quite perfect: VITAE’s win rate isn’t quite zero. Using a uniform weighting³³ Not perfectly accurate, of course, but using i.e. Macondo’s inference system to weight Mack’s possible racks doesn’t change that much. over the possible racks Mack might have, there is a particular fixed Mack strategy such that BestBot wins (or ties) at most 1.91% of the time, however BestBot plays, so that’s the tightest upper bound I can give. Most⁴⁴ According to Claude, roughly 3/4 of the possible 1.91% weighting Mack’s racks uniformly, 2/3 weighting them with Macondo’s rack inference. of BestBot’s winning chances after VITAE are because BestBot actually has a second possible bingo lane, through the blank g in row 14.

Now, all of the possible row 14 bingos require tiles that turned out to be on Mack’s rack (CEGLNOO) — with the tiles that were in the bag, AIILMRR, the only possible bingo is M(AG)ISTRAL, which Mack’s CL(AG) blocked. In fact, VITAE’s win rate is exactly 0% given the rack Mack had in the actual game, supposing Mack plays correctly⁵. But while BestBot could perhaps be expected to have a decent guess at the contents of Mack’s rack, it doesn’t seem at all reasonable to expect it to infer that row 14 bingos are off the table⁶⁶ Indeed, BestBot should probably be trying to play to its outs in this sort of position: even if it could infer that Mack probably has a rack where VITAE loses, most other moves don’t seem to do much better, so playing for a row 14 bingo might be the best strategy even if it usually doesn’t work..

The strategy that Mack can follow such that BestBot wins at most 1.91% of the time regardless of what rack Mack has is pretty straightforward: leave BestBot with the fewest winning bingos. For simplicity of analysis, we preferred to play 3+ tiles to hit a full-information endgame, which Macondo can solve exactly; Mack wins nearly all of these. Often Mack has a rack such that BestBot has no possible bingos, so Mack can just play a move that empties the bag and which scores enough that BestBot can’t catch up without a bingo. If BestBot’s only possible bingos given Mack’s rack are in row 1, then Mack blocks row 1 with moves like 1C GAM, REC, ROC, or RIG; 1D GAL(AG)O; or 1E IM(AG)O or RE(AG)INIC. If BestBot’s only bingos given Mack’s rack are in row 14, block in row 14 with a move like 14I A(g)OG.

Mack can’t always do a three-tile play: in the two possible racks with no vowels, Mack can win with 1E CL(AG), as he did in the actual game. There are also a couple other racks that just do slightly better with a two-tile play: the particular strategy Opus 5.5’s code put together does 14H OR(g) with the rack GILLNOR and 1C GI with the rack CGIILLO.

On the other hand, we were actually able to bound BestBot’s winning chance above zero. Again using a uniform weighting over possible racks, BestBot can win after VITAE at least 0.704% of the time. This doesn’t really contradict DU’s claim that VITAE is essentially concession very much: “the win rate is somewhere in [0.704%, 1.91%]” isn’t exactly a cheery situation for BestBot. But the reason I can’t declare this a clean Macondo misplay is that it’s difficult to demonstrate any other play is actually definitively better.

The move which we were able to give the highest floor on the win rate of was DU’s (IN)TI⁷⁷ Maybe this is because we put more effort into raising the bound on DU’s preferred moves., at 0.79%. This is higher than VITAE’s floor, but lower than VITAE’s ceiling. For what it’s worth, my best guess is that (IN)TI’s win-rate is somewhere around 1.5-4%, but this is somewhat difficult to rigorously show. (IN)TI leaves two more tiles in the bag than VITA(E) does, making it a somewhat more difficult analysis job, and even VITAE we were only able to bound within an interval of size 1.2%.

One thing is for sure: Macondo’s “win rate” percentage is not remotely close to the actual win rate. It can be really quite dramatically far off. So Macondo makes obvious gigantic errors if you’re trying to use it as a win rate estimator. But if you’re just trying to use Macondo to choose the best move, all you really need is that Macondo’s win rate percentage is an increasing function of the actual win rate. Macondo being more screwed than it expected doesn’t necessarily mean in and of itself that some other move beats VITAE.