%--- iliad ---
% title: Open-Source Game Theory
% summary: >-
%   When agents can read each other's code, cooperation needs neither repetition
%   nor trust — but the same transparency starts races to commit first.
%   Conditional commitment, safe Pareto improvements, and a tournament.
%--- end ---
\documentclass[11pt]{article}
\IfFileExists{iliad.sty}{\usepackage[boxes]{iliad}}{\usepackage[boxes]{../iliad}}

% A Unicode glyph this page uses verbatim, which inputenc cannot map.
\usepackage{newunicodechar}
\newunicodechar{□}{\ensuremath{\square}}

% The page header and the nav take the short name, set as `title:` in the
% frontmatter block above; this keeps the author's full one, which is what the
% built PDF shows.
\title{Decision Theory II: Open-Source Game Theory, Commitment Races, and Safe Pareto Improvements}
\author{
  \authorname{Daniel C}
  \and
  \authorname{Satya Benson}\\
  \affiliation{Williams College}
}

\begin{document}
\maketitle


% \subsubsection in the module doc, where these sit under Content > Main
% content. Those two wrappers are day-wide and live on the first page only, so
% here the headings are top-level and are numbered as such -- left as
% subsubsections they came out "0.0.1", the section and subsection counters
% never having been incremented.
\section{Afternoon lecture}
\textbf{Open-source game theory.} When players can read each other's source code, cooperation becomes possible without repetition or prior trust. \textbf{FairBot} cooperates if and only if it can prove its opponent cooperates with it; two FairBots cooperate by Löb's theorem (from \texttt{□(□C\ -\textgreater{}\ C)\ -\textgreater{}\ □C}). It is unexploitable but brittle (it relies on a proof system and exact source). \textbf{Epsilon-grounded bots} (Oesterheld) replace proofs with simulation: cooperate unconditionally with small probability epsilon, else simulate the opponent and copy its move; this terminates almost surely, is a Nash equilibrium, and is robust. The unifying idea is \emph{conditional commitment}: ``I commit to X conditional on you committing to Y.'' Conditional makes it unexploitable; commitment makes it legible. FairBot is exactly this.

\textbf{Commitment races.} Among consequentialists who can commit, there is an incentive to commit \emph{first}: the first mover can pick the point on the bargaining frontier best for itself and leave the responder to best-respond. So ``the best response is not the best response'', and agents race to commit before they even finish learning, risking incompatible lock-in or wasteful conflict (including threats and s-risks). Moving first ``in logical time'' is a genuine safety concern, which makes \emph{avoiding} commitment races a safety goal. The deeper obstacle is \emph{entanglement}: each agent optimizes against a distribution over the \emph{counterfactual} programs the other might submit, so punishing a Pareto improvement means forgoing utility against the opponent's \emph{actual} program; the incentive to make improvements is entangled with those counterfactuals, and this persists even with full conditional commitment.

\textbf{Safe Pareto improvements (SPI).} An SPI modifies the agents' default (conflict-prone) strategies so that \emph{every} player is guaranteed at least as well off regardless of the opponent, a guaranteed weak Pareto improvement (Oesterheld-Conitzer; DiGiovanni-Clifton-Macé). In the open-source/program setting, each player individually prefers to use renegotiation-based SPIs, and the guaranteed floor is the \emph{Pareto Meet Minimum} (your lowest efficient payoff), which is tight under a \emph{participation-independence} assumption. That assumption can fail (agents may become hawkish when conflict is cheap), the ``cheating'' problem. \emph{Entanglement-free SPI} (a research-frontier idea) restructures the interaction into two stages (exchange renegotiation programs, update on the actual one, then submit defaults), which breaks the entanglement and drops participation independence, at the cost of the tight floor. Teach the standard SPI result in full; present entanglement-free SPI briefly as the current frontier.

\section{The tournament}
Full rules are in the \href{https://iliad.au.pe/sessions/decision-theory/handout.html}{tournament handout}; the structure is summarized here so the day can be run from this document.

\textbf{The game.} The one-shot Prisoner's Dilemma, with the twist that your program can read the opponent's program before deciding (source is open). Payoffs (you, them): mutual cooperation \texttt{(C,C)} gives 3 each; mutual defection \texttt{(D,D)} gives 1 each; defecting against a cooperator gives 5 to the defector and 0 to the cooperated. This satisfies \texttt{T\ \textgreater{}\ R\ \textgreater{}\ P\ \textgreater{}\ S} (5 \textgreater{} 3 \textgreater{} 1 \textgreater{} 0) and \texttt{2R\ \textgreater{}\ T\ +\ S} (6 \textgreater{} 5).

\textbf{Two leagues.}

\begin{itemize}
\item
  \emph{League A (one-shot, open source).} Bots see the opponent's source and decide once. This is the program-equilibrium / Löbian-cooperation setting.
\item
  \emph{League B (iterated, history only).} Bots see the move history over a hidden number of rounds (around 100-200) and cannot read source. This is the classic reciprocity setting (Tit-for-Tat and relatives).
\end{itemize}

\textbf{Submitting a bot.} A submission may be in any form (real Python, pseudocode, or a careful English description) plus a short design rationale explaining the decision-theoretic approach; an LLM compiles it into the canonical \texttt{Agent} class:

\begin{verbatim}
class Agent:
    def move_oneshot(self, opp) -> Move:                  # League A
    def move_iterated(self, my_hist, opp_hist, t) -> Move: # League B
\end{verbatim}

In League A a bot may query: \texttt{opp.move\_against(SELF)} (the opponent's move against you), \texttt{opp.move\_against(DEFECT\_BOT)}, \texttt{opp.move\_against(COOP\_BOT)}, and \texttt{opp.source} (the opponent's canonical source). League B provides only the two history lists and the round index \texttt{t}.

\textbf{How circular queries resolve.} Two source-reading bots can each ask ``what does the other do against me?'', which is circular. Rather than simulate the recursion, the engine treats every ``what does the opponent do against X?'' query as a \emph{provability} question and computes the fixed point directly, the same Löbian move that lets two FairBots cooperate. When no stable resolution exists, it applies a \emph{no-wishful-thinking} rule: if cooperation cannot be established, default to D.

\textbf{Reference bots.} League A: CooperateBot, DefectBot, FairBot, PrudentBot, CliqueBot, RandomBot. League B: AllC, AllD, Random, TitForTat, GrimTrigger, GenerousTitForTat, Pavlov, TitForTwoTats. These seed the field and give students targets to beat or cooperate with.

\textbf{Scoring.} Round-robin: every bot plays every other bot, including a copy of itself, and ranking is by total points accumulated across all matchups (not head-to-head wins). League B matches add about 2\% move-flip noise and a randomized hidden length, so brittle strategies are penalized.

\textbf{What good looks like.} A strong submission embodies a clear decision-theoretic concept (commitment, transparency, the limits of self-reference, functional decision theory) and explains it, rather than chasing the leaderboard. The standout lesson is usually that FairBot-style conditional cooperation beats both naive cooperation (exploited) and naive defection (misses mutual cooperation) in League A, while robustness to noise dominates in League B.

\section{Learn more}
\textbf{Multi-agent and open-source game theory.} The \href{https://www.andrew.cmu.edu/user/coesterh/AnnotatedProgEqBibliography.html}{annotated program-equilibrium bibliography} (Oesterheld); the \href{https://arxiv.org/abs/2412.14570}{epsilon-grounded FairBot paper}; \href{https://www.lesswrong.com/posts/cLDcKgvM6KxBhqhGq/when-would-agis-engage-in-conflict}{When would AGIs engage in conflict?}; the safe-Pareto-improvement papers (Oesterheld-Conitzer; \href{https://arxiv.org/abs/2403.05103}{DiGiovanni-Clifton-Macé}). Andrew Critch's work on cooperative and uncooperative institution design is a good, pedagogically simple source for further exercises.

\textbf{Frontier.} Logical updatelessness and logical time; UDT 2 and policy-selection fixes; bargaining and notions of fairness (the Nash bargaining solution and the ``zoo'' of coalitional structures); entanglement-free safe Pareto improvements; reflective oracles and program equilibrium as a single picture.

\end{document}
