Matrix games between full siblings in Mendelian populations

J József Garay T Tamás Varga V Villő Csiszár T Tamás F. Móri A Andras Szilagyi (Systems Biology of Reproduction Lendulet Group, Institute of Molecular Life Sciences, Hungarian Research Network (HUN-REN) Research Centre for Natural Sciences)

Abstract

We develop a model that integrates evolutionary matrix game theory with Mendelian genetics. Within this framework, we define the genotype dynamics that describes how the frequencies of genotypes change in sexual diploid populations. We show that our formal definition of evolutionary stability for genotype distributions implies the stability of the corresponding interior equilibrium point in the genotype dynamics. We apply our findings to a model of familial selection, where the survival rates of siblings in monogamous families are determined by a matrix game between them. According to Mendelian inheritance, the behaviour associated with each genotype is uniquely determined by an autosomal (recessive-dominant or intermediate) allele pair. We provide general conditions for the evolutionary stability of homozygote populations. We find that the payoff matrix and the genotype-phenotype map together determine this stability. In numerical examples we consider the prisoner’s dilemma between siblings. Based on the evolutionary stability of the pure cooperator and defector states, we provide a potential classification of the genotype dynamics. We distinguish between two cases: one in which the total survival rate is higher in cooperator-cooperator interactions (“coordinated” case), and another in which it is higher in cooperator-defector interactions (“anti-coordinated” case). In the coordinated case, global stability of cooperator homozygote population is possible but not necessary, since bistability, stable coexistence of cooperators and defectors, and even global stability of the defector homozygote state are all possible, depending on the interaction between the phenotypic payoff matrix and the genotype-phenotype mapping. In the anti-coordinated case, the cooperator homozygote population cannot be stable. Thus, similarly to the group selection theory, the welfare of the family (the sum of the survival rates of siblings) governs the emergence of cooperative behavior among family members. Finally, in the case of the donation game, the classical Hamilton’s rule determines whether the homozygous cooperator or the homozygous defector population is stable; bistability or stable coexistence are impossible.

Article Details

Journal PLoS ONE
Volume / Issue Vol. 20, Issue 9
Published September 10, 2025
Pages e0331044
ISSN 1932-6203
Publisher Public Library of Science

Journal Info

PLoS ONE

Public Library of Science

ISSN: 1932-6203 Open Access Health Sciences

Authors (5)

J

József Garay

T

Tamás Varga

V

Villő Csiszár

T

Tamás F. Móri

A

Andras Szilagyi

Systems Biology of Reproduction Lendulet Group, Institute of Molecular Life Sciences, Hungarian Research Network (HUN-REN) Research Centre for Natural Sciences