Why isn't there more Neanderthal DNA in the modern gene pool? Because they were outcompeted by Homo Sapiens - Homo Sapiens exhibited behaviours that made their groups more cohesive (like religion). In other words, genes that coded for behaviours only beneficial at the level of the group aided the survival of individuals by ensuring they were part of successful groups. Individually, Homo Sapiens were overmatched by Neanderthalensis; they only succeeded because of their group behaviours.
This isn't a mechanism. Here, lets look at this from a different perspective.
The reproductive success of humans undoubtedly depends in part on the fate of their groups. If a group is annihilated, all the people in it, together with their genes, are annihilated. If a group acquires territory or food or mates, the windfall will benefit some or all of its members. But this brings us to the familiar problem which led most evolutionary biologists to reject the idea of group selection in the 1960s. Any genetic tendency to risk life and limb that results in a net decrease in individual inclusive fitness will be relentlessly selected against. A new mutation with this effect would not come to predominate in the population, and even if it did, it would be driven out by any immigrant or mutant that favored itself at the expense of the group.
Let's take the concrete example of collective aggression. Often the benefits to the self and to the group may coincide. A warrior may scare off a party of attackers and save the lives of his fellow villagers together with the lives of himself and his family. In other cases the benefits may diverge: the warrior may stay at the rear, or sneak off to the side, and let everyone else fight. In still others the outcome may be uncertain, but because selection works on probabilities, he may play the odds, say, taking a one-in-ten chance of getting killed in a raid that promises a one-in-two chance of abducting a few extra wives. We should expect selection to favor traits that maximize the individual's expected reproductive output, given these tradeoffs.
What we
don't expect to see is the evolution of an innate tendency among individuals to predictably sacrifice their expected interests for the interests of the group—to cheerfully volunteer to serve as a galley slave, a human shield, or cannon fodder. Take the extreme case of a gene that impelled a person to launch a suicide attack that allowed his group to prevail over an enemy. That is hardly a gene that could be selected! (I'll put aside for now the potential benefits to the suicide warrior's kin.) What could evolve, instead, is a tendency to manipulate
others to become suicide attackers, and more generally, to promulgate norms of morality and self-sacrifice that one intends to apply in full force to everyone in the group but oneself. If one is the unlucky victim of such manipulation or coercion by others, there's no need to call it altruism and search for an evolutionary explanation, any more than we need to explain the "altruism" of a prey animal who benefits a predator by blundering into its sights.
Thus we have a nice set of competing empirical predictions for any examples of group-benefiting self-sacrifice we do observe in humans. If humans were selected to benefit their groups at the expense of themselves, then self-sacrificial acts should be deliberate, spontaneous, and uncompensated, just like other adaptations such as libido, a sweet tooth, or parental love. But if humans were selected to benefit themselves and their kin in the context of group living (perhaps, but not necessarily, by also benefiting their groups), then any guaranteed self-sacrifice should be a product of manipulation by others, such as enslavement, conscription, external incentives, or psychological manipulation.
To be sure, if we go back to group selection as an explanation of
group traits, particularly cultural ones, then it's easy to see how a group that successfully coerced or manipulated a renewable supply of its own members to launch suicide attacks might expand relative to other groups. But that would have nothing to do with its members' inherited psychology, in this case, their willingness to sacrifice themselves without manipulation. The same is true for less extreme sacrifices.
