Additional robustness analyses confirm that complex societies precede moralizing gods throughout world history

Beheim et al. conclude based on a reanalysis of our publicly available data and code that our title finding that “complex societies precede moralizing gods throughout world history” reverses when correctly analyzed. I am honoured to have such a distinguished group of scholars engage deeply with our research. But their reanalyses contain a crucial flaw in how they deal with known historical expansions of moralizing gods through conquest or diffusion from already complex societies. When this flaw is accounted for, their analyses in fact confirm the robustness of our central finding. More generally, this episode highlights the power of open science to allow rapid exchange of differing opinions outside the traditional journal publication process, but also highlights the need for caution before jumping to conclusions based on complicated analyses that have not undergone peer review.

Thumb patricksavageheadshot
May 05, 2019
3
1

I am currently in the middle of a 10-day Japanese public holiday with no day care for my children, so I am not going to make any pretense at doing a full formal rebuttal to Beheim et al.’s extremely detailed 24-page [Ed: corrected from initial typo ("32-page")] preprint published days ago (Beheim et al., Preprint). But given the media storm surrounding their publication (which has not yet undergone peer review), I think it’s important to quickly respond to the main claim that Beheim et al.’s analysis “corrects” flaws in our original analysis (Whitehouse et al., 2019) and that, when corrected, our main findings are reversed such that “moralizing gods precede complex societies”. This claim is inaccurate and largely based on incorrect historical assumptions. Although I have huge respect for all of the authors of Beheim et al.’s preprint, I cannot agree with their conclusions in this particular case.

“Reversal” of findings through forward bias analysis only appear due to incorrect historical assumptions

The key is this: Beheim et al.’s main claim is that our analyses fail to address “forward bias”, meaning that the date at which beliefs in moralizing gods are recorded in writing may postdate their actual appearance. To correct for this possibility, they reanalyze our data assuming that moralizing gods actually appeared one century earlier, and find that this reverses the main finding such that moralizing gods actually precede complex societies. If this were true, it would suggest that our finding indeed might not be robust (although this would not in fact constitute evidence to support their title claim that “moralizing gods precede complex societies”). But there is a crucial problem: in 6 out of the 12 regions on which this reanalysis is performed, forward bias is not an issue, because we already know how complex societies with moralizing gods appeared in these regions: via historically documented expansions of already complex neighbouring societies with beliefs in moralizing gods. Thus, their reanalysis assumes that in each of these 6 regions (located in modern Pakistan, Uzbekistan, India, Mali, Mongolia, and Japan), religious beliefs somehow preceded the expansions that brought those beliefs. In other words, the analysis assumes that Zoroastrianism was actually present in Pakistan 100 years before it was brought there by the Achaemenid Empire, that Buddhism was actually present in Japan 100 years before it was brought from mainland East Asia, etc. These are simply not logical assumptions to make. Obviously if you mistakenly assume that beliefs appear 100 years before the complex societies that brought those beliefs, then your analysis will mistakenly find that moralizing gods precede complex societies.

So, what happens if we restrict the robustness analysis of forward bias to only the 6 societies in which moralizing gods arose de novo in which forward bias might reasonably be a concern? Beheim et al. performed but appear not to report these reanalyses, so I have summarized them in Table 1. The first column is data we already reported in Extended Data Table 1 from our original paper. The other columns represent the same data when Beheim et al.’s robustness analyses are performed to examine the effect of possible forward bias in the 6 regions for which this is a reasonable concern. This table shows that potential forward bias of 100 years does not “reverse” the direction of findings for any of these six regions, 200 years reverses the findings for 1/6 regions, 300 years reverses the direction for 2/6 regions, and a minimum of 500 years of forward bias would be required to qualitatively reverse half (3/6) of the regions for which forward bias is a potential concern. Half a millennium is an extremely large margin of error, suggesting that our results are highly robust! 

To summarize, in half of the 12 regional analyses performed, we know for a historical fact that complex societies preceded the introduction of beliefs in moralizing gods, and beliefs in moralizing gods would have had to precede their written documentation by at least half a millennium to qualitatively change our findings for even half of the remaining 6 regions. In summary, the robustness analyses kindly performed by Beheim et al. have confirmed that our results are robust to well-known limitations of historical analysis.

Reanalysis of logistic regression without inferring absence confirm the original finding of an association between moralizing gods and complex societies

Beheim et al. make much of the fact that our logistic regression sub-analysis made the assumption that moralizing gods were absent until we have evidence of their presence (i.e., we convert NAs to 0s for this sub-analysis – and only this sub-analysis), and claim that removing this assumption again “reverses” our findings. But in fact our logistic regression analysis was not performed to test for the direction of causality between moralizing gods and complex societies but simply whether an association existed in the first place after controlling for temporal, spatial, and historical autocorrelation. As Beheim et al. admit, it is essentially impossible to account for temporal autocorrelation through our logistic regression analysis without making this assumption. But it is true that we could have done an alternative analysis that ignores temporal autocorrelation and focused only on spatial and historical autocorrelation for historically or ethnographically documented societies where the presence or absence of moralizing gods was explicitly known. Indeed, when Beheim et al. perform such an analysis, they find that the association between moralizing gods and complex societies remains significant, qualitatively confirming – not reversing – the results of our initial logistic regression analysis. In other words, another reanalysis by Beheim et al. confirms the robustness of our initial logistic regression sub-analyses. It is reasonable to debate the statistical issues here, but the fact that multiple alternative analyses by rival teams confirm this finding is strong evidence for the robustness of our finding of an association between moralizing gods and complex societies. 

Minor differences of interpretation regarding historical evidence have no effect on our conclusions

In their preprint - and more extensively in a separate preprint that has been provisionally accepted for publication in the Journal of Cognitive Historiography (Slingerland et al., In press), which will eventually be published in tandem with our own rebuttal (Whitehouse et al., in prep) – Beheim et al. take issue with a number of specific historical details in our database. But these details have no functional effect on the main findings of our analysis. For example, they take issue with the nature of source material used to support codings of ritual frequency in post-Shang China, even though these codings have no effect on any of our analyses. Likewise, they acknowledge that our coding of the first evidence of moralizing gods in W. Zhou China (1100BCE) is supported by the reference we use, but argue that they have experts who would place this 100 years earlier in Late Shang China. Late Shang China was almost as complex as W. Zhou China (0.70 vs. 0.71, respectively, in our analysis, both well above our “megasociety threshold” of 0.6). Slingerland et al. (In press) claim that  reanalysing our data with this modification “would seriously undermine Whitehouse et al.’s conclusion”, but in fact it has no qualitative effect on our findings: rates of change in social complexity in China remain significantly greater BEFORE the first evidence of moralizing gods than after (p < .001, t = -3.7, n = 20 time windows), even if we give their interpretation the benefit of the doubt. In no case do Beheim et al. identify any substantive errors in our codings that qualitatively affect our main findings.

Conclusion

In summary, the statistical reanalyses performed by Beheim et al., when corrected for incorrect historical assumptions, confirm the robustness of our main finding that complex societies precede moralizing gods throughout world history. I appreciate their engagement, which in my view has only confirmed the robustness of our initial findings. Beheim et al. raise many additional issues that will be addressed in future responses, including broader theoretical questions like how best to define “morality”, methods for inferring prehistoric beliefs in the absence of historical evidence, appropriate engagement with expert historians, etc.  Certainly the issue of interpreting absence of evidence in historical studies is a complex one for which there remains no perfect answer (Currie et al., 2018).

Since Beheim et al.’s preprint has not yet undergone peer review, I trust that they will account for these issues in future revised version of this reanalysis, and I caution those on social media and elsewhere not to leap to any conclusions until their preprint has passed rigorous peer review and we have published our own rebuttals (e.g., Whitehouse et al., in prep.). I look forward to continuing constructive – and hopefully friendly! – scholarly debate on this topic.

References:

Beheim, B., Atkinson, Q., Bulbulia, J., Gervais, W., Gray, R. D., Henrich, J., … Willard, A. K. (Preprint). Corrected analyses show that moralizing gods precede complex societies but serious data concerns remain. PsyArXiv preprint. http://doi.org/10.31234/osf.io/jwa2n

Currie, T. E., Turchin, P., Whitehouse, H., François, P., Feeney, K., Mullins, D., … Spencer, C. (2018). Reply to Tosh et al: Quantitative analyses of cultural evolution require engagement with historical and archaeological research. Proceedings of the National Academy of Sciences of the United States of America, 115(26), E5841–E5842. http://doi.org/10.1073/pnas.1807312115

Slingerland, E., Monroe, M. W., Sullivan, B., Walsh, R. F., Veidlinger, D., Noseworthy, W., … Spicer, R. (In press). Historians respond to “Complex societies precede moralizing gods throughout world history.” Journal of Cognitive Historiography. Preprint: http://doi.org/10.31234/osf.io/2amjz

Whitehouse, H., François, P., Savage, P. E., Currie, T. E., Feeney, K. C., … Turchin, P. (In prep.). Historians, archaeologists, anthropologists, and cultural evolutionary scientists respond to Slingerland et al. (2019). Journal of Cognitive Historiography.

Whitehouse, H., François, P., Savage, P. E., Currie, T. E., Feeney, K. C., … Turchin, P. (2019). Complex societies precede moralizing gods throughout world history. Nature, 568, 226–229. http://doi.org/10.1038/s41586-019-1043-4

Medium patricksavageheadshot

Patrick Savage

Associate Professor, Keio University

1 Comments

Thumb default avatar
Bret Beheim 2 months ago

Hello Dr. Savage,


As you say, our response is long, and that is not the key finding.

Rather, the key finding is that nearly 2/3rds of your outcome data were originally unknown values that were recoded as negative responses, i.e. "moralizing gods absent". I have to think this was accidental, as it was only indirectly done in the R code, is very serious, and not mentioned anywhere in the text. Of course, since the populations with unknown values in Seshat are almost always small, pre-literate societies, this choice entirely determines the headline conclusion. It underpins the forward bias problem, the conquering polity problem, and the massively overconfident regression results. Our reanalysis simply shows that, without this strange assumption, the Seshat data support the opposite conclusion.

Sorry for impinging on Golden Week!