Thinking About Collaborators

Replication of Experiment 1 in Xiang et al. (2023)

Jacob C. Zimmerman

j2zimmerman@ucsd.edu

UC San Diego

December 1, 2025

Collaboration is fundamental to humans

• Fundamental to human success is how we’ve evolved to collaborate effectively (Tomasello et al., 2005)
• Collaboration requires judgments of what potential collaborators could bring to the table, and how much effort they will allocate
• How complex are these judgments?

Collaboration is fundamental to humans

Fundamental to human success is how we’ve evolved to collaborate effectively. There are many cognitive processes involved in this.

For example, in order to make decisions about who to work with, we make judgments of what potential collaborators could bring to the table, and how much effort they will allocate

As usual, evolution prefers the simplest solution to problems: so how complex is our solution?

How do we judge what others could bring to the table?

• We observe them work, to update our beliefs about them (Bayesian inference) (Premack & Woodruff, 1978)
  • Belief: how competent they are (Jara-Ettinger et al., 2015)
  • Belief: how much effort they are putting in, given their competence and how much incentive there is (Jara-Ettinger et al., 2016)

So how do we judge what others could bring to the table?

Well, we observe them working to update our beliefs about them; Bayesian inference is a good candidate for how we adjust our beliefs given observations of the world

Some of our beliefs are about how competent they are

and how much effort they are putting in

While we also infer the difficulty of the task, Xiang et al. (2023) is seeking to integrate the independent role of perceived competence.

How do we judge what others could bring to the table, and how much effort they will allocate?

• Simple models of effort allocation: we could think that…
  • Social compensation: people assume others won’t apply any effort, so they allocate effort as though they were working alone (Williams & Karau, 1991) (solitary model)
  • Social loafing: people assume others will apply maximum effort, so they allocate only enough effort to help them (Karau & Williams, 1993; Latané et al., 1979) (compensatory model)
• Xiang et al. (2023) propose a more complex model: we think that…
  • Jointly optimize effort: people infer how much effort others will allocate and how much the others think they will allocate, and then calibrate their effort accordingly – seeking to optimize their combined effort (joint effort model)

After we infer each other’s competence and effort from observing their behavior, how do we judge how they much effort they will allocate?

There are a couple simple solutions that evolution could have chosen for how we calibrate our own effort -> <read slide>

“Optimize combined effort” means calibrate the combined effort to be enough to achieve the task (and for effort also not to be unfairly allocated)

We make some assumptions to make the inferences in the proposed model tractable (Jara-Ettinger et al., 2016): 1. If they succeed, they must have at least some competence and have put some effort in 2. If they are competent enough to succeed, they will allocate the minimum effort needed given their competence 3. Effort is costly, so if they succeed with low incentive, they must really desire the reward (i.e., reward has high utility to them) 4. They discount unequal effort allocation (i.e., they care somewhat about fairness) 5. (Practical assumption) Competence does not change across observations 6. (Practical assumption) Competence and effort each have a defined range (0% to 100%)

Illustrating the basic framework of inferences:

We infer a person’s competence and effort simultaneously by internally modeling their mutual contingencies, and the influence of incentive:

1. Effort turns competence into success

• e.g., if they succeed, they must have some competence and have put some effort in
• Success means combined competence and effort is at least X

2. They don’t put in more effort than necessary, given their competence

• e.g., if they are competent enough to succeed, they allocate the minimum effort needed given their competence
• Success given their competence means at most Y effort

3. Effort is motivated by incentive

• e.g., effort is costly, so if they succeed with low incentive, they must really desire the reward (i.e., reward has high utility to them)
• Success means at least Z desire for the reward and the minimum W effort that this motivates

Xiang et al. (2023): Lift That Box!

• Xiang et al. (2023) used a game-like task to collect participants’ judgments of competence, effort, and likelihood of success of virtual collaborators, in order to see how well the joint effort model predicted these judgments

The authors then sought to validate their more complex model

<Read slide. Then demo the task and point out the rounds 1-2 are individual, and round 3 is joint activity>

So they had many different results, but let’s take a look at the key ones I was interested in replicating

Xiang et al. (2023): Results

1. Participants thought that collaborators can succeed together even if they had always failed alone (\(t(49) = 10.42\), \(p < .001\); \(d = 1.47\))
2. Participants also thought that collaborators’ chance of succeeding together increased with more individual successes (i.e., collaborators inferred each other’s competence and effort from prior attempts)
3. Only the proposed joint effort model predicted both:

(Preview) Notice F,F;F,F, the left-to-right trend, and that joint effort fits the best.

<Read slide>

This evidence suggests that their model is worth the complexity!

Also, they found that the increase was gradual (i.e., collaborators tried to distribute effort fairly)

Replicating Xiang et al. (2023): Power

• I estimated that we needed only 9 participants (as this would give us 97.05% power for the main effect)
• However, to replicate the qualitative pattern, in the absence of a good expectation of how many participants were necessary for this, I chose to collect the full 50 participants
  • As a side-effect, this will also enable future exploratory analysis of the dataset, which I’ll briefly describe later

<Read slide>

Replicating Xiang et al. (2023): Modeling

• For the computational model, I chose to reimplement the model in memo, a new probabilistic programming language (Chandra et al., 2025) for performance and extensibility advantages (for future work)
  • Needed to validate that new model had comparable results
  • Worked through distinguishing trivial implementation differences from implementation mistakes
  • Accomplished this by changing one implementation variable at a time and measuring its impact to determine where discrepancies came from

I expected the basic effect to replicate since there’s high signal to noise in the measure (high t-value and low SE),

and I was 4/5 confident the qualitative pattern would replicate.

<Read slide>

Worked through distinguishing trivial implementation differences (e.g., going from a continuous prior to a discrete prior) from meaningful implementation discrepancies (e.g., errors in the model implementation)

How I changed one variable at a time in modeling: webppl MCMC -> webppl enumerate -> memo enumerate

Replication Results: Confirmatory

Figure 3C (Behavioral Only). Error bars show bootstrapped 95% CIs.

Collaborators can succeed together even if they have failed alone (\(t(49) = 10.42\), \(p < .001\); \(d = 1.47\))

Replication (Behavioral Only). Error bars show bootstrapped 95% CIs.

Replicated! (\(t(49) = 9.26\), \(p < .001\); \(d = 1.31\))

<Read slide, explaining plots>

Replication Results: Exploratory

Figure 3C. Model simulations averaged across 10 iterations. Error bars show bootstrapped 95% CIs.

Collaborators’ chance of succeeding together increased with more individual success, validating joint effort model

Replication. Model simulations use strengths in steps of .03. Error bars show bootstrapped 95% CIs.

Replicated!

<Read slide, explaining plots>

In sum, they replcicated that joint success is possible even if there is no prior evidence of individual success, AND joint success likelihood increases as cases of individual success increases. In conjunction, these results support the joint effort model of reasoning about collaborators’ effort

Discussion

• Replication success was expected due to large effect size and clear pattern
• I observed slight differences:
  • Larger CIs
  • Flat section of the curve (i.e., similarity between two scenarios) not seen before
  • Effect size (d) was lower than original, as should be expected

<Read slide>

Flat part could be explained by the safe joint effort model (can show the figure that includes the safe joint effort model)

Discussion: Next Steps

• To validate the joint effort model further, I will quantify the model fits (and confirm with new data), modeling within-participant variability independently from variability due to scenario (outcomes of rounds 1 and 2)

Figure 3C Replication with participant-level data.

Now some next steps…

<Read slide>

We can see that there may be variance or even heterogeneity across participants that could be modeled; I’d like to estimate model fits, explicitly modeling for the between-participants variability, using model-likelihood measures for each participant then aggregating across participants.

Also, Maybe the greater variance in the data (driving the larger CIs) is due to this heterogeneity that might not have been present in the original data. Or it could just be due to publication bias/effect inflation.

Participant-level variability can be seen as amplified, diminished, or biased probability judgments across scenarios that could be attributed to the participant rather than to the scenarios; we see that there could be a lot of this variability by looking at the underlying participant-level data.

One way I might quantify the model fits with both types of variability is to estimate the likelihood of each participant’s data under each model then summarizing across participants to estimate the likelihood of each model.

Discussion: Task

• Some limitations:
  • The task is not very naturalistic (limits ecological validity)
  • The participant isn’t a collaborator themselves (limits construct validity)
• Future steps:
  • Rich textual vignettes that paint a fuller scene than just stick figures before and after they attempt to lift boxes
  • Include the participant as a collaborator, e.g., with their own incentive related to the virtual collaborators’ outcome

Last,

I want to acknowledge describe some limitations of the task -> <read slide>

Thank you!

Questions?

References

Chandra, K., Chen, T., Tenenbaum, J. B., & Ragan-Kelley, J. (2025). A domain-specific probabilistic programming language for reasoning about reasoning (or: A memo on memo). Proc. ACM Program. Lang., 9(OOPSLA2). https://doi.org/10.1145/3763078

Jara-Ettinger, J., Gweon, H., Schulz, L. E., & Tenenbaum, J. B. (2016). The na&#xef;ve utility calculus: Computational principles underlying commonsense psychology. Trends in Cognitive Sciences, 20(8), 589–604. https://doi.org/10.1016/j.tics.2016.05.011

Jara-Ettinger, J., Gweon, H., Tenenbaum, J. B., & Schulz, L. E. (2015). Children’s understanding of the costs and rewards underlying rational action. Cognition, 140, 14–23. https://doi.org/https://doi.org/10.1016/j.cognition.2015.03.006

Karau, S. J., & Williams, K. D. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65(4), 681–706. https://doi.org/10.1037/0022-3514.65.4.681

Latané, B., Williams, K., & Harkins, S. (1979). Many hands make light the work: The causes and consequences of social loafing. Journal of Personality and Social Psychology, 37(6), 822–832. https://doi.org/10.1037/0022-3514.37.6.822

Premack, D., & Woodruff, G. (1978). Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4), 515–526. https://doi.org/10.1017/S0140525X00076512

Tomasello, M., Carpenter, M., Call, J., Behne, T., & Moll, H. (2005). Understanding and sharing intentions: The origins of cultural cognition. Behavioral and Brain Sciences, 28(5), 675–691. https://doi.org/10.1017/S0140525X05000129

Williams, K. D., & Karau, S. J. (1991). Social loafing and social compensation: The effects of expectations of co-worker performance. J Pers Soc Psychol, 61(4), 570–581.

Xiang, Y., Vélez, N., & Gershman, S. J. (2023). Collaborative decision making is grounded in representations of other people’s competence and effort. J. Exp. Psychol. Gen., 152(6), 1565–1579.