Principle Findings
We performed a series of analyses toward the end goal of validating our SI prediction model, SAIPH [5], in 2 cohorts that provided online survey data based on psychometrically validated scales with suicide questions, and in random samples of Twitter users. Both cohorts represented different demographics of the population, with the student survey cohort deriving from an undergraduate university cohort and the COVID-19 survey cohort deriving from individuals with an expansive range of ages throughout the life span. We were limited in our ability to perform a prediction analysis like our original publication due to the small number of high SI scores in both cohorts, as well as the fact that many individuals with high SI scores did not have a large body of tweets in the initial days before survey completion. While our supplementary results identified elevated AUC values across both independent cohorts, these derive from small sample sizes and must be interpreted with caution. As such, we performed linear modeling against the quantitative SI scores to increase the statistical robustness of our analysis and identified significant interactions of SAIPH scores with the number of tweets per day in both cohorts. The interactions demonstrate that SAIPH scores derived from a higher number of tweets on average are more robust. While the associations from the cumulative data from 50 days before the survey completion were significant in both cohorts, we performed a sliding window analysis to identify periods of peak association at periods more temporally proximal to the survey completion day. We observed a general agreement across both cohorts by periods, with more robust associations at approximately 2 weeks before survey completion and, to a lesser degree, on the day before survey completion. The overlapping significance across both cohorts around 2 weeks is consistent with our previous work that showed a higher frequency of high-risk scoring SAIPH scores preceded death by suicide at approximately 20 days before death, while the elevated signal just before the survey completion day is consistent with our previous work that demonstrated the highest predictive accuracy of SI admissions on Twitter using data most temporally proximal to the SI mention. All associations derived from integrating over 42 days of SAIPH scores were significant across both cohorts, which is consistent with the observation that the association with SI is more robust when there is a higher amount of tweet data from which to derive the score. These findings are consistent with our previous work that demonstrated that mean SAIPH score data from a multiple-day window performed better at predicting SI than that from smaller window periods [5]. Cumulatively, both the temporal patterns and observed associations were largely consistent across both cohorts, adding confidence that SAIPH is detecting SI-associated signals.
The SAIPH algorithm is developed on established suicide theories, integrating scores of intermediate suicide-associated constructs such as loneliness, stress, depression, and perceived burdensomeness, among others. Recent work by Ophir et al [6] has demonstrated the success of comparative models using the integration of suicide theory over merely leveraging a bulk of social media posts, suggesting the integration of these constructs may be important for model performance.
It is our view that it is important to consider the application of suicide risk–detecting tools such as SAIPH and others in the context of the challenges of the suicidal state and in what area of suicide prevention these tools may generate the most benefit. One major challenge of suicide risk prediction is the so-called “low base-rate problem” [16], wherein even the best-performing predictors will generate large numbers of false positives. In light of this major problem, it is our view that risk prediction methods could best be applied in contexts where the interventions are tolerable to both false positives and the prevention systems in which they are deployed. For example, it may not be appropriate to deploy in a clinical setting where the response to a false positive will use vital resources in short supply. By contrast, with the presence of future temporal risk predicting tools, one possibility becomes providing a just-in-time adaptive intervention, such as those leveraging digital data sources to prompt the timing of intervention [17], in the community at a time before suicidal crisis that is tailored such that it is tolerable to false positives. One example for which we envision SAIPH might be developed and ultimately applied would be prompting peer support interventions from a list of confidants (such as friends) selected by the at-risk individual, who could be prompted digitally to reach out at an individually set threshold of risk representing a period before the crisis. Such an intervention, if delivered before a period of crisis, would leverage the protective nature of connectedness without the need for mental health intervention training, where the “intervention” might be an invitation to social interaction such as talking, shared activities such as sports, hiking, or going out socially, for example. This may provide protective benefits while also being tolerable to false positives, would be scalable, and could be evaluated for efficacy.
Our temporal analysis in individuals from the COVID-19 survey cohort demonstrated that variation of average daily SAIPH values within people was associated with variation in SI scores over time. This association was strongest when evaluating survey metrics repeated in shorter time frames. We posit that longer periods between surveys may have periods of higher or lower distress that might be marked by variations in SAIPH score that are not captured by survey time points. This supposition is supported by the observation that limiting measurements to those taken on the day of survey completion despite longer periods exhibits a trend toward positive association with the slope of SI changes over the period. Taken together, SAIPH scores may be capable of acting as an indicator of changing SI over time.
We evaluated the ability of SAIPH scores to mark changes in perceived stress, anxiety, and depression. Within participants, the strongest association we observed was with perceived stress. The ability of SAIPH to mark changing perceived stress may be linked with its association as an indicator of changing SI within participants. Notably and consistent with this hypothesis, we observed significant deviations of high SI endorsing individuals with low SI or control individuals as a function of the declaration of a national emergency and the beginning of social distancing due to the start of the COVID-19 pandemic in the middle of March 2020 (Section S5 in Multimedia Appendix 1). Based on the stress diathesis theory, our interpretation is that vulnerable individuals may have experienced higher distress as a function of this shared stressor relative to more resilient controls. This analysis demonstrates the potential efficacy of using SAIPH as an indicator of SI risk and perceived stress to study temporal trends relative to major events, both on an individual and a population basis. This work is consistent with our previous work that demonstrated that averaging of regionally and randomly sampled SAIPH scores from certain areas was associated with suicide death rates in those regions [5].
In light of our data demonstrating that we could use SAIPH as an indicator of changing SI, this allowed us the opportunity to evaluate the effect of responses to suicidal mentions in 2 cohorts. The strongest result we observed was that individuals who had responses to their suicidal mentions demonstrated a significant improvement in their SAIPH scores over the following 3 weeks, suggesting that responding to those in crisis may contribute to their improvement. This result is intuitive and helps corroborate the ability of SAIPH to act as a proxy of SI trajectory. We next attempted to look for a more nuanced scoring method for the responses to qualitatively score them relative to the individual. The impetus for this theory came from the ASIST suicide intervention that teaches that identification of factors important to the individual is a key step in getting the individual with SI to agree to be safe for the moment. As such, our algorithm effectively measures content in their historical posts and builds a new model relative to their most extreme high and low content. If a response to this person mentions something that was positively reflected in their timeline, it will receive a score denoting it is a “positive” response. By building machine learning models specific to the individual, we demonstrated that we could score the response to their suicidal mention and that those with low scores, denoting a response scored as “positive” to the user with a suicidal mention, were more likely to improve. In our analysis of words and 2-word combinations in responses to those with suicidal mentions, no words appeared overrepresented in those individuals who improved over the following 3 weeks as measured by SAIPH. This may be because, as hypothesized, the responses important for someone to improve contained more contextual and personalized content relevant to the individual and, as such, were not overrepresented across the group. By contrast, words such as “don’t” and phrases such as “please don’t,” “do it,” “everything will,” or “be alright” were overrepresented in responses to those with suicidal mentions that did not improve over the 3-week period. This finding suggests that more standard responses may lack the personal context important for an individual’s improvement. These findings, if corroborated, could lead to the development of a tool allowing for quick scanning and identification of topics of importance to a crisis line user who consents to social media scanning. In combination, we envision a possible application of both the SAIPH method and personalized importance metric that could be used by crisis line responders where, through the future integration of large language models and these scores, would allow responders to interrogate the user’s risk and protective factors preceding and following a user’s historical periods of highest risk to gain insight into the person in need. As individuals with SI in crisis can experience cognitive constriction [18-20] and are not always in a mindset for collaborative problem-solving with the crisis responder, these tools could augment suicide intervention efforts, safety planning, and problem-solving and should be further developed. Beyond prompting an ideal response, the data suggest that any response is better than no response. These findings, while interesting, are preliminary and should be further replicated in larger datasets to validate them and further explore their implications.
Limitations
This study is not without limitations. The SI metrics, while collected as part of psychometrically validated surveys, represented answers to a single question and not, by contrast, the cumulative scores from comprehensive suicide risk scales. Despite this, these metrics are stronger than suicidal mentions on social media as a measure of suicide risk for external validation, as they represent data collections occurring independently from a participant’s social media use and are answers to a standardized set of questions. This fact partially addresses the legitimacy that suicidal mentions on social media are linked to actual suicidal intent as demonstrated in 2 studies from Japan [2,3], and by the fact that our model performs best for identifying BDI SI level 3, which states a desire to kill oneself if possible and thus suggests a level of intent. Despite this, the number of individuals endorsing high levels of SI in both survey cohorts was small and, in some cases, the amount of Twitter data available at the times closest to survey completion was lacking, necessitating the creation of our imputation algorithm. Future work should investigate the efficacy of data imputation using our future predicting SVR method to assess if this improves the robustness of SI prediction in individuals with less data. The primary strategy used involved looking for associations with quantitative SI scores using linear models. The use of the sliding window approach allowed us to investigate the times of most significant association; however, a limitation of this analysis is that we did not correct for multiple testing. Data gathered for this study occurred during the COVID-19 pandemic, which represented a rather anomalous year where in general, people’s usual routines, external influences, and environments were disrupted. In the context of the stress diathesis model of suicide risk, this may have caused a higher efficacy of our prediction methods and observed associations than would not have been observed in more normal times. Our analysis of SAIPH scores before and after this period indicated that the pandemic did seem to affect SAIPH scores in individuals at high suicide risk. Alternatively, it remains possible that the pandemic elevated SAIPH scores in individuals who are not suicidal and led to a weaker observed predictive accuracy than would have otherwise been observed in nonpandemic times. Despite this, the success of SAIPH in predicting SI during pandemic times suggests it is an efficacious tool for suicide risk prevention that is robust in the context of external stressors. Future work should investigate if more personalized approaches, leveraging an individual’s own data to prognosticate risk, may improve predictive performance.