Shooting Straight: Appendices

View the Original Report

• Shooting Straight: What TV Stories Tell Us About Gun Safety, How These Depictions Affect Audiences, and How We Can Do Better

Appendix A: Content Analysis Methodology

Sample Selection

The sample was selected by identifying the 20 most-watched scripted TV dramas from network and cable TV and the five most-watched streaming dramas during each season (2019-2020, 2020-2021), as identified by Nielsen ratings. Unscripted, news, children’s, sports content, and comedies were excluded, as were TV shows set in a time period before 1980 or in the future. The genres of sampled TV shows for each season are shown in Table 1.

Genres of sampled TV shows	2019-2020 (Season 1)	2020-2021 (Season 2)
Police Dramas	9	9
Medical Dramas	4	4
Supernatural Dramas	4	2
Miscellaneous Dramas	8	10

For each series, we randomly selected five episodes from each season. This selection process resulted in a total of 250 episodes (125 per season) from 33 unique series.¹The full sample of shows and episodes is listed in Appendix C.

Coder Training and Interrater Reliability

We trained a total of 16 undergraduate and graduate students from University of Southern California to watch selected episodes and code them for relevant content. The same training process was used for both seasons. For each season, coding followed a two-month training period that included several rounds of testing and refining the codebook and the training procedures.

After an introductory training session where the codebook was reviewed, all coders analyzed the same episode, randomly selected from our sample of police drama episodes. Inconsistencies in responses were identified and discussed, resulting in minor revisions to the codebook. Another episode from the same series (The Rookie) was then randomly selected and assigned, followed by the same discussion and revision process.²These two initial training episodes were not included in the final reliability analysis.

Next, we assigned 10% of the sample (13 episodes) to be coded by two coders to establish interrater reliability. We deliberately included 10 episodes from police dramas in this sub-sample in an effort to maximize reliability on gun-related items from the outset. Items that achieved adequate reliability were retained in the codebook unchanged.³A Cohen’s Kappa or Krippendorf’s Alpha value of 1.00 indicates perfect reliability (100% agreement). Scores between .80 and .99 are considered strong. Scores between .60 and .79 are considered borderline, and scores below .59 are consid- ered inadequate. Once the codebook was finalized, the remaining 112 episodes from each season were divided among coders. Twelve additional episodes were randomly selected for double-coding. The total of 25 double-coded episodes were used for final reliability calculations.⁴The full codebook with reliability analysis can be found in Appendix D. Unless otherwise noted, items that achieved inadequate reliability are not included in this report. Some items with inadequate reliability were retained as qualitative findings if there were very few cases, but these numbers should be interpreted with caution.

Coding Procedure

Coders were instructed to exclude episode recaps and title sequences from the analysis. At the episode level (N = 250 across both seasons), coders identified the presence and absence of safe and unsafe gun practices, such as unsecure gun storage, gun use under the influence of drugs or alcohol, shooting into the air, and leaving a loaded gun unattended. They also identified the ways in which characters acquired guns, when identified, and the geographic location where gun violence took place. Finally, we identified how often gun laws were referenced, and how often discussions about gun statistics, safety, and social problems appeared in episodes.

At the character level, coders analyzed the characteristics of those who held or used guns (N = 812) or were shot (N = 235). These included demographics, role size, whether they were law enforcement officers or military personnel, and whether they faced mental health issues, disabilities, or addiction issues.

Appendix B: Experimental Study Methodology

Sample

Survey recruitment and data collection was administered online by Qualtrics Panels, a third party aggregator of market research panels. Respondents were required to be 18 or older and living in the U.S. In addition to general population sampling, we set up quotas to recruit a minimum of 500 gun-owners and 500 parents with children under the age of 18. Our total sample consisted of 3,358 American adults.

Participants were ideologically and geographically diverse:

• Roughly half (49%) reported having a gun in their household and one third (37%) had children under the age of 18.
• The majority were white/Caucasian (85%), followed by Black/African- American (8%), Hispanic/Latinx (7%), and Asian/Asian-American (2%).⁵Participants could select multiple races/ethnicities when answering this question; as a result, percentages total more than 100.
• They tended toward moderate ideology (37%), as opposed to very liberal (9%) or very conservative (14%).
• Nearly half (49%) reported living in a small city or suburban area, followed by 26% in a rural area, and 24% in a large city or urban area.

Procedure and Measures

After answering screening questions (age, gun ownership, parent status) participants were randomly assigned to view one of three 9-13 minute clips cut from longer storylines:

• Grey’s Anatomy (N = 1,072): The episode “Trigger Happy” (2016) addresses the consequences of an accidental shooting that occurs after a child accesses a weapon at home during a playdate.
• New Amsterdam (N = 1,205): The episode “Perspectives” (2020) focuses on a child who is experiencing symptoms of trauma after participating in an active shooter drill with simulated gun violence at school.
• Chicago Med (N = 1,080): A multi-episode arc from the 2020-2021 season involves a foster child facing a medical emergency. This storyline was selected as the control condition because, like the gun storylines, it is from a medical drama and involves a child, but does not include any gun-related content.

All participants, regardless of which storyline they viewed, were then asked about their attitudes and knowledge towards gun-safety measures and policies, responses to the storyline, and demographic characteristics.⁶See Appendix E for survey items.

Analytic Strategy

Data were downloaded from Qualtrics into SPSS v28 for data cleaning and analysis. We analyzed differences in outcomes between each gun storyline and the control condition using analysis of covariance (ANCOVA), controlling for political ideology, as well as t-tests. We then used the PROCESS Macro in SPSS⁷Hayes, A. F. (2017). Introduction to mediation, moderation, and conditional process analysis: A regression-based approach. Guilford Publications. for moderation analyses to examine differences between audience segments, as well as mediation analyses to examine reactance as a mechanism.

Appendix C: Content Analysis Episode Sample

Appendix D: Codebook and Reliability Scores

Episode-Level Variables

ITEM	SCORE8A Cohen’s Kappa or Krippendorf’s Alpha value of 1.00 indicates perfect reliability (100% agreement). Scores between .80 and .99 are considered strong. Scores between .60 and .79 are considered borderline, and scores below .59 are consid- ered inadequate. Items with borderline reliability are shaded in blue, and items with inadequate reliability are shaded in grey. Some shaded items are still included in this report but should be interpreted with caution.	COEFFICIENT9Cohen’s Kappa was used for all categorical variables, Krippendorf’s Alpha was used for all numeric (count) variables.
TV Series Name	1.00	Kappa
Season #	1.00	Kappa
Episode #	1.00	Kappa
In which American state does this episode primarily take place?	1.00	Kappa
Is there any firearm-related content in this episode (visual or dialogue)? [Yes/No]	1.00	Kappa
Does this episode provide any LINGUISTIC or CLEAR VISUAL information about where/how a character came into pos- session of a firearm? (Other than being issued in the line of duty.)	.65	Kappa
If the firearm was acquired from a person, what was the relationship? • Stranger • Acquaintance • Coworker • Friend • Romantic partner • Extended family member (cousin, uncle, etc.) • Close family member (child, sibling, parent)	.25	Kappa
Does this episode clearly show a firearm in STORAGE or in the process of being stored OR UNSTORED?	.82	Kappa
Does this episode show a firearm being LEFT OUT in the open—away from the presence of the possessor/owner?	.57	Kappa
Does this episode show a non-law-enforcement bystander attempting to intervene in an active shooting?	.49	Kappa
Does this episode show an active shooting that clearly takes place on or adjacent to a school campus?	1.00	Kappa
Does this episode have dialogue or text with explicit mention or discussion of “gun safety” or safe gun practices?	1.00	Kappa
Does this episode have dialogue or text explicitly discussing or referencing potential causes of “gun violence” as an abstract public issue?	1.00	Kappa
Does this episode have dialogue or text that addresses govern- mental policy issues, regulations, laws, or programs pertaining to the prevalence of firearms (gun show loophole, right to bear arms, etc.)?	1.00	Kappa
Does this episode have dialogue or text explicitly discussing the relationship between accessibility to guns (abstract or particular) and SUICIDE probability or statistics (abstract or particular)?	1.00	Kappa
Does this episode have dialogue or text explicitly discussing the relationship between accessibility to guns (abstract or particular) and DOMESTIC VIOLENCE probability or statistics (abstract or particular)?	1.00	Kappa
Are any of the following unsafe gun behaviors depicted? Select all that apply • Impaired firearm use (clearly under the influence of alco- hol/drugs) • Shooting into the air (intentionally shooting upwards, not at a particular target) • Holding a firearm on or next to a school campus	1.00	Kappa
How many characters are shown holding at least one firearm in-hand (unholstered)?	.98	Alpha
How many direct VICTIMS of gun violence are shown in this episode?	.96	Alpha

Character-Level Variables

ITEM	SCORE	COEFFICIENT
CHARACTER-LEVEL VARIABLES: CHARACTERS WITH GUNS
What is the name of the [nth] character who holds an unholstered firearm?	1.00	Kappa
What is the name of the ACTOR who plays this character?	1.00	Kappa
How central is this character to the series? • Minor character (appears in fewer than 20% of season’s episodes) • Recurring character (appears in 20-84% of current season’s episodes) • Main character (appears in at least 85% of current season’s episodes)	.87	Kappa
Character gender • Man (cis or trans) • Woman (cis or trans) • Non-binary • Can’t tell/unable to judge/not applicable	1.00	Kappa
Character age • 10 or younger • 11-17 • 18-30 • 31-64 • 65+ • Can’t tell/unable to judge/not applicable	.60	Kappa
Actor year of birth	1.00	Kappa
Character race/ethnicity • White • Black/African American • Hispanic/Latinx • East Asian/Pacific Islander • Middle Eastern/South Asian • Mixed/other • Can’t tell/unable to judge/not applicable	.84	Kappa
Is the character explicitly identified as a law enforcement officer?	.97	Kappa
Is the character explicitly identified as an active military soldier, officer, or personnel?	1.00	Kappa
Is the character explicitly identified as a school employee?	1.00	Kappa
Is the character explicitly identified as a documented immigrant?	1.00	Kappa
Is the character explicitly identified as an undocumented immigrant?	1.00	Kappa
Is the character explicitly identified as transgender?	1.00	Kappa
Is the character explicitly identified as having a long-term disability?	1.00	Kappa
Is the character explicitly identified as having a mental health condition?	1.00	Kappa
Is the character explicitly identified as addicted to a controlled substance other than tobacco?	1.00	Kappa
Does the character discharge a firearm? • No • Yes, intentionally • Yes, unintentionally	.97	Kappa
Does this character discharge a firearm in the range of a law enforcement agent? • No • Yes • Unclear	.85	Kappa
How many bullets does the character fire? • 0 • 1 • More than 1	.95	Kappa
Does the character shoot at least one person? (Bullet makes impact with any human body.) • No • Yes	.92	Kappa
What is the valence of this character? • Negative • Positive • Mixed or ambivalent	.70	Kappa
CHARACTER-LEVEL VARIABLES: GUN CASUALTIES
What is the [nth] VICTIM’s name?	1.00	Kappa
What is the name of the ACTOR who plays this character?	1.00	Kappa
Was this character shot by a law enforcement officer? • No • Yes • Unclear	1.00	Kappa
How central is this character to the series? • Minor character (appears in fewer than 20% of current season’s episodes) • Recurring character (appears in 20-84% of current season’s episodes) • Main character (appears in at least 85% of current season’s episodes)	.94	Kappa
Character gender • Man (cis or trans) • Woman (cis or trans) • Non-binary • Unclear	1.00	Kappa
Character age • 10 or younger • 11-17 • 18-30 • 31-64 • 65+ • Unclear	.69	Kappa
Character race/ethnicity • White • Black/African American • Hispanic/Latinx • East Asian/Pacific Islander • Middle Easter/South Asian • Mixed/other • Can’t tell/unable to judge/not applicable	1.00	Kappa
What is the valence of this character? • Negative • Positive • Mixed or ambivalent	.65	Kappa
Is the character explicitly identified as a law enforcement officer?	1.00	Kappa
Is the character explicitly identified as an active military soldier, officer, or personnel?	1.00	Kappa
Is the character explicitly identified as a documented immigrant?	1.00	Kappa
Is the character explicitly identified as an undocumented immigrant?	1.00	Kappa
Is the character explicitly identified as transgender?	1.00	Kappa
Is the character explicitly identified as having a long-term disability?	1.00	Kappa
Is the character explicitly identified as having a mental health condition?	1.00	Kappa
Is the character explicitly identified as addicted to a controlled substance (other than tobacco)?	1.00	Kappa

Appendix E: Survey Items

GUN STORAGE OUTCOME VARIABLES
To the best of your knowledge, please indicate which of the following statements is/are TRUE: [select all that apply]10True statements indicated in bold. Having a firearm in the home, even when it’s properly stored, makes you more likely to be shot. • Having a firearm in the home triples the risk of a suicide. • The majority of gun owners with children at home do not properly secure their guns. • Storing a gun properly (locked and unloaded) can help prevent unintentional shootings, though not fully eliminate risk. • More guns in a community (whether for self-defense, concealed carry, or recreational use) is associated with more homicides in that community. • None of the above are true
Does/would having a gun in your household make you feel: • A lot less safe than you would feel without a gun in your household • A little less safe than you would feel without a gun in your household • No more or less safe • A little safer than you would feel without a gun in your household • Much safer than you would feel without a gun in your household
Thinking about gun owners who DO NOT have children in their home, how important do you think it is for them to: • Keep all their guns in a locked place • Keep all of their guns unloaded • Learn proper gun safety measures • Advise adult visitors that there are guns in the house • Advise visitors with children that there are guns in the house • Store guns and ammunition separately • Train – practice regularly at a shooting range
Thinking about gun owners who HAVE CHILDREN in their home, how important do you think it is for them to: • Keep all their guns in a locked place • Keep all of their guns unloaded • Learn proper gun safety measures • Advise adult visitors that there are guns in the house • Advise visitors with children that there are guns in the house • Store guns and ammunition separately • Train – practice regularly at a shooting range • Talk to their children about gun safety

ACTIVE SHOOTER TRAINING OUTCOME VARIABLES
To what extent do you think each of the following would be EFFECTIVE at preparing schools and/or students for a real-life active shooter event? • Bringing together groups of students and teachers to talk through a hypothetical active shooter situation step-by-step without engaging in simulation (i.e., tabletop exercises). • Conducting active shooter school drills that simulate gun violence, but all individuals involved know that there is no real threat (i.e., announced active shooter drills). • Conducting active shooter school drills that simulate gun violence, in which some or all individuals involved believe it is a real-life active shooter event (i.e., unannounced active shooter drills). • Conducting lockdown drills, where students and teachers practice responding to an active shooter drill (e.g., hiding, staying quiet, clearing hallways), but includes NO simulated gun violence. • Online or software-based training only for teachers and/or students. • Employing law enforcement officers to provide armed security in school. • Allowing school security officers to carry guns in school. • Allowing teachers and other non-security school staff to carry guns in school. • Implementing policies and programs to prevent and intervene prior to a shooting, such as having anonymous tip lines, threat assessment teams, safe school design, and promoting secure storage of firearms by parents.
Regardless of how effective you think each would be at preparing students and teachers for real-life active shooter situations, to what extent do you think each of the following would create EMOTIONAL HARM for students and teachers? • Bringing together groups of students and teachers to talk through a hypothetical active shooter situation step-by-step without engaging in simulation (i.e., tabletop exercises). • Conducting active shooter school drills that simulate gun violence, but all individuals involved know that there is no real threat (i.e., announced active shooter drills). • Conducting active shooter school drills that simulate gun violence, in which some or all individuals involved believe it is a real-life active shooter event (i.e., unannounced active shooter drills). • Conducting lockdown drills, where students and teachers practice responding to an active shooter drill (e.g., hiding, staying quiet, clearing hallways), but includes NO simulated gun violence.
To what extent would you support or oppose the following policies/legislation regarding public schools: • Outlawing any type of active shooter drills in public schools that simulate gun violence. • Requiring schools to obtain parental consent before conducting any type of active shooter drill. • Requiring schools to screen all students for emotional trauma (e.g., PTSD) after active shooter drills to provide counseling when needed. • Requiring all public schools to participate in some sort of active shooter training (including lockdown drills with no gun simulation). • Employing law enforcement officers to provide armed security in school. • Allowing school security officers to carry guns in school. • Allowing teachers and non-security school staff to carry guns in school. • Requiring schools to have policies and programs to prevent and intervene prior to a shooting, such as having anonymous tips lines, threat assessment teams, safe school design, and promoting secure storage of firearms by parents.
THEORETICAL VARIABLES: REACTANCE (Brehm, 1966)
Indicate the extent to which you agree or disagree with each of the following statements (1= strongly disagree to 7= strongly agree): • The storyline tried to make a decision for me. • The storyline tried to pressure me. • The storyline threatened my freedom to choose. • The storyline tried to manipulate me.
SCREENING ITEMS
Please confirm your age: • I am 18 or older • I am under 18 years old
Are there any guns in your household (NOT including air guns, such as paintball, BB, or pellet guns)? • Yes, I have a gun • Yes, a member of my household has a gun • No, I/we don’t have any guns
Do you have any children under the age of 18? • Yes • No
DEMOGRAPHIC VARIABLES
With which gender do you most identify? • Male • Female • Non-binary • Decline to state
Are you Hispanic or Latinx? • Yes • No
Which of the following best represents your race? [Select all that apply] • American Indian or Alaska Native • Asian • Black or African American • Native Hawaiian or other Pacific Islander • Middle Eastern • White • Other (please specify) • Decline to state
What is the highest level of school that you have completed or the highest degree you have received? • Less than high school degree • High school graduate (high school diploma or equivalent including GED) • Some college but no degree • Associate’s degree in college (2-year) • Bachelor’s degree in college (4-year) • Master’s degree • Doctoral degree/ professional degree (JD, MD, PhD) • Other (please specify)
Information about income is very important to understand. Would you please give us your best guess? Please indicate the answer that includes your entire household income in 2020 before taxes. • $0 to $9,999 • $10,000 to $14,999 • $15,000 to $19,999 • $20,000 to $34,999 • $35,000 to $49,999 • $50,000 to $74,999 • $75,000 to $99,999 • $100,000 to $199,999 • $200,000 or more • Not sure / Decline to state
Which of the following best represents your religious affiliation, if any? • None / Atheist / Agnostic • Buddhist • Catholic • Christian (please specify) • Hindu • Jewish • Muslim • Other (please specify) • Don’t know / Decline to state
Overall, what description best represents your political ideology? • Very liberal • Liberal • Moderate • Conservative • Very conservative • Don’t know / Decline to state
Which of the following best represents where you live? • Large city / Urban area • Small city / Suburban area • Countryside / Rural area • Other (please specify) • Not sure / Decline to state
What is your current age? • 18 to 24 years • 25 to 34 years • 35 to 44 years • 45 to 54 years • 55 to 64 years • 65 to 74 years • 75 to 84 years • 85 years or above • Decline to state