Generative large language models (LLMs) can whip up original content based on custom prompts and at lengths you specify in just seconds. These tools write in a way that sounds human and can answer tons of different questions (Zhao et al. Reference Zhao, Zhou and Li2023). Popular platforms like Chat GPT (Brown et al. Reference Brown, Mann, Ryder and Larochelle2020) and Google Gemini are either free or pretty cheap, which has some people worried students might use them to automatically create essays or test answers. There’s concern that LLMs could make “contract cheating”-paying someone else to do your work-even more common since AI tools cost less. It’s hard to get solid data on how much students are using LLMs because of social-desirability bias and sampling problems, but awareness and use of these tools have grown fast since late 2022.
Universities and teachers are reacting in different ways. The big discussions focus on whether using LLMs counts as plagiarism; if these tools help or hurt student learning and prep for careers; and whether AI should be banned outright, included as part of teaching, or encouraged. You can see this split in official university policies. For example, Harvard Law School’s “Statement on Use of AI Large Language Models” says using AI “in preparing to write, or writing, academic work for courses…is prohibited unless expressly identified in writing by the instructor as an appropriate resource….” But at Arizona State University’s Sandra Day O’Connor College of Law, applicants are allowed to use generative AI openly because it’s common in legal work and fits with their goal “to educate and prepare the next generation of lawyers and leaders.”
In classrooms themselves, instructors usually decide their own rules about generative AI use. Princeton University says it “doesn’t intend to ban Chat GPT or to levy a top-down edict about how each instructor should address the AI program….” Similarly at De Paul University, “Faculty have the discretion to allow or encourage students to use AI in class activities and/or assignments.” Across all types of schools-from research universities (R1s), mid-level research institutions (R2s), to liberal arts colleges-professors mostly figure out their own assessments for students. We’re not trying here to say what the right policy is-that depends on each teacher-but instead offer data showing what political science educators are actually doing with these new AI tools plus some key points from colleagues we surveyed that might help guide choices. This study fits into a growing body of teaching research looking at both opportunities and challenges posed by LLMs (e. g., Kasneci et al. Reference Kasneci, Sessler and Küchemann2023; Tlili et al. Reference Tlili, Shehata and Adarkwah2023).
Specifically, we share results from two surveys done together with the American Political Science Association (APSA) between April and August 2023 (Wu and Wu Reference Wu and Wu2024). These asked political science faculty about how they see generative LLMs helping or affecting their classes. The first was a brief survey sent out during APSA’s 2023 membership survey targeting current/recent members. The second was longer: we asked a smaller group more detailed questions about steps they’ve taken because these tools became widespread-and tested whether they could tell if essays were written by bots versus real students from our essay bank. We found that (1) overall educators lean slightly negative on AI’s effects but do think some uses like copyediting help; (2) many plan changes mostly focused on enforcement rather than embracing AI fully; (3) those who tried identifying bot vs student essays did no better than flipping a coin-success rates were only around 45-53%.
Next up: we explain how we gathered our data then go over detailed findings from both surveys before wrapping up with a look at popular tactics teachers report using against rising LLM use.
DATA AND METHODS
Between April and August 2023, we ran two surveys alongside APSA aiming to get insights into how political science academics view generative LLMs’ usefulness & impact. Our research ethics board at University of Toronto approved this project (#00044085). The first survey had four questions embedded within APSA’s annual membership poll sent out roughly to 11K people-getting back 1,615 responses total: About 62.6% faculty members; 11.8% grad students; 4.6% adjunct/visiting lecturers; 3.7% postdocs mainly based in US (81%). Respondents came from Ph D-granting schools (~938), MA-granting (~146), four-year colleges (~224), two-year/community colleges (~31), others (~18); some skipped this question (~258). On average respondents earned their Ph Ds around 2004-we call this our “membership survey.” The second was longer & targeted folks involved with APSA’s Teaching & Learning Conference over last five years-the “TLC survey”-sent out to ~3,442 people yielding 198 responses so far-likely folks more interested in pedagogy than average professor-which let us add questions like identifying bot vs human essays.
Together these surveys give us broad views across many political science educators while letting us dig deeper into opinions around new tech like generative AIs too-all administered opt-in by APSA itself which probably helped boost response rates compared with if we’d sent them alone ourselves though obviously sample is limited only to current/recent APSA members-but still reached lots who’d otherwise be tough for us researchers alone.
Since LLMs mostly impact written assignments courses assign anyway-we started by asking about importance colleagues place on written work then moved onto attitudes toward LLM usage generally plus specific tool applications-in TLC survey also asked if/how they’re responding pedagogically plus had respondents judge essays as bot- or human-written from essay bank -results follow next section.
RESULTS
Here are six main takeaways from our two surveys showing mixed feelings-some doubt but also recognition-that AI will shake things up when it comes teaching political science subjects.
Writing Assignments Matter A Lot To Political Science Teachers
People who teach political science agree it’s really writing-heavy major stuff going on here! All TLC respondents said written assignments count big time especially upper-level classes-with only tiny fraction saying it doesn’t matter much for lower-level ones either-and same trend showed up again among broader membership group too. Figure 1 shows most important goals connected with written tasks include sharpening writing skills overall plus building argumentative style along with creating fresh ideas through writing -excluding those who didn’t answer/didn’t know.
Figure 1 Pedagogical Value Of Written Assignments (Membership Survey)
Note: Question asked was “Generally speaking.. what educational value do you see in written assessments.. for yourself or your students?”
We also let folks type freeform answers about what matters most re writing tasks – common themes popped up such as critical thinking & analysis skills development; understanding course materials better; putting knowledge pieces together thoughtfully; weighing concepts carefully; building job-ready skills etcetera.
Among TLC participants averages showed around 62% upper-level grades + almost half lower-level grades come straight off take-home papers/exams-which poses clear headaches if instructors want clamp down on student misuse via LLM aids! Taking tests/writing inside classroom doesn’t seem great fix since limited time reduces chances for deeper thinking/writing execution anyway though.
Teachers Mostly Lean Toward Doubt Or Uncertainty About Ai Tools
On whole colleagues seemed more doubtful than hopeful regarding AIs rating -0.45 average score across scale -2 (“very bad”) through +2 (“very good”). Plurality chose neutral middle ground believing AIs neither harm nor help classrooms much(41.5 %), only small minority(13.6 %) felt positive/good effects existed(Figure 2). This isn’t shocking given current limits + possible risks undermining teacher goals tied closely with assigned student writings. Small TLC sample echoed similar vibe largely consistent findings reported below too.
Figure 2 Responses To Question:“Overall Do You Think These Ai Tools Are Good Or Bad For Your Political Science Classes?”(Membership Survey)An open-ended query followed probed reasons why folks saw positives / negatives / neutral views related AIs impacts involving education. Dozen themes rose quickly including worries re lost independent thought originality(e. g.“It does nothing teach learn think critically independently formulate evidence-based views”); lack critical engagement(e. g.“They provide low-threshold ways avoid hard long learning”); academic honesty issues(e. g.“Students solely rely ai breaking integrity”); misinformation biases generated(Llms often hallucinate false info)(Gao et al. Reference Gao, Yen, Yu Chen2023). On flip side many saw benefit tasks editing grammar fixes formatting(e. g.“Very useful catching mistakes improving prose”) plus support non-native English speakers formulating arguments well. Another camp viewed them simply inevitable changes professors must adapt/help manage properly keeping honest usage aligned w expectations.”Support Changes Depending On What Ai Is Used ForEven though feelings leaned negative overall toward Ai functions, it helps seeing nuanced positions depending application type. Ai can create full essays answering wide topics drawing massive text corpora. But assistive roles fixing grammar cleaning formatting got higher thumbs-up(Figure three shuffled order below).Figur e three Support For Various Ai Tool Usages(Membership Survey )Note : Question Asked:“Do You Favor/Oppose Neither Favor Nor Oppose Following Uses?”Respondents disliked using ai generate entire essays/multiple-choice/fill blanks. Also opposed generating ideas, outlines, citations, evidence. LLMS synthesize huge text sets online yet don ‘t reliably cite sources sometimes giving random wrong info (“hallucinations”) explaining negative vibes.(Gao et al.)In contrast correcting grammar+formatting seen positively. Such uses similar accepted software(like Grammarly)/writing center assistance. Mostly helpful ESL learners improve paper quality handle tedious citation styles. Support strongest when used assistively not fully generating content. Attitudes similar regardless institution type educator role(details online appendix ).…instructors tend favor ai assistive uses instead full content generation.Teachers Plan Changes Mainly Focused On Detecting And Preventing Ai UsageOver sixty percent tlc responders expect half-plus their students will turn t o ai help soon. With figure four showing planned reactions commonly considered below :Figure4 Planned Adjustments To Teaching Due To Ai(TLC Survey)Despite valuing writing heavily30 % plan reducing papers due l lm rise while nearly half keep assignment numbers steady. Most consider testing questions via chatgpt plus adopting detection systems. These methods give sense l lm approaches/topics but ai output highly prompt sensitive&detection systems prone false alarms risking unfair penalties raising stress workload everyone involved.Most See Importance Of Ai But Aren’t Keen To Add Into CoursesAround seventy-three percent tlc respondents say its moderately-or-more crucial for students learn gen ai. But over fifty percent unlikely/slightly likely integrate ai formally. More than thirty percent said absolutely no chance. Figure five lays full responses out.
Many recognize importance yet hesitate actual course integration despite belief majority will use ai anyway.
Figure Five Importance And Integration Of Ai(Tlc Survey)
So basically those skeptical prioritize enforcing restrictions rather than welcoming integration so far.
Educators Aren’t Always Good At Telling Student Vs Bot Essays ApartOne big issue blocking enforcement now? No foolproof way spot ai-generated texts. In our tlc poll, we tested this asking people assess sample essays taken actual undergrad prompts. Wanted them guess if piece was human-made vs GPT-written & rate confidence levels.
They’re not always reliable judges here either.
Different topics held varying number samples. The first topic international relations footnote¹ had two student pieces plus two gpt4 versions. Second american politics footnote² one student essay +two gpt4 third political economy footnote³ equal mix. Gpt outputs included basic prompt-only version + another where prompt guided by extra subject detail. Two gpt versions per essay.
Only roughly half correctly picked source(human vs machine); rest guessed wrong/couldn’t tell(Table One). Having field-specific knowledge helped just slightly-for IR instructors54.5 % correct AP51.22 %, comparative politics50 %. If teachers build tougher ‘ai-proof’ prompts maybe performance better real life. Still suggests judging standard topics isn’t reliable game confidence scores middling near five out ten details online appendix reinforce challenge telling apart gpt4/student texts.
DISCUSSION AND RECOMMENDATIONS