Yoav Goldberg, August 2024

In her "Presidential Address" at the ACL 2024, Emily Bender gave a talk called "ACL is not an AI Conference". For those who did not attend (or were not paying close attention), you can find the slides in the following link: https://faculty.washington.edu/ebender/papers/ACL_2024_Presidential_Address.pdf

Somewhat surprisingly, I found myself agreeing with some core aspects of her argument. Perhaps less surprisingly, there is also a substantial part which I strongly disagree with. This text is a response to this address, and, beyond just responding, may also shed some light on what is ACL, and what is NLP. I of course welcome discussion on these topics, either on the comments section here (unfortunately not very convenient) or on Twitter (not convenient in a different way). Ok, Let's go.

First, before we go into the core of the issue, I will touch on something a bit tangential. Throughout the talk, Emily referred to ACL as a "Computational Linguistics" or "Computational Linguistics or NLP Conference". This was very intentional, perhaps in a Sapir-Whorfian attempt to change reality.

But let's get this out of the way: unless we redefine what Computational Linguistics means to mean "NLP" (and I really don't think Emily intended to make such redefinition), ACL is in no way or form a Computational Linguistics conference. Yes, it has CL in its name, that is an unfortunate historical artifact.

But ACL does not deal with computational linguistics (a sub-field of linguistics that attempts to answer linguistic questions using computational methods), but with NLP (computational methods of transforming, "understanding" or generating natural language).

These are not the same thing. In fact they are very different things. While an argument can be made regarding having both topics in the same conference (an argument can also be made against it), the fact is that computational linguistics works are not presented at ACL, will have a very hard time getting accepted to ACL, and if such a work happens to be accepted to ACL, the authors will fill very lonely at the conference. There in fact is a venue for actual computational linguistic works, it is called SCiL ("Society for Computation in Linguistics"), and there is a community that publishes there.

Pretending that ACL is also a form of computational linguistics venue is just silly, and beyond silly, I believe it is also harmful to the real computational linguistics community, who really should have their own venue where they can publish, be read, and attend conferences with people who care about and are excited about their work, and not have to occasionally submit to ACL by mistake just to be confused by the reviews.

Having that out of the way, let's go on to the main issue.

Emily titled her talk "ACL is not an AI Conference", I strongly agree with that. ACL is an NLP conference, and NLP is not AI.¹

As Emily said, "Language processing is a prerequisite for 'AI', but that doesn’t mean that 'AI' is the only goal of CL/NLP".

I strongly agree with this as well (if we ignore that 'CL' that got inserted there): I certainly did not enter the field because I cared about AI. I entered the field to do NLP: to understand how to build algorithms that work with human language, and to enable applications that process and/or interact with human language.

So, yeah, ACL is not an AI conference. ACL is an NLP conference. NLP is great.

Emily also gives a "best practices in CL²/NLP research" (slides 10-15). I agree with these as well, of course (though once we recast "Knowledge of How Languages Work" as "Knowledge of the domain one is working in", we are left with just a list of good overall scientific and engineering practices. Which is great, but not really about NLP).

But the full title of this slide is "Contrast with best practices in CL/NLP research", where the contrast is "AI research" which leads me to...

A major issue with the piece is that it attacks a straw-man. In slides 5-7 Emily explains what she means by "AI" in "ACL is not an AI conference".

AI as a research & commercial field Asks questions like:

• How do we build “thinking machines” that can do “human-like” reasoning?
• How do we build “thinking machines” that can “surpass” humans in cognitive work?
  • (and cure cancer, solve the climate crisis, make end-of-life decisions, etc)
• How do we automate the scientific method?
• How do we automate away such creative work as painting and writing?
  • Or: How do we steal artwork at scale and try to convince people this is “for the common good”?

Slide 5 is reasonable, though somewhat reductive of what AI is or aims to do as a research field (well, very reductive).

The last bullet (and especially its sub-bullet) then ventures into "commercial AI", which leads us to slides 6 and 7:

AI as a research & commercial field

And makes assertions like:

• Humanityʼs destiny is to merge with machines and become “transhuman”
• The singularity is coming: “AGI” is inevitable and will outstrip people in all ways that matter
• “AI” (really synthetic text extruding machines) is a suitable replacement for the services we owe each other (education, healthcare, legal representation)
• All of this is inevitable and refusal is futile

AI as a research & commercial field

Suffers from multiple scourges:

• Intense (though maybe waning?) interest from venture capital
• Intense (and not waning) interest from billionaires
• The racist history and present of the notion of “intelligence”/IQ
• Intense interest from proponents of TESCREAL ideologies (Gebru & Torres 2024)

Here it becomes clear that Emily has in mind a very particular notion of "AI", the one that is peddled by doomers and/or billionaires and which aligns with "TESCREAL ideologies". Well, sure, there are some commercial parties, and some individuals, who subscribe to this version of "AI". But these are, by and large, not what the academic field of "AI" is concerned with. By this definition of "AI", ICLR, NeurIPS, ICML, and also IJCAI and AAAI are not AI conferences. Indeed, by this definition there are no academic AI conferences (well maybe NeurIPS is. But then, I did say academic conferences ;) ). And since all the argumentation is built around contrasting with this very specific---and somewhat irrelevant to our context---definition of AI, things fall apart very quickly for me.

I get it, the commercial AI trends are bad, TESCREAL ideologies may be problematic. Certainly Emily is part of a campaign against them, and this campaign may be justified (I am not judging). But what does it have to do with ACL, with NLP, or with Academic AI and ML more generally? And why mix the ideological campaign with what NLP is or is not? (and is Emily conflating this wacko definition of AI with "LLMs"? hmmm)

This is just a bad argument.

The second-to-last slide lists "ACL 2024 papers that have nothing to do with AI". I think she meant "Have nothing to do with LLMs" or "Have nothing to do with ML". Which is (a) telling; and (b) departs from her previous---straw-man---definition of "AI" in a somewhat sneaky way.

Putting the straw-man issue aside, and maybe the most important, I think the talk gives a very conservative, and very restrictive, view of what NLP is, what language is, what good research practices are, and NLP ought to be.

Let's get back the the first bullets of slide 5:

• How do we build “thinking machines” that can do “human-like” reasoning?
• How do we build “thinking machines” that can “surpass” humans in cognitive work.

I already wrote I think this is a very reductive view of what "AI Research" is. But by contrasting it to NLP, as Emily does, it also a highlights a very reductive view of what language is, and of what language processing is.

Yes, I also observed a few months ago that the fields shifted from what I considered to be "classic" NLP work to notions like "reasoning", and my immediate reaction was "hey this is not about language anymore". But after a bit of reflection, I realized that, hey, it is very much about language, because reasoning process is part of language. You cannot decouple (certain notions of) reasoning abilities from language use, and you cannot even perform certain kinds of reasoning without natural language (just look at how much trouble semanticists are taking to try and formalize language with logic, without coming with a complete, let alone elegant, solution). People reason in language. Communication involves reasoning. Forming, perceiving and responding to "communicative intents", a term Emily likes to use, involves a hefty bit of reasoning. Sure, not all reasoning is linguistic, but to contrast reasoning with NLP is just bananas.

Similarly for "thinking", yes, language and thinking are intimately correlated. Especially if we consider that we want to emulate "human-like" behavior. This is all hard-core natural language stuff, it does not by any means contrast with it.

And finally, of course we want to "surpass" humans (in cognitive work or otherwise), this is what computers are for. They are faster and more accurate in certain things, and this is good!

Emily also lists (same slide 5) "cure cancer, solve the climate crisis, make end-of-life decisions, etc" and "How do we automate the scientific method?" as (problematic) aspects of AI. First, these are all good things that we should be doing. Second, some people in NLP work on these and similar problems, and this is a good thing!! Yes, we want to cure cancer, we want to solve climate crisis, we want science to be more automated, faster, more efficient, and yes, NLP can be of great help with that.

Slide 8 lists a bunch of things NLP research is asking. And true, it does. But it is not ALL that it does, and I am truly surprised a linguist would miss that and dismiss it as "(the bad part of) AI".

Finally, slide 10 lists "Bad research practices" that are "due to AI".

I already covered the "due to AI" part in the straw-man section above. Let's now focus on the content.

• Demands to evaluate against "SOTA" closed models
• Unmanageably large datasets (=> lack of held out data)
• Exploitative research & development practices

The last one is not a research practice at all, so I am not sure what it does in this list rather than for bashing AI. But let's focus on the first two: they are research practices, but I strongly disagree that they are bad ones. In fact, I believe that not doing them is in many cases a bad research practice.

Eval against closed models: yes, it is annoying that we do not know how the models work. Yes, this limits the conclusions we can reach by studying them. But this does not mean we should pretend that they don't exist, and in many (though not all!) cases one really should compare to closed models if they want to convince that their point holds. Not everything is observable, this is a fact of nature. Science deals with that with finding more robust ways of forming valid conclusions despite the non-observability, not by ignoring reality. The bad practice is not to demand evaluating against closed models, this is actually a good practice. The bad practice is to demand an evaluation against closed models in situations in which this is irrelevant to the argument the authors are trying to make.

Unmanageably large datasets: it is very clear to anyone who has been paying even slight attention that scale helps. A lot. For better or worse, we need these large datasets. And already tools appear to aid in querying and inspecting them. Rather than labeling the large corpora as "bad practice", "danger", "too large", we should promote work on improving the inspection tools, and promote looking into ways that allow to make convincing claims and form valid conclusions despite not having access to the training data. Big-data seems to be a major part of LLMs. The good practice is to study it, not to ignore its existence.

Let me summarize my main points:

• ACL is not an AI conference.
• ACL is not a CL conference.
• ACL is an NLP conference.
• Academic AI is not TESCERAL nor doomerism.
• NLP intersects with AI, but is neither subsumed by nor subsumes AI.
• NLP conferences should focus on language processing.
• Language processing is expanding its boundaries, and this is a good thing.
• Closing your eyes and ignoring reality is not good science.
• Embracing reality and finding way around limitations is good science.
• Activism leads to bad science.