01/02/2024
Poster!
Recording
Thank you so much to Dominik Lukeš for treating us to a talk which really stretched our minds, and explained just how weird ChatGTP is! This is a recording which we will want to watch a few times in order to grasp the enormity that is AI! We are so grateful to Dominik for his time and for a superbly crafted presentation.
Thanks to Joe Dale for introducing us to Dominik, and thanks to Antonio Marcio da Silva for co-moderating the session on behalf of ALL. Thanks to Joe Dale for uploading the recording to his YouTube channel.
Session Description
ChatGPT is a weird linguist: What should language teachers know about how LLMs work? Large Language Models (LLMs) are not superintelligent beings but they are superpolyglots. Their ability to interact in almost any language at a level of high proficiency was one of the biggest revelations. But this should not be a surprise – transformers, the technology behind LLMs, was invented in 2017 by Google researchers to improve machine translation. So it was more of a surprise that they can do all the things that are not language-related. But as polyglottic as they are, LLMs are very strange linguists. The way they were trained and the way they generate output are very different from how humans learn and use language. And this has consequences that are easy to overlook when we stay on the surface. Many of these strange properties are only revealed when we try to have LLMs do the sort of things language teachers do: Give examples in another language according to specified criteria, correct and explain grammar, or a have an interaction at the level of a beginner. Superficially it looks that LLMs are really good at these tasks but they often struggle to be consistent in subtle and unpredictable ways. This talk will cover how LLMs work, how they were trained and what are the consequences for their use in the language classroom. It will focus on the strengths and weaknesses of LLMs when it comes to languages and possible mitigations through task design and prompt crafting.
Presenter
Dominik Lukeš is the Assistive Technology Officer at the University of Oxford where he started and manages the Reading and Writing Innovation Lab, assessing devices and apps for academic content.
He conducts AI workshops and contributes to key reports like “Beyond ChatGPT: The State of Generative AI in Academic Practice (Autumn 2023)” and “Transforming Higher Education: Harnessing AI in Teaching and Assessment.” He maintains a LinkedIn newsletter on AI in Academic Practice and has spoken on AI at international conferences.
Dominik serves on advisory boards for MacGraw Hill and Bett UK.
Before focusing on educational technologies, Dominik was a long-time teacher of Czech as a foreign language and is an author of a textbook and grammar book of Czech for non-native speakers. He also taught English as a foreign language and run TEFL training workshops in 16 countries.
Chat transcript
00:23:08 Antonio/圣明: https://www.facebook.com/groups/languageteachingwithai/
00:23:14 Antonio/圣明: This is Joe’s group
00:23:41 Alicia: yes! hello!
00:23:54 Joe Dale: Thanks Antonio!
00:24:01 Vincent Everett: ALL member since 1995ish…
00:24:19 Antonio/圣明: If you haven’t become a member, please have a look at the page: https://www.all-languages.org.uk/
00:26:16 Kevin Dunne: I’m a proud ALL member, and remember meeting Vincent at LW in Manchester in 1999? I’m also Acting Vice Chair of ISMLA, www.ismla.co.uk
00:26:45 Burkey: Hello everyone,
00:27:00 Vincent Everett: Hi Kevin!! Good to hear from you.
00:27:08 Burkey: It has been a while. I am Nolwenn and I have moved to Taunton.
00:30:39 Glenn: French
00:30:39 salamas@cardiff.ac.uk: Arabic
00:30:42 EMcAssey: german
00:30:42 Nicola Cristiano Insignares: Russian
00:30:42 Luisa Blake: French and Spanish
00:30:43 Enza Currenti: Italian and French
00:30:43 Helen Myers: French
00:30:45 Martine Rheaume: French
00:30:45 Mila Duncan: Spanish
00:30:46 Nadiah: French
00:30:47 Mm: German
00:30:48 Elizabeth Maldonado: I teach Spanish at undergraduate level.
00:30:49 Alicia: French, Spa, Italian, Ger
00:30:50 Naomi Brown: French, German, Italian
00:30:51 Vincent Everett: French and Spanish
00:30:52 Nadiah: Spanish
00:30:59 Burkey: I teach French and Spanish and I used to teach Latin
00:31:06 Chris Fry: I taught English as a Foreign Language and I’m interested in Speech recognition & readability
00:32:12 Chris Fry: I’ve played around with it A LOT!
00:32:30 Nadiah: Lesson plans
00:33:14 Alicia: wrote a letter of complaint to an airline – I got a refund so it worked!
00:33:29 Naomi Brown: Generating texts to illustrate a particular grammar point to students
00:33:36 Vincent Everett: Also played using it for feedback on pupils’ work
00:33:37 Chris Fry: I’m interested in speech recognition and feedback on transcriptions
00:33:42 Helen Myers: Used it to get quick dates when songs were published … alist of about 50 songs
00:34:02 Martine Rheaume: conversation partner
00:34:13 Martine Rheaume: Perplexity.ai
00:34:16 Naomi Brown: ChatGPT
00:34:21 Chris Fry: Replying to “Also played using it…”
Spoken or written?
00:34:21 Naomi Brown: LingoTeach.ai
00:34:22 Luisa Blake: ChatGPT
00:34:23 Helen Myers: YouTUbeers also adding comments
00:34:25 Mila Duncan: Canva
00:34:33 Vincent Everett: Bing, copilot, chatgpt, bard
00:34:34 EMcAssey: chatgpt
00:34:45 Mila Duncan: Twee
00:34:46 Glenn: https://twee.com/
00:34:47 Joe Dale: ChatGPT, Claude, Poe, Bard, Bing Chat, CoPilot
00:35:00 Joe Dale: 🙂
00:35:15 Elizabeth Maldonado: Curious.
00:35:30 Glenn: Reacted to “Curious.” with 👍
00:35:33 Paul Joyce: Reacted to “Curious.” with 👍
00:35:33 Alicia: super excited!
00:35:35 Martine Rheaume: How did I survive without Chat GPT before? I could not live without “him”
00:35:37 Vincent Everett: If it wasn’t hyped I would say it was amazing. But given the hype I am disappointed
00:36:01 Sara Gomez Villa – Loughborough University: Reacted to “Curious.” with 👍
00:36:03 Naomi Brown: Excited, it has amazing potential. Worried about it making students lazy..?
00:36:31 Chris Fry: I agree, so much!
00:37:58 Chris Fry: It’s hard to keep pace as things change almost every day
00:38:08 Joe Dale: I agree, Chris!
00:40:43 Chris Fry: I used it to write a formula for Excel as it was too much work to remember how to do it, even though I have studied it
00:43:30 Chris Fry: Copilot seems to offer ChatGPT for free, but I got some ads the other day for cough medicine and something else
00:44:06 Vincent Everett: Q: is the copilot/bing chatgpt 4 fully functional?
00:44:08 Joe Dale: Any questions, please put them in the chat starting with a Q
00:44:10 Jeanne Y: Reacted to “Twee” with 👍
00:44:31 Joe Dale: @Vincent What do you mean ‘fully functional’?
00:45:03 Vincent Everett: Q: is it true that an LLM is actually small enough to fit on a usb flash memory device?
00:45:25 Alicia: no
00:45:39 Joe Dale: Yes, ChatGPT Plus
00:45:55 Svitlana: No
00:46:03 Chris Fry: I don’t think I’m paying for Copilot
00:46:26 Vincent Everett: No. But I have heard that some of the paid ones actually work
00:46:52 Chris Fry: I use Copilot every day but I don’t think I’m paying anything
00:47:01 Vincent Everett: Copilot on the bing app is free but I don’t know if it is fully functional
00:47:31 Joe Dale: @Vincent Thanks for clarifying
00:48:20 Chris Fry: Getting feedback on students’ English
00:49:12 Svetlana Kandybovich: generate text (if that’s about LLMs)
00:49:18 Mila Duncan: it should support enhancing human creativity
00:51:00 Vincent Everett: It can also do knitting patterns
00:51:58 Helen Myers: My hardware has a few faults
00:53:35 Chris Fry: It’s better about using language than it is about the way grammar works. Just the opposite of most students
00:53:41 Vincent Everett: I asked it to conjugate invented Spanish verbs. It could nearly do it.
00:54:28 Helen Myers: Brilliant!
00:54:47 Helen Myers: That is so impressive
00:54:55 Joe Dale: 🙂
00:57:39 Vincent Everett: Although they are very often biased/stereotyped.
01:00:12 Svetlana Kandybovich: Q: Aside from the big 4, have you tried installing and running open-source models on a computer?
01:00:54 Joe Dale: @Svetlana Kandybovich Good question!
01:01:33 Helen Myers: I am intrigued by this
01:02:08 Chris Fry: The moral is not to ask Gen-AI to do things it wasn’t designed to do
01:02:52 Joe Dale: Yes, Chris and the challenge is to work out what it can do well and what it can’t do well
01:02:58 Svetlana Kandybovich: Autocomplete on steroids))
01:03:16 Joe Dale: Yes, Svetlana!
01:03:25 Svetlana Kandybovich: Reacted to “Yes, Svetlana!” with 🙂
01:04:16 Helen Myers: https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
01:04:38 Martine Rheaume: Is it possible to enter a set of data in excel, ask to perform a statistical analysis and write out the interpretation
01:07:54 Joe Dale: Fascinating!
01:10:02 Chris Fry: Reacted to “https://writings.ste…” with 👏🏼
01:10:50 Vincent Everett: Ask it for a list of French nationality adjectives ending in ain. It’s hilarious
01:11:13 Vincent Everett: Espagnolien
01:13:07 Chris Fry: I have lots of problems when I give Gen-AI instructions to mark mistakes in bold
01:14:32 Vincent Everett: Q So how does it know to stay in its lane with one language and not swap all the time?
01:15:24 Joe Dale: I’m collecting all the questions, so keep them coming!
01:15:49 Naomi Brown: Q: Do the differences in the tokens between languages mean that it is less accurate in producing text in some language compared to others?
01:20:10 Helen Myers: https://www.sketchengine.eu/
01:22:19 Vincent Everett: Is that because it is working in English and translating into Czech?
01:22:41 Helen Myers: https://arxiv.org/abs/1706.03762
01:23:02 Helen Myers: (Attention is all you need)
01:23:35 Chris Fry: Reacted to “https://www.sketchen…” with 😀
01:23:51 Helen Myers: https://karpathy.github.io/
01:24:04 Helen Myers: Andrej Karpathy blog
01:24:19 Joe Dale: Thanks for the links, Helen!
01:24:46 Chris Fry: Reacted to “Thanks for the links…” with 👏🏼
01:25:07 Helen Myers: Take your time Dominik!!!
01:25:12 Vincent Everett: At GCSE that kind of referring back and forward is typical in the questions they set!
01:25:13 Joe Dale: Thanks Dominik!
01:26:10 Helen Myers: If you have to go, no problem .. we will carry on recording and if you have asked questions, we will ask them on your behalf!
01:26:36 Helen Myers: https://literarydevices.net/anaphora/
01:26:50 Helen Myers: (definition of anaphora!)
01:27:13 Martine Rheaume: Merci Dominik!
01:27:20 Chris Fry: All of us users are providing the third stage when we rate responses
01:28:09 Vincent Everett: I don’t know how you “reward” or incentivise the mschine learning. What sort of incentives? Sweets? Survival?
01:29:13 Svetlana Kandybovich: Reacted to “I don’t know how you…” with 😂
01:29:48 Helen Myers: https://arxiv.org/abs/2203.02155
01:30:00 Helen Myers: [Submitted on 4 Mar 2022]
Training language models to follow instructions with human feedback
01:30:32 Helen Myers: RLHF – a new acronym to impress our friends!
01:31:07 Joe Dale: 🙂
01:31:14 Helen Myers: https://arxiv.org/abs/2201.11903
01:31:20 Helen Myers: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
01:32:50 Helen Myers: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4573321
01:32:59 Helen Myers: Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality
01:33:29 Helen Myers: Love the honesty!
01:33:34 Helen Myers: !!!!
01:33:35 Joe Dale: I agree
01:34:08 Vincent Everett: It is designed to be “plausible”
01:34:28 Mm: Thank you. Very helpful!
01:34:51 Vincent Everett: Thank you very much! Very interesting.
01:35:00 Chris Fry: Finding what Gen-AI is good at is vital. Sharing what we find it is, too
01:35:13 Mimi Melkonian: Thank you very much!
01:35:45 Elisa Alvarez Silva: Thank You! Very interesting and helpful!
01:36:14 Vincent Everett: Ah. So copilot uses ChatGPT in the same way that it uses DALL-E?
01:36:27 Helen Myers: If you have to go, feel free to thank Dominik in chat and we will pass this on!
01:36:28 Chris Fry: The same prompt may give you different results in Copilot and ChatGPT
01:36:42 Svetlana Kandybovich: It could also be about the policies and temperatures they are using.
01:36:54 Jennifer Eddy: Thank you, Dominik.
01:37:31 Helen Myers: Great to see you Jennifer!
01:40:29 Svetlana Kandybovich: It works better when you give a few particular examples {input-output} to the model in the prompt. Not working 100% though.
01:45:01 Chris Fry: It probably doesn’t understand that there are different languages. It just deals with language/communication.
Even computer code is part of ‘language’
01:46:24 Vincent Everett: It’s the same as sentences ending in apple. It can’t work from the end backwards.
01:46:42 Chris Fry: Replying to “It probably doesn’t …”
just so many token
01:46:46 Helen Myers: It seems that it is more incredible that it gets some of them right having seen the token illustration.
01:47:10 Chris Fry: Replying to “It probably doesn’t …”
just so many tokens
01:47:14 Jeanne Y: Well, now I know (at the least I have an answer my mind is happy with) – Why is ChatGPT horrid when used in Norwegian (bokmål)? tokens Why does ChatGPT not give me what I want, after my delightful, clear prompt and examples? tokens Do not even try math on ChatGPT – reason tokens. Word of the evening -tokens. The telling story vs. free-style rap comparison really was helpful. Thank you Dominik
01:47:38 Helen Myers: Agreed, Jeanne!
01:47:49 Chris Fry: Reacted to “Well, now I know (at…” with 😉
01:48:57 Elizabeth Maldonado: Thank you very much! A lot of food for thought.
01:49:12 Elizabeth Maldonado: How do I erase an emoji going wrong? Thanks!
01:55:08 salamas@cardiff.ac.uk: Q: Are there AI tools that can help in teaching and learning Arabic?
01:55:23 Vincent Everett: Thank you!!
01:58:38 Chris Fry: Thank you, so much, Dominik!
01:59:31 Svetlana Kandybovich: Many thanks, Dominik!