Our global village is becoming smaller with each passing day, and with it, the need to learn new languages is growing ever more present. While some have postulated that advancements in AI will make the need to learn a new language redundant, with earpieces reminiscent of Douglas Adams’ Babel Fish rendering interpreters superfluous, this reality is still a long way away, and ignores other drivers to learn languages.
It’s unlikely that advancements in technology will completely negate the need and desire for humans to learn languages, but they could make language-learning more accessible to those who are unable to afford or attend classes. What’s certain is that the AI revolution spurred on by Open AI’s launching of ChatGPT is having, and will continue to produce, ramifications and opportunities for the EdTech sphere.
With this in mind, on the 25th and 26th of March 2023, the bitmark Association held their first hackathon in Bern for their markup language, bitmark, “the code that makes education accessible” . bitmark, created by Thomas Gabathuler, is the strictly content-first open-source standard for creating learning content. What does this mean for non tech-minded teachers? This means that the number of tools available to create interactive, gamified exercises for your language students will soon be on the up. You can already try out a few bits at UPUCATE.
The hackathon saw 15 teams from all over the world tuning in, both in person and online, to create an open language learning app based on bitmark in twenty-four hours (to the participants’ dismay, this turned out to be twenty-three hours due to daylight saving time). In this article we will explore one of the solutions and highlight how we were able to integrate bitmark and ChatGPT to create a language-learning app, Julia, before suggesting future avenues for innovation.
The Concept
The final proposed apps were to be judged by a panel based on the following parameters: scale of impact, degree of innovation, level of creativity, depth of accessibility, and integration and utilisation of bitmark. With regard to the scale of impact, our idea was to build one learning app that catered to the needs of every user, using few-shot learning that would theoretically be applicable to different areas, not only language acquisition. For our specific use case with our app, Julia, we focussed on a template directed towards learning English, but it would be perfectly possible to scale this up to include other languages.
Iona, a language teacher in our team, emphasised that a successful language learning app should enable the student to practise both receptive and productive skills, and collect information on their native language, target language level, aspirations, and interests. The app would use this information to determine which grammatical topics to teach, such as prepositions of place or conditional structures, and allow users to choose personalised topics for a CLIL-based learning experience. When the user opens the app up, they would be prompted to list their interests, such as fishing, cycling and travel. This would then provide the app with a personal topic that it would be able to combine with the aforementioned grammatical topics.
Once we knew what inputs we needed from the end user, we created a scenario template, something that would be used to generate endless personalised scenarios for each user. This is what made our solution innovative, in the speed at which it would be able to adapt input to reduce lesson creation, which would take a teacher hours, days, or weeks for larger curriculums, to less than a couple of minutes. It also allowed our solution to adhere to the requirement of creativity, in that it would combine human and machine competencies to mix and match ideas to adapt to the learner’s specific needs.
Lesson Concept
Anyone who has ever tried to learn a language will know, immersion is key. It is from this acknowledgement that the CLIL Methodology (Content and Language Integrated Learning) was developed, whereby students learn their target language through studying a different topic. This methodology has been implemented throughout European schools in various years, with students studying Natural Sciences, or Business Studies in English.
It is certainly difficult to recreate this learning environment in an app, but difficult does not mean that one can’t be aspirational. When learning a language, one must train two different types of skills, receptive, that of reading and listening, and productive, that of writing and speaking. Some apps on the market attempt to do this, such as Duolingo, but while their app successfully employs Spaced Repetition Theory to enable users to memorise a wide range of vocabulary in a short space of time, perhaps the brevity of their exercises doesn’t allow for full immersion. However it’s worth noting that Duolingo recently launched the ‘stories’ feature on their app, which allows students to read and listen to longer texts. Another app, Elsa Speak, focuses on improving the users’ speaking and reading skills. Our lesson concept was based on including activities to practise all of these receptive skills, but in activities that were longer than those found in other language-learning apps.
The exercises to practise receptive skills would feature first in each learning module, while those aimed at writing and speaking would feature later, to allow the student to first absorb the information before trying to practise it. Each skills-based activity would be followed by ‘concept checking’ questions, which would allow the app to assess how much of the listening (See Image 2) or reading task (See Image 3) the user had understood.
It’s worth noting that the listening comprehension task we generated was generated as a transcript rather than an audio file (See Image 2).
Bitmark Standard - Bits
Bits are the smallest data pieces that are bundled into a book which represents a unit. There are static bits for information and quiz bits for interactive tasks and assignments. For the purposes of our use case we elected to only use a selection of bits, including “multiple choice”, “cloze”, “true or false”,“matching” and “flashcard” for the concept checks and “message”, “essay”, “article” and “chapter” to structure the lesson. A bit has a human readable representation called “bitmark”, similar to markdown. It can be parsed to a well-structured machine readable bit-JSON format that allows the use of the learning content across multiple platforms without the need of integration. For example, a cloze bit looks like this
[.cloze] This sentence is a [_cloze][_gaptext][!noun] with [_two][?one or two]
gaps including an instruction for the first and a hint for the second gap.
The respective bitmark JSON looks as follows
{
"markup": "[.cloze] This sentence is a [_cloze][_gap text][!noun] with [_two][?one or two] gaps including an instruction for the first and a hint for the second gap.",
"bit": {
"type": "cloze",
"format": "text",
"item": [],
"lead": [],
"instruction": [],
"hint": [],
"body": [
{ "type": "text", "text": "This sentence is a " },
{
"type": "gap",
"solutions": ["cloze", "gap text"],
"item": [],
"instruction": [ { "type": "text", "text": "noun" } ],
"hint": [],
"isCaseSensitive": true,
"isExample": false,
"example": []
},
{ "type": "text", "text": " with " },
{
"type": "gap",
"solutions": ["two"],
"item": [],
"instruction": [ { "type": "text", "text": "noun" } ],
"hint": [ { "type": "text", "text": "one or two" } ],
"isCaseSensitive": true,
"isExample": false,
"example": []
},
{ "type": "text", "text": " gaps including an instruction for the first and a hint for the second gap." }
]
},
...
}
See bitmark for all available bits. The advantage of bitmark over other standards is the strict separation of the data and the visual representation. This is especially beneficial in the educational area as learning content should be integrable across multiple platforms without any barriers to make the content as much accessible as possible.
ChatGPT
In order to enable Julia to scale and generate multiple new learning scenarios for the users, we decided to integrate ChatGPT. To allow everyone to reproduce the results, we use the ChatGPT 3.5 turbo version which is also publicly available. ChatGPT supports few-shot learning as it is able to generalise quickly from only a few and sometimes even one example. Few-shot learning is a type of machine learning method where the training dataset contains limited information. In our case, ChatGPT has seen – if at all – only a few examples. However, the model is still able to quickly learn what to do by providing a context on which it can build upon. This is exactly what we did in the first step.
Process
The Process consists of four steps, namely
(I) Prompt Input
(II) User Input and
(III) ChatGPT-Processing and
(IV) User Individual Output
(I) Prompt Input
This part is hidden from the user and provides ChatGPT with the required context information and instructions so that it is able to make sense of the user input.We created a fictional character Julia, the reference lesson concept and the instruction.
Interests: Gothic Architecture, Hiking, Japanese, Cuisine, Travel, Neomodernist Architecture, Cooking.
Learning objectives: speak more fluently about work, improve use of prepositions of time and place, and learn more phrasal verbs.
Job: Architect.
Native language: Spanish.
Current Target Language Level: English, B2
Aspirational Level: C1.
She asked for a lesson according to the previous information. The following is a lesson based on one of the aforementioned interests:
[.chapter]
[# Neomodernist Architecture]
[.message]Hello Julia! You are in Edinburgh, walking through the city to your new job, and encounter impressive neomodernist buildings. You will practice your English skills while exploring your passion for neomodernist architecture. Let's begin!
[.chapter]
[# Reading Comprehension]
[.essay&article]
[!Read the following article about a famous neomodernist building in Edinburgh]
[&article:Edinburgh's Quadrancy Building is a shining example of neomodernist architecture in the heart of the city. Designed by architect Nina Lancaster, it serves as a cultural center where art, technology, and community come together. The building's design emphasizes key principles, such as connectivity and transparency that make it a welcoming public space. The open-concept design connects different areas within the building, promoting fluid movement and interaction. The exterior features vast glass surfaces, allowing natural light to flood the interiors and creating a sense of openness. Additionally, the Quadrancy Building utilizes energy-efficient features, sustainable materials, and green spaces to promote eco-conscious living in the city.]
[.assignment] Answer the following questions after reading:
[.essay] 1. Who is the architect of this building?
[.essay] 2. What is the building used for?
[.essay] 3. How does the architecture contribute to the building's purpose?
[.chapter]
[# Concept Check 1: Multiple Choice]
[.assignment]Based on the reading comprehension task, answer the following multiple choice questions:
[.multiple-choice-1]
[! 1. What style of architecture does the building have?]
[-Neoclassical] [+Neomodernist] [-Gothic]
[! 2. What is the main purpose of the Quadrancy Building?]
[-Business offices] [-Shopping center] [+Cultural center]
[! 3. Who is the architect of the Quadrancy Building?]
[+Emma Thompson] [-Victoria Beckham] [+Nina Lancaster]
[.chapter]
[# Concept Check 2: Fill in the blanks]
[.cloze] The building's design was influenced by the concept of [_connectivity] connectivity and [_transparency].
[.chapter]
[# Listening Comprehension Task]
[.interview]
[!Listen to a podcast episode about neomodernist architecture and its impact on urban spaces and answer the questions:]
===
1. What are the key principles of neomodernist architecture?
===
2. How does this architectural style influence urban life?
===
[.chapter][#Concept Check 3: True or False]
[.true-false-1]
[+Neomodernist architecture promotes the use of organic materials.]
[+ Neomodernist architecture aims to create more green spaces in urban areas.]
[.chapter][#Concept Check 4: Matching]
[.match][!Match the following vocabulary from the listening comprehension task with their definitions:]
===
[#Noun]
==
[#Noun]
===
Sustainable development
==
Balance between economic growth, social well-being, environmental protection.
===
Minimalism
==
Minimalism: less is more, simplicity is key.
===
[.chapter][#Writing Task]
[.essay]Write a 150-200 word email to a friend describing your experience working at your new job, and how the neomodernist environment impacts your daily life.
[.chapter][#Speaking Task]
[.record-audio]
[!Discuss aspects of neomodernist architecture using the vocabulary and grammar from the previous tasks. Record yourself speaking for 2-3 minutes.]
[.chapter][#Summary Task: Vocabulary Table]
Flip the card to revise the vocabulary from this chapter.
[.flashcard-1]
Neomodernist [!Native Language]
==
Neomodernista
[.flashcard-1]
Minimalism [!Native Language]
==
Minimalismo
[.flashcard-1]
Sustainability [!Native Language]
==
Sostenibilidad
[.flashcard-1]
Connectivity [!Native Language]
==
Conectividad
[.flashcard-1]
Transparency [!Native Language]
==
Transparencia
Produce a lesson with the same structure but adapted to one of the interests of the user.
(II) User Input
The user input is the basis for the personalisation of the provided lesson concept. As described, it includes personal preferences and the learning setting. Below, you can find Maria’s profile.
Hello, my name is Maria.
Interests: Arts and crafts, hiking, Jazz music.
Learning objectives: to speak more fluently about work, improve the use of the definite and indefinite articles as well as the perfect tenses.
Job: Manager of a Jazz club.
Native language: Polish.
Current Target Language Level: English, B1
Aspirational Level: B2.
(III) ChatGPT
In this part, the lesson concept is modified so that it matches the learners demands. This includes (a) the difficulty of the tasks, (b) the topic the tasks deal with and (c) the overall generation of texts and tasks in the bitmark format so that we end up with the Output.
(IV) Output
The generated output is a personalised lesson similar to the provided lesson concept with fully interactive tasks in the bitmark markup format. It can be parsed to bitmark JSON and is ready-to-use within any learning app or platform that supports the standard.
[.chapter]
[# Arts and Crafts]
[.message]Hello Maria! In this lesson, we will explore your passion for arts and crafts while practicing your English skills. Let's begin!
[.chapter]
[# Reading Comprehension]
[.essay&article]
[!Read the following article about an artist who uses recycled materials in her work]
[&article:Emma Smith is a contemporary artist who specializes in creating art with recycled materials. She uses a variety of everyday objects like paper, plastic, and metal to create intricate sculptures that challenge the viewer's perception of waste. Smith's work is not only visually stunning but also carries an important message about sustainability and the impact of consumer culture on the environment. In one of her most recent exhibitions, she showcased a series of sculptures made entirely out of discarded plastic bottles. Visitors were encouraged to reflect on their own consumption habits and the role they play in creating the global waste crisis. Smith's work is a testament to the power of art to create change and inspire action.]
[.assignment] Answer the following questions after reading:
[.essay] 1. What is Emma Smith's specialty as an artist?
[.essay] 2. What materials does she use in her sculptures?
[.essay] 3. What is the message behind Smith's art?
[.chapter]
[# Concept Check 1: Multiple Choice]
[.assignment]Based on the reading comprehension task, answer the following multiple choice questions:
[.multiple-choice-1]
[! 1. What is Emma Smith's message with her art?]
[-To create beautiful sculptures] [+To raise awareness about sustainability] [-To make a profit]
[! 2. What materials does Emma Smith use in her art?]
[-Precious stones and metals] [+Recycled materials] [-Artificially made materials]
[! 3. What is the purpose of Emma Smith's recent exhibition?]
[+To make people reflect on their consumption habits] [-To showcase her artistic abilities] [-To sell her sculptures]
[.chapter]
[# Concept Check 2: Fill in the blanks]
[.cloze] Emma Smith is an artist who specializes in creating art with [_recycled] materials.
[.chapter]
[# Listening Comprehension Task]
[.interview]
[!Listen to a podcast episode about the benefits of arts and crafts and answer the questions:]
===
1. What are some of the benefits of participating in arts and crafts activities?
===
2. How can these activities improve mental health?
===
[.chapter][#Concept Check 3: True or False]
[.true-false-1]
[+Participating in arts and crafts can reduce stress levels.]
[+Arts and crafts can improve cognitive function.]
[.chapter][#Concept Check 4: Matching]
[.match][!Match the following vocabulary from the listening comprehension task with their definitions:]
===
[#Noun]
==
[#Definition]
===
Therapeutic
==
Relating to the healing of disease or injury.
===
Creativity
==
The use of imagination or original ideas.
===
[.chapter][#Writing Task]
[.essay]Write a 150-200 word email to a friend describing your experience working at your new job and how you plan on incorporating arts and crafts into the Jazz club's atmosphere.
[.chapter][#Speaking Task]
[.record-audio]
[!Discuss the benefits of arts and crafts using the vocabulary and grammar from the previous tasks. Record yourself speaking for 2-3 minutes.]
[.chapter][#Summary Task: Vocabulary Table]
Flip the card to revise the vocabulary from this chapter.
[.flashcard-1]
Recycled [!Native Language]
==
Przetworzony
[.flashcard-1]
Therapeutic [!Native Language]
==
Terapeutyczny
[.flashcard-1]
Creativity [!Native Language]
==
Kreatywność
You can reproduce this process using UPUCATE’s page here. No doubt you’ll soon see the benefits – one can produce personalised lesson plans for individual students based on their levels of comprehension and interests in minutes, rather than hours.
For autodidactic students, there are multiple benefits to using an app that implements this process. Not only will you be able to learn with an environment personalised to your language needs and interests, but it would arguably be more accessible in terms of price and availability than hiring a personal tutor or attending group classes, the only obvious downside being the lack of person-to-person interaction. Basing the lesson structure on the CLIL methodology allows for a unique, engaging and enjoyable learning experience, which should ultimately result in better academic outcomes and increased target language proficiency.
Reflection
We want to share our experience also with regards to the things we tried that did not work out. Most of the time, we spend on prompt engineering. At the beginning, we provided ChatGPT with context independent sample bits which did not work at all. After creating an exact bitmark lesson concept in combination with the provision of all bits, we achieved first reliable results. Still, the quality was bad and many structural errors occurred. We decided to reduce the possible bits to a bare minimum so that the concept still has various tasks but GPT has less freedom. This step brought us closer to the final result and we already had a good performance.
For the final optimisation we also provided a fictive persona (Julia) and instructed GPT to produce the provided lesson concept for the given persona. Given that, we ended up with a proper personalised result and only a few remaining structural bitmark errors.
Summarised, small changes in the way we designed the prompt led to large changes in the result. We are optimistic that it is possible to produce nearly error-free outputs with any bit, which is subject to future work.
Qualitative Evaluation of a Generated Lesson Concept
The breakdown of the vocabulary according to the CEFR level in the reading comprehension text produced by ChatGPT is as follows: A1: 40.48%, A2: 13.10%, B1: 26.19%, B2: 15.48%, C1: 2.38% AND C2: 2.38%. The majority of vocabulary at A1 and A2 level pertained to articles, prepositions and simple verbs, which is to be expected, but there was a sufficient number of words at the user’s current and aspirational target language level to enable productive vocabulary acquisition. Aside from vocabulary, the grammar used is more suitable for B1 level, the current level of the user, as the present simple and past simple are mainly used, with one example of the past simple passive, “were encouraged”. Given that this is one scenario, and the first activity, this is not necessarily an issue.
The reading comprehension task is followed by some open ended questions and a concept check exercise. ChatGPT repeats some of the questions here, “What is Emma Smith’s message with her art?” / “What is the message behind Smith’s art?”, “What materials does she use in her sculptures?” / “What materials does Emma Smith use in her art?”, basically rendering either the concept check or the open ended questions redundant. More variety would be needed in future iterations.
The concept check could also focus more on the vocabulary. E.g. “What word is used that has a similar meaning to, “to exhibit?” Answer: showcased / to showcase. Only one example sentence has been given for Concept Check 2: Fill in the blanks. This is not enough to foster successful language acquisition.
ChatGPT didn’t generate any text for the Listening Comprehension task, so it’s impossible to assess whether or not it adheres to the desired CEFR level. It produced two open-ended questions about the imaginary text and two true/false questions. As with the concept checks for the reading comprehension, this is not sufficient. Following this, it only produced two options for the vocabulary matching exercise.
Without integrating an automatic text checker such as that from LanguageTool, or a Speech Recognition API, the writing and speaking tasks wouldn’t be of much use as the user wouldn’t be able to receive feedback.
All in all, the output constitutes a promising demonstration on how ChatGPT can be used to produce bitmark, but falls short on delivering an experience that would benefit the user in the long term. It would need integration with a number of APIs to allow for precise, actionable feedback and the production of listening tasks. Moreover, none of the concept checks addressed the learning objectives section of user input, in this case, “improve the use of the definite and indefinite articles as well as the perfect tenses.” More work would need to be done to enable ChatGPT to produce exercises to practise specific grammatical topics, and perhaps insert bits that allow for comparisons between grammatical concepts in the native language and the target language. ChatGPT’s default variant for English seems to be American English, a minor issue for any students wanting to learn the British variant for specific exam needs.
Looking at the structure of the output with regards to the bitmark standard, it can be said that no mistake was made.
Remarks & Future Work
Our work has certain limitations, mostly due to the time restriction. We did not integrate and test all bits which would be a good enhancement as diverse lessons are imperative for maintaining the user’s interest and allowing them to practise a variety of receptive and productive skills. As mentioned before, ChatGPT only produces text. To make the learning experience smooth, it would be imperative to integrate a text-to-speech API to enable the production of audios from the text generated by ChatGPT. It would also be imperative to integrate an intelligent text corrector API such as that offered by LanguageTool to enable instant feedback on a user’s response. The lesson structure finishes with some concept-checking questions and a flashcard activity with the five most important words from the lesson.
Since ChatGPT is limited to 4000 tokens, it is not possible to provide all bits in the prompt. That is, the user needs to change the provided Lesson Concept and the bits in the Input prompt. In the future, we want to create an interface that translates lesson concepts and the used types of tasks into bits and dynamically provide it to GPT. Furthermore, to exploit the full power and get even better results, we will evaluate the approach with GPT4.
It’s also important to note that we fell short when it came to the parameter of depth of accessibility. Julia would theoretically address accessibility issues in terms of giving students who can’t afford access to a private tutor the opportunity to learn with an app that utilises teacher competences. Moreover, it is accessible in the fact that it adapts lesson concepts to learners’ competences and issues. However, if we’re talking about needs-based accessibility issues, this was not addressed by us. Our shortcomings would need to be addressed in future iterations, where we would be able to collect more input from the user regarding needs such as colour blindness, auditory and visual competencies.
Advancements in AI have had and will continue to have significant effects on the ambit of education. When combined with bitmark, our solution shows that it’s possible to scale up lesson planning, the implications of which are certainly exciting. It’s clear that there are limitations that ought to be addressed in future to really make a difference. Nonetheless, it will be interesting to see how AI continues to evolve in the EdTech space, and what new opportunities and challenges arise.
Authors: Iona Douglas and Jonas Nöhl
15.05.2023
