A language immersion tool for batch sentence mining.

Overview

Immersion only works at your level

The biggest jump in vocabulary in the first few months of learning a language is going from textbook and children's content to media on topics you actually enjoy, due to the large gap in domain-specific language.

Immersion is supposed to close that gap, but only pays off when you already understand most of what is presented to you. Below that line, the immersion content is indistinguishable from noise. The hard part in this process for me was finding Telugu media at my level before spending time on it.

Lack of Telugu NLP infrastructure

Telugu does not have substantive NLP infrastructure to support a sentence mining pipeline. There is no popular lemmatizer, and because Telugu is agglutinative, it appends suffixes for case markers, pronouns, and tense. So, we have to account for countless combinations of inflected forms compared to English.

Solution

Swaralu is a language immersion tool. At its core, is a trackable pop-up dictionary for all media formats: video, audio, and text. The pipeline turns phrases you look up into flashcards you can review using a spaced repetition system like Anki. It currently ships Telugu, but can be generalized for any language.

I started building Swaralu four months into learning Telugu, because I love Telugu media, especially music, and wanted to effectively learn from immersion.

Guided immersion

View a Telugu web video or audio, or open a text page. With the Chrome extension active, Swaralu renders the text as clickable tokens. These are subtitle cues that can override the native web subtitles. Hovering over a phrase defines the word with a dictionary popup. Every word tracks a known status (unknown or tracking or known or ignored), with inline color indicators, so a transcript becomes an intuitive map of what to learn next.

Flashcards

A system of building cards which relies on only having the English word and it's counterpant in your target language facilitates a review style leaning heavy on rote memorization or mnemonics. The deeper problem is that this card format abstracts away the context of the word. The context surrounding a word extends to the enveloping sentence, the paragraph, even the entire body of text. It would be unreasonable to jam all of this information into each flashcard, but you can structure the vocabulary so that it jogs that part of your memory and resurfaces the full context.

In Swaralu, word pop-ups can be sent to the card creator in one click, which fills flashcard fields with the root lemma, the definition, the example sentence the word was found in, English transliterations, an audio clip from the video/audio source, and a screenshot from the video source.

This card format pulls the vocabulary from the source without loss of context, and includes visual, audible, and contextual cues to target various learning styles. The direct word definition of your target language also means that you are learning the actual sentiment of the target word, as languages often do not have direct translations with identical sentiment.

Cards can be saved in the app or exported to Anki to review through spaced repetition. Swaralu also supports a native media player, allowing you to upload your own video, audio, and text media.

Recommendation System

To close the gap between introductory learning and immersion learning, Swaralu features an extensive media library that features videos, podcasts, music, poems, books, lessons, and movies. The recommendation system is tailored towards a user's vocabulary level, with comprehension scores for each media item which grades the media based on the approachability of the transcript and vocabulary.

Language Learning

Comprehensible input

Comprehensible input is a popular, well-reviewed technique in the language learning community theorizing that you acquire a language best by understanding sentences just beyond your current level, which he wrote as i+1, and the practice of mining sentences out of native media into flashcards was popularized by the online Japanese learning community.

n+1 is a general rule describing a sentence where you know every word but one, allowing you to "mine" the word in context of the sentence.

This also exposes a limit in the comprehension score, which is that knowing you know some percentage of the words in a video does not tell you the density of n+1 sentences. I estimate the usable threshold as somewhere around 50% when close studying and 70% to for seamless use.

Swaralu transcript view allows you to select all n+1 sentences at once and send them all to the card creator for batch sentence mining. The comprehension score also gives information on the number of n+1 and recommended sentences to learn.

n+1

Word is easily mineable in context of the surrounding sentence.

n+3

Many unknown words can make it hard to anchor vocabulary to context.

The cold-start problem

The largest directional shift I encountered when building was how to revamp this technology from serving someone like myself, who is very familiar with popular language learning systems, to users who may find the most value out of this software, like heritage Telugu speakers or new learners. Most people have never used Anki, which the cold-start relied on for importing a list of all your known words in order to get value out of the comprehensible input ladder.

The inevitable tradeoff is to sacrifice the exact accuracy of the known statuses in a user's vocabulary to make the onboarding process of assess a user's vocabulary level as efficient as possible. For new users, this exists in the form of a 5-minute long assessment, where users select words in a list that they know, and we algorithmically iterate until we can confidently conclude a likely list of words that the user knows based on semantically and lexically similar words that they know. As more users onboard this way, tracking analytics and false positives can be used to refine the ML model to improve the accuracy of the onboarding flow.

Text processing

Telugu is an agglutinative language, which means it appends suffixes for case markers, tense, subject agreement, etc. So string matching will miss most words. Instead, we need an NLP model which can process words, chunks, and sentences into their lexical attributes, like surface form.

Reflection

Building a tool for a language I am actively learning as a non-native means I can research the grammar and check output to a decent standard, but the vision and scope are bounded by what a learner knows to look for. In a case like this, it is critical to identify the target audience and market. I've built with the aim of serving heritage speakers and new learners, which is the area I can best service as a new learner myself. I rely heavily on a robust user feedback loop and native Telugu speakers as advisors to guide my project direction.

Beyond Telugu, the infrastructure can be generalized to any language. It is mostly a matter of finding a good NLP model for the target language to identify lexical boundaries, so tokens can be analyzed in the context of their source.

Swaralu is still in development and is rolling out soon to beta testers. If you are working on something similar, or are interested in this project, feel free to reach out.