Engineering language solutions using neural networks

MSc students in the Department of Earth Science & Engineering are using machine learning to unravel extinct languages in a hackathon style project

Our Applied Computational Science and Engineering MSc students enjoy the challenge of intensive mini-projects for each module of the programme. For the mini-project associated with the Machine Learning module, the 31 students worked in groups of three or four on a hackathon style project to develop Neural Networks. Artificial neural networks are a class of machine learning models that are inspired by biological neurons and their connectionist nature.

The work was based on the application of neural networks to the prediction on Japanese Kuzushiji characters, a cursive Japanese script that has been used for over 1000 years. In the 19th century, Japan reformed its official language and over time Kuzushiji became extinct, causing millions of documents of Japanese culture and history to be inaccessible for most people. Through the creation of Machine Learning models, these documents can be deciphered and texts translated.

As you can see, the applications of machine learning stretch far and wide, from solving engineering and geoscience problems, to language.

Language Dataset

The eight groups used the k-MNIST dataset, which contains 60,000 (28x28) grayscale images of hand-written Kuzushiji characters. There are ten different specific characters, or hiragana, written in a range of ways in k-MNIST. A first glance at the data reveals their large variation within each class. In the image below, the first column represents the reference character for each of the classes. Looking along each row the variation in the representation of each classes is shockingly different, making the exercise quite difficult, even for the human eye.

The k-MNIST dataset is available on the Github and Kaggle, which is often used for hackathons, as it provides a great interface for such challenges, and was used for the mini-project as it was very similar to a hackathon.

Developing the Neural Network

The aim was for each group to develop an optimal neural network and training approach on the 60,000 labelled images, that would yield the highest accuracy for recognizing the characters.

For testing the performance of each neural network, 10,000 other images were used as a test dataset, with their labels hidden to the group. Twice a day, each group was allowed to test its current neural network on the test set, being given the accuracy (proportion of correctly labelled characters) of their network on the test set. Note that the current state of the art networks, based on Resnets, perform at 98.9% test accuracy on k-MNIST test set (see pdf).

In order to avoid the groups just picking the best existing Resnet implementation from the Internet, the groups were constrained to use only “vanilla” convolutional neural networks. Therefore, they had to focus on other ways to improve their convolutional neural networks, by using such approaches as Data Augmentation, Drop-Out, Transfer Learning or Clustering.

Deep Learning

Professor Olivier Dubrule was impressed by the students: “To our pleasant surprise, after just 4 days of hard work, the eight groups obtained some accuracies on the test set between 97.75% and 98.83%; that is performances close to those of the more sophisticated networks that they were not allowed to use!”

To the module teachers, this was a proof first of the quality of the students, but also that their main goals as instructors had been achieve: The basic foundations of Deep Learning had been fully grasped by the groups, together with the Python implementation challenges (the Pytorch library was used) associated with the exercise.

The Applied Computational Science & Engineering MSc runs for 12 months from September 2019. Find out more, or get in touch with Ying Ashton, course administrator with any queries: Email Ying