A context-aware lemmatization model for setswana language using machine learning

Lemmatization is an important task which is concerned with making computers understand the

relationship that exists amongst words written in natural language. It is a prior condition needed

for the development of natural language processing (NLP) systems such as machine translation

and information retrieval.

In particular, Lemmatization is intended to reduce the variability in word forms by collapsing

related words to a standard lemma. There is a limited research on lemmatization of Setswana

language. A large part of the available research on Setswana lemmatization relies on rule driven

strategy, which takes time to construct, lacks context of how words are used, and needs extremely

qualified language skills. Moreover, it has been discovered that the treatment of language with

hand coded regulations lacks generalization component as it requires a continual redesign every

time new data appears and this complicates the scalability of systems. With such rich vocabulary

and complex morphology, lemmatization of Setswana cannot be easily unraveled using explicit

rules developed by programmers.

In this thesis we describe how a supervised machine learning approach that employs the use of

Naive Bayes algorithm can solve Setswana lemmatization with regard to how words are used in

sentences. The contribution of this study includes; first, context aware lemmatization model,

that handles most of the morphologically productive classes. Second, we experiment with the

strongest multi-class algorithm Naive Bayes, which to our best knowledge has never been used

to address lemmatization in Setswana. The accuracy of the lemmatization model obtained from

the experiments reached 70.32%. The model shifts from entirely hand programmed rules and is

able to lemmatize words based on the context how they are used. In Setswana lemmatization

should be done according to sentence intension, the model again ensures that as long as the

data is a good example of the goal concept the generalization is simultaneously created, which

allows the model'’s future performance to continue improving.

Furthermore, given that this is a young area of research with no standard datasets for training

and testing, we also contribute with a considerable medium sized dataset which remains a coveted

resource for research community. The experimental results obtained from this study shows

that machine learning approaches are more reliable than rule based approaches in lemmatizing

Setswana inflectional words with regard to the context of how they are used.