About of the metric homogeneity of texts in Slavic languages

Abstract

In the studies of R. Gray and K. Atkinson by the statistical analysis of related words, W. Chang, C. Cathcart, D. Hall and A. Garrett using statistical modeling and A. S. Kasyan and A. V. Dybo on the basis of lexicostatistical classification, in addition to discussing historical issues, geneological trees are presented, reflecting both kinship and divergence of modern Slavic languages. There are a lot of such trees, they are similar in general terms and differ in small details, see, for example. The area of the formerly common language is now divided into three groups - the eastern one, consisting of the Belarusian, Russian and Ukrainian languages, the western one from the Czech, Slovak, Polish, Kashubian and Lusatian languages, and the southern one, consisting of the Bulgarian, Macedonian, Serbo-Croatian and Slovenian languages. Using the example of a randomly generated model collection of 26 texts in 13 languages (2 works from each language), the article establishes the applicability of the γ-classifier for automatic recognition of the belongingof texts to a particular group of Slavic languages based on the frequency of a set of Latin characters that is universal for all languages. The mathematical model of the γ-classifier is presented in the form of a triad composed of a digital portrait (DP) of the text - the distribution of the frequency of Latin symbolic unigrams in the text; formulas for calculating the distances between DP texts and a machine learning algorithm that implements the hypothesis of “homogeneity” of works from one language group and “heterogeneity” of works belonging to different groups of languages. The tuning of the algorithm using a table of paired distances between all products of the model collection was carried out by selecting the optimal value of the real parameter γ, which minimizes the number of errors in violation of tho “homogeneity” hypothesis. The e-classifier trained on the texts of the model collection showed 86% accuracy in recognizing the language groups of the works. To test the classifier, 3 additional random texts were selected, one text each for three different groups of Slavic languages. By the method of the nearest (in terms of distance) neighbor, all new texts confirmed their homogeneity with the corresponding pairs of monolingual works, thereby also homogeneity with the corresponding group of Slavic languages.