An evaluation measurement in automatic text classification for authorship attribution
Article Sidebar
Published:
Jul 7, 2016
Keywords:
classification systems, evaluation measurements, authorship attribution
Main Article Content
Abstract
In authorship attribution, the task of correctly assigning an anonymized document to an author within a predefined set of subjects, various measurements to evaluate classification systems have been used in the research literature. As will be discussed in this article, some of these measurements may differ diametrically. For research purposes, the evaluation of an automatic text classification system, such as the one that may be used for authorship attribution, may report a number of different performance measurements. However, some of the previously used figures are either too optimistic or lack generalizability. In addition to this issues, law-oriented research has pointed out the importance of having an error rate for the legal admissibility not only of this type of text classification task but of any piece of potential evidence in general. Considering the circumstances, the use of a single measurement in authorship attribution is proposed in this paper. Also, the implications of using this figure instead of others presented by researchers are discussed. At the same time, the importance of presenting this measurement along other relevant experimental settings, such as the number of categories (or authors in this context), is explained. The discussion is supported with the presentation of a set of authorship attribution experiments that utilize data from users of crime-related social media.
Downloads
Download data is not yet available.
Article Details
How to Cite
Rico Sulayes, A. (2016). An evaluation measurement in automatic text classification for authorship attribution. Ingenio Magno, 6(2), 62-74. Retrieved from http://revistas.ustatunja.edu.co/index.php/ingeniomagno/article/view/1093
Section
Artículos Vol. 6-2
DECLARATION OF ORGINIALITY OF SUBMITTED ARTICLE
With this document, I/We certify that the article submitted for possible publication in the institutional journal INGENIO MAGNO of the Research Center Alberto Magno CIIAM of the University Santo Tomás, Tunja campus, is entirely of my(our) own writing, and is a product of my(our) direct intellectual contribution to knowledge.
All data and references to completed publications are duly identified with their respective bibliographical entries and in the citations thus highlighted. If any adjustment or correction is needed, I(we) will contact the journal authorities in advance.
Due to that stated above, I(we) declare that the entirety of the submitted material is in accordance with applicable laws regarding intellectual and industrial property, and therefore, I(we) hold myself(ourselves) responsible for any complaint related to it.
If the submitted article is published, I(we) declare that I(we) fully relinquish publishing rights of the article to the University Santo Tomás, Tunja campus. As remuneration for this relinquishment of rights, I(we) declare my(our) agreement to receive two (2) copies of the edition of the journal in which my(our) article appears.
References
Argamon, S., Šari, M. & Stein, S. S. (2003). Style mining of electronic messages for multiple authorship discrimination: first results. Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
Baayen, H., van Halteren, H., Neijt, A. & Tweedie, F. (2002). An experiment in authorship attribution (pp. 29-37). Proceedings of JADT 2002: Sixth International Conference on Textual Data Statistical Analysis.
Bolle, R. M., Connell, J. H., Pankanti, S., Ratha, N. K. y Senior, A. W. (2004). Guide to biometrics. New York: Springer-Verlag.
Burns, R. B. & Burns, R. A. (2008). Business research methods and statistics using SPSS. UK: Sage.
Burrows, J. (2002). Delta: a measure of stylistic difference and a guide to likely authorship. Literary and Linguistic Computing, 17(3), 267-86.
Chaski, C. E. (2005). Who’s at the keyboard? Authorship attribution in digital evidence investigations. International Journal of Digital Evidence, 4(1), 1-13.
Chaski, C. E. (2007). The keyboard dilemma and authorship identification. In P. Craiger & S. Shenoi (Eds.), Advances in Digital Forensics III (pp. 133-146). New York: Springer.
Foros Blog del Narco. (2010). Retrieved from http:// www.foro.blogdelnarco.com/
Grant, T. (2007). Quantifying evidence in forensic authorship analysis. International Journal of Speech, Language and the Law, 14(1), 1-25.
Grieve, J. (2007). Quantitative authorship attribution: an evaluation of techniques. Literary and Linguistic Computing, 22(3), 425-442.
Howald, B. S. (2008). Authorship attribution under the rules of evidence: empirical approaches - a Layperson’s Legal System. International Journal of Speech, Language and the Law, 15(2), 219-247.
Jurafsky, D. & Martin, J. H. (2008). Speech and language processing: an introduction to language natural processing, computational linguistics, and speech recognition (2nd ed.). Upper-Saddle River: Pearson-Prentice Hall.
Manning, C. D., Raghavan, P. y Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge.
McMenamin, G. R. (2002). Forensic linguistics: advances in forensic stylistics. Boca Raton: CRC.
Mikros, G. K. & Argiri, E. K. (2007). Investigating topic influence in authorship attribution. In B. Stein, M. Koppel & E. Stamatatos (Eds.), Proceedings of the SIGIR 2007 International Workshop on Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection, PAN 2007.
Koppel , M., Schler, J., & Argamon, S. (2009). Computational Methods in Authorship Attribution. Journal of the American Society for Information Science and Technology, 60(1), 9-26.
Koppel, M., Schler, J., & Messeri, E. (2008). Authorship Attribution in Law Enforcement Scenarios. In C.S. Gal, P. Kantor, & B. Saphira (Eds.), Security Informatics and Terrorism: Patrolling the Web (pp.111-119). Amsterdam: IOS.
Peng, F., Schuurmans, D., Keselj, V. & Wang, S. (2003). Language independent authorship attribution using character level language models. Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics: Vol. 1 (pp. 267-274). Stroudsburg: Association for Computational Linguistics.
Petrovska-Delacretaz, D., Chollet, G. y Dorizzi, B. (2009). Guide to biometric reference systems and performance evaluation. London: Springer-Verlag.
Rico-Sulayes, A. (2011). Statistical authorship attribution of Mexican drug trafficking online forum posts. International Journal of Speech, Language and the Law, 18(1), 53-74
Rico-Sulayes, A. (2012). Quantitative authorship attribution of users of Mexican drug dealing related online forums (PhD dissertation, Georgetown University). Retrieved from https://repository.library.georgetown.edu/ handle/10822/557726
Rico-Sulayes, A. (2014). Técnicas de reducción, algoritmos resistentes al ruido o ambos. Opciones para el manejo de rasgos clasificatorios en la atribución de autoría. Research in Computing Science, 80.
Solan, L. M. & Tiersma, P. M. (2004). Author Identification in American Courts. Applied Linguistics, 25(4), 448-465.
Solan, L. M. & Tiersma, P. M. (2005). Speaking of Crime: The Language of Criminal Justice. Chicago: University of Chicago.
Spassova, M. S. (2008). Las perífrasis verbales del español en la atribución forense de autoría. In R. Monroy & A. Sánchez (Eds.), 25 años de lingüística en España: hitos y retos. Actas del XXVI Congreso de AESLA (pp. 605-614). Murcia: Universidad de Murcia.
Spassova, M. S. (2009). El potencial discriminatorio de las secuencias de categorías gramaticales en la atribución forense de autoría de textos en español (PhD dissertation, Universitat Pompeu Fabra, Barcelona). Retrieved from http://repositori.upf.edu/handle/10230/12285
Spassova, M. S. & Turell, M. T. (2007). The use of morphosyntactically annotated tag sequences as markers of authorship. In M. T. Turell, J. Cicres, and M. S. Spassova (Eds.), Proceedings of the 2nd European IAFL Conference on Forensic Linguistics / Language and the Law 2006 (pp. 229-237). Barcelona: Documenta Universitaria.
Stamatatos, E., Fakotakis, N. & Kokkinakis, G. (2001). Law, 18(1), 53-74. Computer-based authorship attribution without lexical measures. Computers and the Humanities, 35, 193-214. 73
Tambouratzis, G. & Vassiliou, M. (2007). Employing online forums (PhD dissertation, Georgetown University). thematic variables for enhancing classification accuracy Retrieved from https://repository.library.georgetown.edu/ within author discrimination experiments. Literary and handle/10822/557726 Linguistic Computing, 22(2), 207-224.
Witten, I. H., Frank, E. & Hall, M. A. (2011). Data mining: practical machine learning tools and techiniques (3rd ed.). Burlington: Morgan Kaufmann.
Zheng, R., Li, J., Chen, H. & Huang, Z. (2006). A framework for authorship identification of online messages: writingstyle features and classification techniques. Journal of the American Society for Information Science and Technology, 57(3): 378-393.
Zvetco Biometrics. (2012). Biometric Knowledge Center. Retrieved from http://www.zvetcobiometrics.com/Support/ definitions.jsp
Baayen, H., van Halteren, H., Neijt, A. & Tweedie, F. (2002). An experiment in authorship attribution (pp. 29-37). Proceedings of JADT 2002: Sixth International Conference on Textual Data Statistical Analysis.
Bolle, R. M., Connell, J. H., Pankanti, S., Ratha, N. K. y Senior, A. W. (2004). Guide to biometrics. New York: Springer-Verlag.
Burns, R. B. & Burns, R. A. (2008). Business research methods and statistics using SPSS. UK: Sage.
Burrows, J. (2002). Delta: a measure of stylistic difference and a guide to likely authorship. Literary and Linguistic Computing, 17(3), 267-86.
Chaski, C. E. (2005). Who’s at the keyboard? Authorship attribution in digital evidence investigations. International Journal of Digital Evidence, 4(1), 1-13.
Chaski, C. E. (2007). The keyboard dilemma and authorship identification. In P. Craiger & S. Shenoi (Eds.), Advances in Digital Forensics III (pp. 133-146). New York: Springer.
Foros Blog del Narco. (2010). Retrieved from http:// www.foro.blogdelnarco.com/
Grant, T. (2007). Quantifying evidence in forensic authorship analysis. International Journal of Speech, Language and the Law, 14(1), 1-25.
Grieve, J. (2007). Quantitative authorship attribution: an evaluation of techniques. Literary and Linguistic Computing, 22(3), 425-442.
Howald, B. S. (2008). Authorship attribution under the rules of evidence: empirical approaches - a Layperson’s Legal System. International Journal of Speech, Language and the Law, 15(2), 219-247.
Jurafsky, D. & Martin, J. H. (2008). Speech and language processing: an introduction to language natural processing, computational linguistics, and speech recognition (2nd ed.). Upper-Saddle River: Pearson-Prentice Hall.
Manning, C. D., Raghavan, P. y Schütze, H. (2008). Introduction to information retrieval. New York: Cambridge.
McMenamin, G. R. (2002). Forensic linguistics: advances in forensic stylistics. Boca Raton: CRC.
Mikros, G. K. & Argiri, E. K. (2007). Investigating topic influence in authorship attribution. In B. Stein, M. Koppel & E. Stamatatos (Eds.), Proceedings of the SIGIR 2007 International Workshop on Plagiarism Analysis, Authorship Identification, and Near-Duplicate Detection, PAN 2007.
Koppel , M., Schler, J., & Argamon, S. (2009). Computational Methods in Authorship Attribution. Journal of the American Society for Information Science and Technology, 60(1), 9-26.
Koppel, M., Schler, J., & Messeri, E. (2008). Authorship Attribution in Law Enforcement Scenarios. In C.S. Gal, P. Kantor, & B. Saphira (Eds.), Security Informatics and Terrorism: Patrolling the Web (pp.111-119). Amsterdam: IOS.
Peng, F., Schuurmans, D., Keselj, V. & Wang, S. (2003). Language independent authorship attribution using character level language models. Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics: Vol. 1 (pp. 267-274). Stroudsburg: Association for Computational Linguistics.
Petrovska-Delacretaz, D., Chollet, G. y Dorizzi, B. (2009). Guide to biometric reference systems and performance evaluation. London: Springer-Verlag.
Rico-Sulayes, A. (2011). Statistical authorship attribution of Mexican drug trafficking online forum posts. International Journal of Speech, Language and the Law, 18(1), 53-74
Rico-Sulayes, A. (2012). Quantitative authorship attribution of users of Mexican drug dealing related online forums (PhD dissertation, Georgetown University). Retrieved from https://repository.library.georgetown.edu/ handle/10822/557726
Rico-Sulayes, A. (2014). Técnicas de reducción, algoritmos resistentes al ruido o ambos. Opciones para el manejo de rasgos clasificatorios en la atribución de autoría. Research in Computing Science, 80.
Solan, L. M. & Tiersma, P. M. (2004). Author Identification in American Courts. Applied Linguistics, 25(4), 448-465.
Solan, L. M. & Tiersma, P. M. (2005). Speaking of Crime: The Language of Criminal Justice. Chicago: University of Chicago.
Spassova, M. S. (2008). Las perífrasis verbales del español en la atribución forense de autoría. In R. Monroy & A. Sánchez (Eds.), 25 años de lingüística en España: hitos y retos. Actas del XXVI Congreso de AESLA (pp. 605-614). Murcia: Universidad de Murcia.
Spassova, M. S. (2009). El potencial discriminatorio de las secuencias de categorías gramaticales en la atribución forense de autoría de textos en español (PhD dissertation, Universitat Pompeu Fabra, Barcelona). Retrieved from http://repositori.upf.edu/handle/10230/12285
Spassova, M. S. & Turell, M. T. (2007). The use of morphosyntactically annotated tag sequences as markers of authorship. In M. T. Turell, J. Cicres, and M. S. Spassova (Eds.), Proceedings of the 2nd European IAFL Conference on Forensic Linguistics / Language and the Law 2006 (pp. 229-237). Barcelona: Documenta Universitaria.
Stamatatos, E., Fakotakis, N. & Kokkinakis, G. (2001). Law, 18(1), 53-74. Computer-based authorship attribution without lexical measures. Computers and the Humanities, 35, 193-214. 73
Tambouratzis, G. & Vassiliou, M. (2007). Employing online forums (PhD dissertation, Georgetown University). thematic variables for enhancing classification accuracy Retrieved from https://repository.library.georgetown.edu/ within author discrimination experiments. Literary and handle/10822/557726 Linguistic Computing, 22(2), 207-224.
Witten, I. H., Frank, E. & Hall, M. A. (2011). Data mining: practical machine learning tools and techiniques (3rd ed.). Burlington: Morgan Kaufmann.
Zheng, R., Li, J., Chen, H. & Huang, Z. (2006). A framework for authorship identification of online messages: writingstyle features and classification techniques. Journal of the American Society for Information Science and Technology, 57(3): 378-393.
Zvetco Biometrics. (2012). Biometric Knowledge Center. Retrieved from http://www.zvetcobiometrics.com/Support/ definitions.jsp