Language AI Playbook
  • 1. Introduction
    • 1.1 How to use the partner playbook
    • 1.2 Chapter overviews
    • 1.3 Acknowledgements
  • 2. Overview of Language Technology
    • 2.1 Definition and uses of language technology
    • 2.2 How language technology helps with communication
    • 2.3 Areas where language technology can be used
    • 2.4 Key terminology and concepts
  • 3. Partner Opportunities
    • 3.1 Enabling Organizations with Language Technology
    • 3.2 Bridging the Technical Gap
    • 3.3 Dealing with language technology providers
  • 4. Identifying Impactful Use Cases
    • 4.1 Setting criteria to help choose the use case
    • 4.2 Conducting A Needs Assessment
    • 4.3 Evaluating What Can Be Done and What Works
  • 5 Communication and working together
    • 5.1 Communicating with Communities
    • 5.2 Communicating and working well with partners
  • 6. Language Technology Implementation
    • 6.1 Navigating the Language Technology Landscape
    • 6.2 Creating a Language-Specific Peculiarities (LSP) Document
    • 6.3 Open source data and models
    • 6.4 Assessing data and model maturity
      • 6.4.1 Assessing NLP Data Maturity
      • 6.4.2 Assessing NLP Model Maturity:
    • 6.5 Key Metrics for Evaluating Language Solutions
  • 7 Development and Deployment Guidelines
    • 7.1 Serving models through an API
    • 7.2 Machine translation
      • 7.2.1 Building your own MT models
      • 7.2.2 Deploying your own scalable Machine Translation API
      • 7.2.3 Evaluation and continuous improvement of machine translation
    • 7.3 Chatbots
      • 7.3.1 Overview of chatbot technologies and RASA framework
      • 7.3.2 Building data for a climate change resilience chatbot
      • 7.3.3 How to obtain multilinguality
      • 7.3.4 Components of a chatbot in deployment
      • 7.3.5 Deploying a RASA chatbot
      • 7.3.6 Channel integrations
        • 7.3.6.1 Facebook Messenger
        • 7.3.6.2 WhatsApp
        • 7.3.6.3 Telegram
      • 7.3.7 How to create effective NLU training data
      • 7.3.8 Evaluation and continuous improvement of chatbots
  • 8 Sources and further bibliography
Powered by GitBook
On this page

8 Sources and further bibliography

Previous7.3.8 Evaluation and continuous improvement of chatbots

Last updated 1 year ago

Uszkoreit, H. (1997). “”. Deutsches Forschungszentrum für Künstliche Intelligenz.

Ideo. (2015). “”. Journal design kit the human-centered design toolkit.

Mark, D. (Aug 13, 2020). “”. Mark Douthwaite.

Mark, D. (Jul 22, 2020). “”. Mark Douthwaite.

Jan, K. (2023). Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ) GmbH.

Kalika Bali, Monojit Choudhury, Sunayana Sitaram, Vivek Seshadri. (December 2019) “ UNESCO International Conference on Language Technologies for all (LT4All)

David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, et al.. 2022.. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3053–3070, Seattle, United States. Association for Computational Linguistics.

Adelani, D.I., Masiak, M., Azime, I.A., Alabi, J.O., Tonja, A.L., Mwase, C., Ogundepo, O., Dossou, B.F., Oladipo, A., Nixdorf, D., Emezue, C.C., Al-Azzawi, S.S., Sibanda, B.K., David, D., Ndolela, L., Mukiibi, J., Ajayi, T.O., Ngoli, T.M., Odhiambo, B., Owodunni, A.T., Obiefuna, N.C., Muhammad, S.H., Abdullahi, S.S., Yigezu, M.G., Gwadabe, T.R., Abdulmumin, I., Bame, M.T., Awoyomi, O.O., Shode, I., Adelani, T.A., Kailani, H.A., Omotayo, A., Adeeko, A., Abeeb, A., Aremu, A., Samuel, O., Siro, C., Kimotho, W., Ogbu, O.R., Mbonu, C.E., Chukwuneke, C.I., Fanijo, S., Ojo, J., Awosan, O.F., Guge, T.K., Sari, S.T., Nyatsine, P., Sidume, F., Yousuf, O., Oduwole, M., Kimanuka, U., Tshinu, K.P., Diko, T., Nxakama, S., Johar, A.T., Gebre, S., Mohamed, M.A., Mohamed, S.A., Hassan, F.M., Mehamed, M.A., Ngabire, E., & Stenetorp, P. (2023). “ ArXiv, abs/2304.09972.

“”. (2022) Appen and CLEAR Global.

Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, et al.. 2022.”. Transactions of the Association for Computational Linguistics, 10:50–72.

Santini, M., Strandqvist, W., Nyström, M., Alirezai, M., Jönsson, A. (2018). “”. In: Elloumi, M., et al. Database and Expert Systems Applications. DEXA 2018. Communications in Computer and Information Science, vol 903. Springer, Cham.

Öktem A. (2022) “”.

Öktem, A., DeLuca, E., Bashizi, R., Paquin, E., & Tang, G. (2021). . ArXiv, abs/2103.10734.

Anastasopoulos, A., Cattelan, A., Dou, Z., Federico, M., Federman, C., Genzel, D., Guzm'an, F., Hu, J., Hughes, M., Koehn, P., Lazar, R., Lewis, W., Neubig, G., Niu, M., Oktem, A., Paquin, E., Tang, G., & Tur, S. (2020). . ArXiv, abs/2007.01788.

Nekoto, W., Marivate, V., Matsila, T., Fasubaa, T.E., Kolawole, T., Fagbohungbe, T.H., Akinola, S.O., Muhammad, S.H., KABENAMUALU, S.K., Osei, S., Freshia, S., Niyongabo Rubungo, A., Macharm, R., Ogayo, P., Ahia, O., Meressa, M., Adeyemi, M., Mokgesi-Selinga, M., Okegbemi, L., Martinus, L., Tajudeen, K., Degila, K., Ogueji, K., Siminyu, K., Kreutzer, J., Webster, J., Ali, J.T., Abbott, J.Z., Orife, I., Ezeani, I.U., Dangana, I.A., Kamper, H., ElSahar, H., Duru, G., Kioko, G., Murhabazi, E., Biljon, E.V., Whitenack, D., Onyefuluchi, C., Emezue, C.C., Dossou, B.F., Sibanda, B.K., Bassey, B.I., Olabiyi, A., Ramkilowan, A., Oktem, A., Akinfaderin, A., & Bashir, A.M. (2020). ArXiv, abs/2010.02353

Alp Öktem, Muhannad Albayk Jaam, Eric DeLuca, Grace Tang. (2020) “ In: 2020 IEEE Global Humanitarian Technology Conference (GHTC) 2020 October 29 - November 1; Virtual.

∀, Iroro Orife, Julia Kreutzer, Blessing Sibanda, Daniel Whitenack, Kathleen Siminyu, Laura Martinus, Jamiil Toure Ali, Jade Abbott, Vukosi Marivate, Salomon Kabongo, Musie Meressa, Espoir Murhabazi, Orevaoghene Ahia, Elan van Biljon, Arshath Ramkilowan, Adewale Akinfaderin, Alp Öktem, Wole Akin, Ghollah Kioko, Kevin Degila, Herman Kamper, Bonaventure Dossou, Chris Emezue, Kelechi Ogueji, Abdallah Bashir. “” In: AfricaNLP workshop @ Eighth International Conference on Learning Representations (ICLR 2020) 2020 April 26; Addis Ababa, Ethiopia (Online).

Alp Öktem, Mirko Plitt, Grace Tang. “”. In: AfricaNLP workshop @ Eighth International Conference on Learning Representations (ICLR 2020) 2020 April 26; Addis Ababa, Ethiopia (Online).

Aimee Ansari (2023) “”. CLEAR Global Blog

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022.. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popovic. 2015. . In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.

Way, A. (2018). In: Moorkens, J., Castilho, S., Gaspari, F., Doherty, S. (eds) Translation Quality Assessment. Machine Translation: Technologies and Applications, vol 1. Springer, Cham.

Müller, Mathias. 2021, University of Zurich, Faculty of Arts.

Bahdanau, D., Cho, K., & Bengio, Y. (2014). . CoRR, abs/1409.0473.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. In Advances in Neural Information Processing Systems 30, pages 5998–6008.

Language Technology A First Overview
The field guide to Human-Centered Design
Serverless ML: Deploying Lightweight Models at Scale
A Brief Introduction to Serverless Computing
Creating community-driven datasets: Insights from Mozilla Common Voice activities in East Africa.
ELLORA: Enabling Low Resource Languages with Technology”.
A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation
MasakhaNEWS: News Topic Classification for African languages”.
Language Specific Peculiarities Document for Sheng as Spoken in Kenya
“Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Can We Quantify Domainhood? Exploring Measures to Assess Domain-Specificity in Web Corpora
Toolkit for marginalized and under-resourced languages
Congolese Swahili Machine Translation for Humanitarian Response
TICO-19: the Translation Initiative for Covid-19
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages.
Gamayun – Language Technology for Humanitarian Response”
IEEEXplore link
Masakhane -- Machine Translation For Africa
Tigrinya Neural Machine Translation with Transfer Learning for Humanitarian Response
Hello, Porcupine! Using AI to support farmers to adapt to climate change
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust
Bleu: a method for automatic evaluation of machine translation.
chrF: character n-gram F-score for automatic MT evaluation
Quality Expectations of Machine Translation.
https://doi.org/10.1007/978-3-319-91241-7_8
Robust Neural Machine Translation Systems.
Neural Machine Translation by Jointly Learning to Align and Translate
Attention is All you Need.
Page cover image