User-Centred Design Actions for Lightweight Evaluation of an Interactive Machine Learning Toolkit

  • Francisco Bernardo Department of Computing Goldsmiths, University of London London SE14 6NW
  • Mick Grierson Department of Computing Goldsmiths, University of London London SE14 6NW
  • Rebecca Fiebrink Department of Computing Goldsmiths, University of London London SE14 6NW
Keywords: User-centred Design, Interactive Machine Learning, Application Programming Interfaces, Toolkits, Creative Technology,


Machine learning offers great potential to developers and end users in the creative industries. For example, it can support new sensor-based interactions, procedural content generation and end-user product customisation. However, designing machine learning toolkits for adoption by creative developers is still a nascent effort. This work focuses on the application of user-centred design with creative end-user developers for informing the design of an interactive machine learning toolkit. We introduce a framework for user-centred design actions that we developed within the context of an European Union innovation project, RAPID-MIX. We illustrate the application of the framework with two actions for lightweight formative evaluation of our toolkit—the JUCE Machine Learning Hackathon and the RAPID-MIX API workshop at eNTERFACE’17. We describe how we used these actions to uncover conceptual and technical limitations. We also discuss how these actions provided us with a better understanding of users, helped us to refine the scope of the design space, and informed improvements to the toolkit. We conclude with a reflection about the knowledge we obtained from applying user-centred design to creative technology, in the context of an innovation project in the creative industries.  

Author Biographies

Francisco Bernardo, Department of Computing Goldsmiths, University of London London SE14 6NW
Department of Computing, Researcher and PhD Candidate
Mick Grierson, Department of Computing Goldsmiths, University of London London SE14 6NW
Department of Computing, Professor
Rebecca Fiebrink, Department of Computing Goldsmiths, University of London London SE14 6NW
Department of Computing, Senior Lecturer


Abras, C., Maloney-Krichmar, D., & Preece, J. (2004). User-Centered Design. In W. Bainbridge (Ed.), Encyclopedia of Human-Computer Interaction (Vol. 37, pp. 445–56). Thousand Oaks: Sage Publications.

Amershi, S., Chickering, M., Drucker, S. M., Lee, B., Simard, P., & Suh, J. (2015). ModelTracker: Redesigning Performance Analysis Tools for Machine Learning. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15, 337–346.

Bernardo, F., Tanaka, A., Fiebrink, R., Parkinson, A., Meala, S., & Bevilacqua, F. (2015). D2.1 User-Centred Design Methodology. Retrieved from

Bernardo, F., Zbyszyński, M., Fiebrink, R., & Grierson, M. (2017). Interactive Machine Learning for End-User Innovation. In Proceedings of the Association for Advancement of ArtificiaI Intelligence Symposium Series: Designing the User Experience of Machine Learning Systems (pp. 369–375).

Bevilacqua, F., Bernardo, F., Mealla, S., & Fiebrink, R. (2015). D2.2 Design Guidelines for Prototyping. Retrieved from

Blackwell, A. F. (2017). End-User Developers – What Are They Like? In F. Paternò & V. Wulf (Eds.), New Perspectives in End-User Development (pp. 121–135). Cham: Springer International Publishing.

Caramiaux, B., Montecchio, N., Tanaka, A., & Bevilacqua, F. (2014). Adaptive Gesture Recognition with Variation Estimation for Interactive Systems. ACM Transactions on Interactive Intelligent Systems, 4(4), 1–34.

Cherry, E., & Latulipe, C. (2014). Quantifying the Creativity Support of Digital Tools through the Creativity Support Index. ACM Transactions on Computer-Human Interaction, 21(4), 1–25.

Chowning, J. M. (1973). The Synthesis of Complex Audio Spectra by Means of Frequency Modulation. Journal of the Audio Engineering Society, 21, 1–10.

Clarke, S. (2011). How Usable Are Your APIs? In A. Oram & G. Wilson (Eds.), Making Software: What Really Works, and Why We Believe It (1 edition, pp. 545–565). O'Reilly Media.

Cooper, A., Reimann, R., & Cronin, D. (2007). About Face 3: The Essentials of Interaction Design. Indianapolis, Indiana: Wiley.

Edwards, W. K., Bellotti, V., Dey, A. K., & Newman, M. W. (2003). Stuck in the Middle : The Challenges of User-Centered Design and Evaluation for Infrastructure. Chi 2003, 297–304.

Fails, J. A., & Olsen, D. R. (2003). Interactive Machine Learning. In Proceedings of the 8th international conference on Intelligent user interfaces IUI 03 (pp. 39–45).

Fiebrink, R., Cook, P. R., & Trueman, D. (2011). Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the 2011 annual conference on Human factors in computing systems - CHI '11 (p. 147).

Françoise, J., Schnell, N., & Bevilacqua, F. (2013). A multimodal probabilistic model for gesture-based control of sound synthesis. Proceedings of the 21st ACM International Conference on Multimedia - MM '13, 705–708.

Hall, M. A., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The WEKA Data Mining Software: An Update. SIGKDD Explorations, 11(1), 10–18.

Hartmann, B., Abdulla, L., Mittal, M., & Klemmer, S. R. (2007). Authoring Sensor-based Interactions by Demonstration with Direct Manipulation and Pattern Recognition. In Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '07 (p. 145).

Holtzblatt, K., Wendell, J. B., & Wood, S. (2004). Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design. San Francisco, CA: Morgan Kaufmann Publishers Inc.

Kapoor, A., Lee, B., Tan, D., & Horvitz, E. (2010). Interactive Optimization for Steering Machine Classification. Proceedings of the 28th international conference on Human factors in computing systems - CHI '10. New York, New York, USA, NY, USA: ACM Press.

Katan, S., Grierson, M., & Fiebrink, R. (2015). Using Interactive Machine Learning to Support Interface Development Through Workshops with Disabled People. In CHI '15: Extended Abstracts on Human Factors in Computing Systems 2015.

Lieberman, H., Paternó, F., Klann, M., Paternò, F., & Wulf, V. (2006). End-user development: An emerging paradigm. End User Development, 9, 1–8.

McPherson, A. (2017). Bela: An embedded platform for low-latency feedback control of sound. The Journal of the Acoustical Society of America.

Monk, A. (2007). Lightweight Techniques to Encourage Innovative User Interface Design. In L. E. Wood (Ed.), User Interface Design: Bridging the Gap from User Requirements to Design. CRC Press LLC.

Myers, B. A., & Stylos, J. (2016). Improving API usability. Communications of the ACM, 59(6), 62–69.

Nielsen, J. (1994). Usability Engineering. San Francisco, CA: Morgan Kaufmann Publishers Inc.

Norman, D. A. (2013). The Design of Everyday Things. MIT Press.

Norman, D., & Draper, S. W. (1986). User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, NJ, USA: L. Erlbaum Associates Inc.

Parkinson, A., Zbyszyński, M., & Bernardo, F. (2017). Demonstrating Interactive Machine Learning Tools for Rapid Prototyping of Gestural Instruments in the Browser. In Web Audio Conference 2017 (pp. 1–2). Retrieved from

Patel, K., Bancroft, N., Drucker, S. M., Fogarty, J., Ko, A. J., & Landay, J. (2010). Gestalt: Integrated Support for Implementation and Analysis in Machine Learning. In Proceedings of the 23nd annual ACM symposium on User interface software and technology (pp. 37–46).

Ritter, F. E., Baxter, G. D., & Churchill, E. F. (2014). Foundations for Designing User-Centered Systems: What System Designers Need to Know about People. London: Springer London Heidelberg New York Dordrecht.

Steiglitz, K. (1996). A Digital Signal Processing Primer: With Applications to Digital Audio and Computer Music. Addison-Wesley.

von Hippel, E. (1986). Lead Users: An Important Source of Novel Product Concepts. Management Science, 32(7), 791–805.

Zbyszyński, M., Grierson, M., & Yee-king, M. (2017). Rapid Prototyping of New Instruments with CodeCircle. In NIME 2017 Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 227–230).

Zbyszyński, M., Grierson, M., Yee-king, M., & Fedden, L. (2017). Write once run anywhere revisited: machine learning and audio tools in the browser with C++ and emscripten. In Web Audio Conference 2017 (pp. 1–5). Centre for Digital Music, Queen Mary University of London.