On the Use of Natural User Interfaces in Physical Rehabilitation: A Web-based Application for Patients with Hip Prosthesis

This study aims to develop a telemedicine platform for self-motor rehabilitation and remote monitoring by health professionals, in order to enhance recovery in patients after hip replacement. The implementation of such a technology is justified by medical (improvement of the recovery process by the possibility to perform rehabilitation exercises more frequently), economic (reduction of the number of medical appointments and the time patients spend at the hospital), mobility (diminution of the transportation to and from the hospital) and ethics (healthcare democratization and increased empowerment of the patient) purposes. The Kinect camera is used as a Natural User Interface to capture the physical exercises performed at home by the patients. The quality of the movement is evaluated in real-time by an assessment module implemented according to a Hidden-Markov Model approach. The results show a high accuracy in the evaluation of the movements (92% of correct classification). Finally, the usability of the platform is tested through the System Usability Scale (SUS). The overall SUS score is 81 out of 100, which suggests a good usability of the Web application. Further work will focus on the development of additional functionalities and an evaluation of the impact of the platform on the recovery process


| INTRODUCTION
During the last decades, tele-rehabilitation has been rapidly growing (Holden et al., 2007;Antón et al., 2015;Rybarczyk et al., 2018).This trend is supported by the boom of the high velocity internet and a will to reduce the costs of the medical care.The concept consists of enabling patients to complete part of a rehabilitation program at home, using a Web application that permits them to share with health professionals the evolution of the recovery process (Rybarczyk & Vernay, 2016).The current applications are mainly developed for the rehabilitation of cognitive disabilities (Rybarczyk & Gonçalves, 2016).Due to a complex and relatively expensive implementation, the technological tools for physical therapy are rarer.Nevertheless, the recent emergence of Natural User Interfaces (NUI), such as the Kinect, allows for an affordable system to capture the human movement.Once the threedimensional coordinates of the body joints are automatically extracted by the device, motion data can be processed for applications in different fields, such as gaming (Gameiro et al., 2014), ergonomics (Plantard et al., 2016), psychology (Coelho et al., 2014), education (Carrasco et al., 2013), health (Brook et al., 2014), among other.This paper presents a Web-based platform for motor tele-rehabilitation applied to patients after hip arthroplasty surgery.This orthopaedic procedure is an excellent case study, because it involves people who need a postoperative functional rehabilitation program to recover strength and joint mobility.The development of a tele-rehabilitation system is justified by the condition of these individuals that makes difficult their transportation to and from the physiotherapist's office.The proposed approach considers two fundamental conditions for the development of a suitable tele-rehabilitation platform.First, the motion capture must be based on a low-cost device, in order to be economically viable.Second, the platform should automatically detect the correctness of the executed movement to provide the patient with real-time feedback (Kleine Deters & Rybarczyk, 2018).
The development of such a tele-rehabilitation platform involves four steps, which are described in the rest of the manuscript.Section 2 summarizes the main stages of the therapeutic program.Section 3 is a presentation of the Web application that includes backend, frontend, user interfaces and database.Section 4 focuses on the assessment module, which automatically classifies the quality of the movement through a Hidden Markov Model approach.Section 5 consists of a description of an experiment to test the usability of the Website.Finally, the results of the assessment performance and platform usability are discussed in the conclusion of the paper.

| REHABILITATION PROGRAM
A systematic literature review performed in MEDLINE, PEDro and ScienceDirect databases shows that the rehabilitation program should be divided into two stages.Each stage is defined by different therapeutic objectives and durations (Peter et al., 2014).Then, in stage 2, the transition from early to late Post-Acute Rehabilitation (PAR) depends on the clinical progress of each patient.
The first stage is characterized by three aspects.The first one is an education of the patients, in order to prepare them to manage their rehabilitation at home (Jäppinen et al., 2016).The second one consists of completing a cryotherapy that aims to reduce pain and contain blood lost (Ni et al., 2015).Finally, an active workout involves a repetition in supine position of the exercises as follows: respiration, core stability training, and lower limb movement (Peter et al., 2014).
In early PAR of stage 2, the rehabilitation program sets exercises of active range of movements (ROM) of the lower limb and neuromuscular function training with the same frequency and duration as in the stage 1 (Heiberg et al., 2016).Late PAR includes the weight bearing training to improve functional performance and increase muscle strength (Tsukagoshi et al., 2014), along with a progressive return to walk (Schega et al., 2014), and static/dynamic balance training (Jogi et al., 2016).

ARCHITECTURE
The rehabilitation platform is based on a clientserver architecture (Figure 1).The server side is divided into two parts, which are the application server and the database server.The functionalities of the application server depend on the type of user.The patients can access to the therapeutic exercises, which are presented as a game.A feedback of their performance is automatically displayed thanks to the movement assessment module.The health professionals (physiotherapist and physician) can carry out a remote monitoring of the patient's performance and update the rehabilitation program.The database stores an heterogenous source of information such as: demo videos of the exercises; joint coordinates of the performed movements; questions/responses to medical questionnaires; and exchange of messages between the stakeholders.

BACKEND AND FRONTEND
The Website is based on the Django technology, a Web framework written in Python.For an effective development, Django CMS, the content management system version of Django, is used in order to simplify the implementation of the Web application.Node.js, a Javascript run-time environment, is used as a gateway between the Kinect and the Django Website to manage the motion capture data flow.It allows the creation of an application coded in JavaScript and HTML and a communication with the server.The backend manages the data from the Kinect and generate an HTML page for the browser.Node.js is chosen for its wide range of modules and, more specifically, the library Kinect2 which simplifies the communication with the Kinect.The library http is used to start the application.Figure 2 shows an example of the created JavaScript application to display the avatar of the patient in a browser.
The client-server connection is established through a WebSocket communication protocol.Node.js includes a library named Socket.IO and allows the deployment of a real-time bidirectional event-based communication.To do so, the destination address of the data (the address of the server hosting the Website) must be declared in the Node.jsapplication, and the IP address of the sender must be declared in the Django Website.Through this method, the library Socket.IO for Python and Node.js are able to communicate by using a function call.

USER INTERFACES
The Website can accommodate different types of users, such as patients and therapists.The functionalities are singular for these two kinds of users.Figure 3 shows the workflow of the patient account.Three main activities are available: (i) practice exercises; (ii) consult performances; and (iii) send/read a message to/from the health professionals.Before the rehabilitation begins, patients must answer a questionnaire, which evaluates their ability to complete the exercises.They must self-assess pain levels, skin problems, and potential edemas.If the questionnaire outputs a low score, the patients are impeded to perform the physical exercises.In the opposite case, the patients can proceed with the rehabilitation protocol.
They must choose an available stage (see section 2) and the various associated exercises.Once the exercise selected, a demonstration video of the movement to perform is displayed on the screen (Figure 4).The movement recoding starts when the hand of the user's avatar is inside the red square of the patient's visual interface (see Figure 2).The 3D joint coordinates received from the Kinect and the assessment associated are saved into the database.At the end of the exercise, the patients can review their movements and have a detailed feedback on their performance.The same questionnaire as previously mentioned is asked to the user at the end of a practical session, in order to provide the therapist with information on the state of the patient after performing the physical exercises.
Several functionalities are available for the health professionals.First, they have to create an account if they are not yet registered.After that, as shown in Figure 5, it is possible to: (i) add a new patient and/or update information on existing patients; (ii) monitor the performance, progress and movements of the patients; and (iii) send and/or receive messages from patients.Therapists have also the possibility to update their personal data.

DATABASE
The database is directly associated with the Django Website.Django CMS integrates a relational database management system named SQLite3, which is adequate for the tele-rehabilitation platform.The database is divided into three parts according to the kind of stored information: (i) personal data; (ii) questionnaire data; and (iii) exercise data.The personal table stores different information about the users, such as: name, email, age, occupation, address… The questionnaire table stores all the qualitative responses provided by the patients and enables the health professionals to follow the evolution of the patient's medical condition.CITARJ

2-19
The physical exercise data modeling is more complex.It involves a relationship between five tables.The Main An automatic assessment of the quality of the performed exercises is implemented according to a Hidden Markov Models (HMM) approach (Papadopoulos et al., 2014).This technic provides insights in the ontological structure of the rehabilitation exercises and represents a probabilistic interpretation of the correctness of a movement execution (Mahapatra et al., 2014).Here, the ontology of different compensatory movements associated with an exercise composed with a sequence of forward-sideway-backward movement of the right leg are evaluated from an experiment on healthy participants.

HIDDEN MARKOV MODELS
HMM is a probabilistic approach that aims to model a given action into hidden states.These states represent an arbitrary decomposition of the whole movement into successive phases.In the present application, the objective is to classify the performed exercise into six possible executions (classified according to their correctness).A different model is trained for each of these categories, such as: • HMM II -steps too short • HMM III -no movement of the center of mass • HMM IV -steps too large • HMM V -steps with bended knee • HMM VI -steps with bended knee and flexed torso A total of twelve states are used.The six models have a fully connected structure, which means that the transition from any state to another is possible with a certain probability (Figure 6).
When calculating the probabilities of a sequence of observations and comparing the probabilities of all the HMMs, the sequence is classified (Class) as the category that provides the highest probability, as described in Equation 1(where λ i represents a determined model and O is a sequence of observations): The three-dimensional coordinates of the body joints, which are extracted by the Kinect camera, correspond to the features used as observations.The angle of rotation of the joints are calculated from the Cartesian coordinates.In this experiment, four joints only are used: ankles, knees, hips, and spine.Figure 7 shows a graphical representation of the used features in relation to the skeleton image provided by the camera.

EXPERIMENTAL PROTOCOL
Four subjects participated in the experiment.They were asked to take place at approximately two meters' distance from a Kinect camera.The motion capture device was placed at the height of the subject's xiphoid apophysis (center of the torso).Each participant executed 70 movements leading to a total of 280 records.The rehabilitation exercise was a sequence, in which the subjects had to do one step forward, one step sideways and one step backward, with variations.These variations are staged executions of errors or compensatory movements that can occur during the rehabilitation in practice.The exercises were performed in batches of ten in the following order: (I) correct execution, (II) steps too short, (III) execution without moving the center of mass, (IV) steps too large, (V) steps with bended knee, (VI) steps with bended knee and flexed torso.

RESULTS
The classification performance shows a high level of accuracy (Table 1) in classifying a whole sequence into the classes (I-VI).A value of 1 means the model always gave the highest probability, with respect to the other models and for any sequence of the related movement, whereas a value of 6 indicates the lowest probability.The values in this table are averaged prediction ranks for each model of each movement (I-VI).The average prediction rank of HMM I is the highest (2.78), which means that the execution type I (correct movement) is most closely related to all the other types.
The overall performance of the classification for each class (I-VI) is shown in Table 2.It is to note that the execution type III (i) is more likely to be   classified as type I, and (ii) has the lowest prediction accuracy compared with the other classes.This could be caused by the difficulty in staging this type of execution or a lack of descriptive power in the gesture representation.

EXPERIMENTAL PROTOCOL
In order to evaluate if the platform can be easily used by the end users, a usability test was carried out.subjects (11 males and 11 females) between 20 and 50 years old participated in the experiment.Participants were informed about the purpose of the study and they gave us their consent to participate.The experiment consisted of asking subjects to perform the nine tasks as follows: • Create a new physiotherapist's account • Finally, enter in the "Stage 2" and launch a movement recording by using the Kinect Once these tasks performed, the participants had to fill in a questionnaire based on the System Usability Scale (SUS).SUS was chosen as usability test, because it enables us to get a reliable subjective evaluation by the users in few questions (high correlation with other tests of subjective measures) (Brooke, 1996;Sauro, 2011).SUS is a questionnaire based on a Likert scale, in which a statement is made and the respondent then indicates the degree of agreement or disagreement with the statement on a 5-point scale, from "strongly disagree" (0) to "strongly agree" (4).The statements, adapted for the purpose of this specific study, were as follows: 1.As a patient or physiotherapist, I think I would like to use this platform frequently 2. I found the platform unnecessarily complex 3.I thought the platform was easy to use 4. I think I would need the support of a technical person to be able to use this platform 5.I found that the various functionalities of the platform were well organized 6.I thought there was too much inconsistency in this platform 7. I would imagine that most people would learn to use this platform very quickly 8.I found the platform very cumbersome to use 9.I felt very confident using the platform 10.I needed to learn a lot of things before I could get going with this platform The odd statements are positive observations and the even are negative.To calculate the SUS score, first sum the score contributions from each item.Each item's score contribution will range from 0 to 4. For items 1, 3, 5, 7 and 9, the score contribution is the scale position minus 1.For items 2, 4, 6, 8 and 10, the contribution is 5 minus the scale position.Multiply the sum of the scores by 2.5 to obtain the overall value of system usability.SUS score has a range of 0 to 100.

RESULTS
The value of the system usability is obtained by averaging the SUS scores of all the participants.The average value is 81 out of 100, which can be considered as a good evaluation of the usability of the platform.Figure 8 represents the average assessment for each statement (range from 0 to 4).The variance of the SUS scores of all the statements overlap on the y-axis.Although it could suggest a similarity in the assessment made by the participants regarding the ten statements, the Kruskal-Wallis statistical test shows a significant difference between the score of all the statements (p<0.05).It is due to the difference of variances between the even and odd items, which are larger for the former than the latter (see Figure 8).This can be explained by the fact that it is a little bit more confusing for the subjects to gauge a negative statement than a positive one.
The main suggestions made by the subjects, in order to improve the usability of the user interface, are:

•
Increasing the size of some buttons

•
Providing some guidance on the way to use the Kinect

| CONCLUSIONS
This paper presents a tele-rehabilitation platform for the self-reeducation of patients after hip surgery replacement.The system is composed with a user interface for the patients performing the therapeutic physical exercises and another interface to enable the health professionals (physiotherapist and physician) to remotely monitor the recovery process and update the rehabilitation program.
The assessment module based on an HMM approach seems to capture to some extend the quality of the performed movement (overall correct classification of 92%).In order to increase the classification performance, it would be necessary to: (i) include an optimization of the class separability by extending descriptive power through additional features (e.g., ankle/torso displacement and normalized speed/acceleration paths of these different joints); (ii) design an optimal capture rate; and (iii) create a preliminary detection method for the recognition of noise.
Finally, the current version of the platform seems to present a quite high usability, since it got a SUS score of 81 out of 100.Nevertheless, some aspects of the design of the user interface must be improved such as the size of certain icons and the interaction with the Web application through the Kinect.Next steps will consist of: (i) adding further functionalities; (ii) performing advanced recording with the Kinect; and (iii) assessing the effect of the use of the platform on the quality and velocity of the recovery process.Gameiro, J., Cardoso, T., & Rybarczyk, Y. (2014).Kinect-Sign: teaching sign language to listeners through a game.Innovative and Creative Development in Multimodal Interaction Systems. Heidelberg: Springer, pp. 141-159. Heiberg, K., & Figved, W. (2016).Physical functioning and prediction of physical activity after

Figure 2 |
Figure 2 | Patient's avatar displayed in the Web browser.

Figure 1 |
Figure 1 | Architecture of the tele-rehabilitation platform.

Figure 4 |
Figure 4 | Screenshot of an exercise demonstration video.

Figure 6 |
Figure 6 | Structure of the used HMMs.

Figure 8 |
Figure 8 | Average SUS scores for the 10 statements.
Table synthetizes the data about the exercises completed by the patient: date; kind of exercise; rehabilitation stage; completion time; and outcome.During the execution of an exercise, fractions of the entire movement are assessed in real-time, in order to evaluate the correctness of the performed movement.Then, a single outcome (perfect, good, or bad) is produced based on the sum of these individual assessments (see next section for more details on the assessment module).The Demo Table is linked to the Main Table through a 1:N relationship, and stores details about each exercise and a demonstration video of the movement that must be performed.The Coordinate Tables are also linked to the Main Table by a 1:N relationship, and store an information regarding the movement performed during an exercise such as the 3D joint coordinates captured by the Kinect.These data can be used to display a replay of the movement carried out by the patients, in order to provide the physiotherapist with a more ecologic visualization of the actual patient's performance.The Assessment Table is linked to the respective Coordinate Table, and stores the evaluation calculated by the assessment module for each part of the movement (see Figure 3, Markov Model box).
Finally, the Set Table is linked to the Main Table, by a 1:N relationship, and identifies the nature of the exercise (balance, coordination, ROM and force).

Table 1 |
Confusion matrix of the executed movements (I-VI).

Table 2 |
Performance of the classification of movements I-VI.