Variability of Likert scales Effects on data collection results Class 7 – 2011/2012 MIMED - FMUP

Questionnaires Likert scale: what is it? Different Likert scales Study objective Methods - Study participants - Study design Results Conclusions

Questionnaries Likert Scale and Its Variables - A Comparative Study Currently, people’s opinion about an issue or service is a valuable way for researchers to acquire information in diverse areas of knowledge. Introduction Objective Methods Results Conclusion A questionnaire is an essential instrument that aims to collect this information based on the inquisition of the population in study; It consists in a certain amount of questions covering a topic of interest to investigators; Is useful in almost every kind of job. Questionnaries

Likert scale Likert Scale and Its Variables - A Comparative Study Having in mind the need to measure and classify this information, one reaches the conclusion of the pressing necessity for scales, tools that permits measuring people’s attitudes and opinions[1]. Introduction Objective Methods Results Conclusion Likert scale 1. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500.

Measure pain, quality of life Likert Scale and Its Variables - A Comparative Study What is it? A scale that consists in asking people to respond to a series of statements about a topic, in terms of the extent to which they agree or disagree with them. Introduction Objective Methods Results Conclusion [2] Applications Measure pain, quality of life [3] Likert scale 2. Zou KH, C.M., Quinn SA, Beta-mapping and beta-regression for changes of ordinal-rating measurements on Likert scales: a comparison of the change scores among multiple treatment groups. Stat Med, 2010. 29(24): p. 2486-500. 3.Huber JF, H.J., Zumstein MD, Ruflin G, Lüscher M., Visual circle scale (VCS)-a patient-friendly scale to measure pain compared to VAS and Likert scale. Z Orthop Unfall, 2007. 145(6): p. 795-7.

Different Likert Scales Likert Scale and Its Variables - A Comparative Study Different scales produce different results Then, different Likert scales may produce different results Introduction Objective Methods Results Conclusion Search: some articles were found confirming our idea [4] Different Likert Scales 4 Dawes, J., Do data characteristics change according to the number of scale points used? An experiment using 5-point, 7-point and 10-point scales. International Journal of Market Research, 2008. 50(1): p. 61-77.

Influence of a central answer Likert Scale and Its Variables - A Comparative Study Does a neutral answer influence the results? Already  studied [5] Introduction Objective Methods Results Conclusion  Different Likert scales produce different results? Does a "I don't want to answer" option influence the results? Apparently not studied yet. Does both "I don't want to answer" and neutral options influence the results? Apparently not studied yet. Influence of a central answer 5. Kulas JT. , S.A., Middle category endorsement in odd-numbered Likert response scales: Associated item characteristics, cognitive demands, and preferred meanings. Journal of Research in Personality, 2009. 43: p. 489–49

Evaluate the reproducibility of the scales used in the questionnaire. Likert Scale and Its Variables - A Comparative Study Introduction Objective Methods Results Conclusion Discover if the addition of a neutral element, a “refusal to answer” option or both combined has repercussions in the results obtained. Evaluate the reproducibility of the scales used in the questionnaire. [6] Objectives 6. Hansson LM, G.M., Bergström R, Factors affecting reproducibility of dietary reports using food frequency questionnaires. Eur J Clin Nutr, 2000. 54(8).

Likert Scale and Its Variables - A Comparative Study Study Design: One questionnaire with two versions Questionnaire 1 Questionnaire 2 Number of questions in each questionnaire: 12 Same questions in both questionnaires Introduction Objective Methods Results Conclusion Methods

Type of study Our study is included in a Longitudinal Study type. Likert Scale and Its Variables - A Comparative Study Our study is included in a Longitudinal Study type. BECAUSE : There is a follow-up period of the individuals in study, which can be longer or shorter; Data is gathered at least in two points in time which allows us to observe changes in the population. Introduction Objective Methods Results Conclusion Type of study

Students from classes 6 and 7 Likert Scale and Its Variables - A Comparative Study Study Participants: Introduction Objective Methods Results Conclusion All students that belong to the first year of the Integrated Master’s Degree in Medicine of Faculdade de Medicina da Universidade do Porto, distributed by classes 1 to 22 Except Students from classes 6 and 7 Methods

These ten blocks where randomly divided in two groups with SPSS. Likert Scale and Its Variables - A Comparative Study Study Participants: Introduction Objective Methods Results Conclusion Monday Tuesday Wednesday Thursday Friday 8:30-11 h 1 4 6 9 14-16:30h 2 x 7 10 16:30-19h 3 5 8 These ten blocks where randomly divided in two groups with SPSS. Methods Questionnaire 1 Questionnaire 2

About the questionnaires: Likert Scale and Its Variables - A Comparative Study About the questionnaires: Introduction Objective Methods Results Conclusion Theme of the questionnaire: Evaluation systems and study methods Why? Theme with less opinion change probability Methods

About the questionnaires: Likert Scale and Its Variables - A Comparative Study About the questionnaires: Introduction Objective Methods Results Conclusion Half of the participants answer to version 1 and the other half to version 2 One month later, the first half answer to version 2 and the other half to version 1 Why? To avoid certain biases. Methods

About the questionnaires: Likert Scale and Its Variables - A Comparative Study About the questionnaires: Introduction Objective Methods Results Conclusion The questionnaires ought to be confidential. Why? How? In order to receive from the participants their truly opinion The participants wrote their names on the questionnaires and we ensured confidentiality . Methods

About the questionnaires: Likert Scale and Its Variables - A Comparative Study About the questionnaires: Introduction Objective Methods Results Conclusion Comparison scale: I totally agree I agree I disagree I totally disagree … plus a neutral answer: Neither agree nor disagree … plus refusal to answer: I don’t want to answer Methods

About the questionnaires: Three types of comparison Likert Scale and Its Variables - A Comparative Study About the questionnaires: Introduction Objective Methods Results Conclusion Three types of comparison Comparison scale Vs Comparison scale plus refusal to answer option Comparison scale Vs Comparison scale plus neutral and refusal to answer option Comparison scale Vs Comparison scale plus neutral answer Comparison scale: Likert scale with “I totally agree”, “I agree”, “I disagree” and “I totally disagree” Methods

About the questionaires: To evaluate the reproducibility Likert Scale and Its Variables - A Comparative Study About the questionaires: Introduction Objective Methods Results Conclusion To evaluate the reproducibility Comparison scale plus refusal to answer option Vs Comparison scale Vs Comparison scale plus neutral and refusal to answer option Vs Comparison scale: Likert scale with “I totally agree”, “I agree”, “I disagree” and “I totally disagree” Methods

Questionnaire 1 Questionnaire 2 I 1. Comparison Scale 3. Comparison Scale + neutral answer 4. Comparison Scale 5. Comparison Scale + refusal to answer 6. Comparison Scale + neutral answer + refusal to answer 7. Comparison Scale + refusal answer II 1. Comparison Scale + refusal to answer 2. Comparison Scale + neutral answer + refusal to answer 3. Comparison Scale 4. Comparison Scale + neutral answer + refusal to answer 5. Comparison Scale Questionnaire 2 I 1. Comparison Scale + neutral answer 2. Comparison Scale 3. Comparison Scale 4. Comparison Scale + neutral answer 5. Comparison Scale 6. Comparison Scale + neutral answer + refusal to answer 7. Comparison Scale II 1. Comparison Scale + refusal to answer 3. Comparison Scale + refusal to answer 4. Comparison Scale 5. Comparison Scale + neutral answer + refusal to answer

Results 158 Study Participants: 250 234 Likert Scale and Its Variables - A Comparative Study Study Participants: Introduction Objective Methods Results Conclusion 250 234 158 Results

Reproducibility Introduction Objective Methods Results Conclusion   Valid (n) Question Scale Proportion of Agreement [Confidence interval] Kappa [Confidence interval] Weighted Kappa 157 I.2 O sistema de avaliação por exames é justo. Standard 0,76 [0,68;0,84] 0,54 [0,42;0,67] 0,60 [0,49;0,71] 156 II.1 O meu estudo rende mais quando trabalho em grupo. Standard+NR 0,64 [0,56;0,72] 0,47 [0,36;0,58] N.A. I.6 O número de alunos por turma no 1º ano da FMUP é excessivo. Standard+NR+Neutral [0,39;0,55] 0,30 [0,20;0,41] Introduction Objective Methods Results Conclusion

Standard scale vs. Neutral option scale Valid n Question Answers’ Options n(%) Standart Scale Standard Scale + Neutral Option Proportion of Agreement [Confidence Interval] 155 I.4. É justo ser necessário ter frequência a uma disciplina (frequentar pelo menos 2/3) das aulas para ser admitido a exame teórico. Discordo Totalmente 15 (9,7) 12 (7,7) 0.29 [0,12;0,52] Discordo 37 (23,9) 33 (21,3) 0.37 [0,24;0,52] Não Concordo nem Discordo ------- Concordo 79 (51,0) 57 (36,8) 0,37 [0,28;0,48] Concordo Totalmente 24 (15,5) 20 (12,9) 0,26 [0,13;0,44] 156 I.1 O sistema de avaliação contínua é vantajoso para os alunos.   3 (1,9) 2 (1,3) 0 [0;0,54] 25 (16,0) 30 (19,2) 0,41 [0,26; 0,58] 31 (19,9) 104 (66,7) 73 (46,8) 0,57 [0,47;0,66] 24 (15,4) 20 (12,8) 0,38 [0,22;0,56] I.3 Penso que o grau de exigência da FMUP é superior ao de outras. 0 (0) 1 (0,6) 0 [0;0,94] 7 (4,5) 0,43 [0,12;0,80] -------- 46 (29,7) 0,43 [0,32;0,55] 102 (65,8) 82 (52,9) 0,64 [0,55;0,73] Introduction Objective Methods Results Conclusion

Standard scale vs. Refusal TO answer option scale Valid n Question Answers’ Options n(%) Standard Scale Standard Scale + Refusal to Answer Option Proportion of Agreement [Confidence Interval]   158 I.5 Os professores do 1º ano da FMUP estão sempre disponíveis para tirar dúvidas Discordo Totalmente 1 (0,6) 0 (0,0) 0 [0;0] Discordo 28 (18,1) 30 (19,4) 0,35 [0,21;0,51] Concordo 113 (72,9) 87 (56,1) 0,54 [0,45;0,63] Concordo Totalmente 13 (8,4) 29 (18,7) 0,14 [0,05;0,30] Não respondo ------- 9 (5,8) I.7 Quando há avaliação contínua, acho antipedagógico os alunos não irem tendo ao longo do semestre feedback do seu desempenho 3 (1,9) 12 (7,6) 11 (7,0) 0,15 [0,04;0,39] 90 (57,3) 70 (44,6) 0,47 [0,37;0,57] 55 (35,0) 64 (40,8) 0,49 [0,38;0,60] 9 (5,7) II.3 Tenho necessidade de recorrer a bebidas energéticas e/ou vitaminas para acompanhar o ritmo de estudo exigido pelo curso. 70 (44,3) 71 (44,9) 0,52 [0,41;0,62] 57 (37,1) 50 (31,6) 0,35 [0,25;0,47] 25 (15,8) 22 (13,9) 0,31 [0,17;0,48] 6 (3,8) 0,31 [0,10;0,61] 4 (2,5) Introduction Objective Methods Results Conclusion

Standard scale vs. Both options combined scale Valid n Question Answers’ Options N(%) Standard Scale Standard Scale + Both Combined Options Scale Proportion of Agreement [Confidence Interval]   158 II.2 O estudo é mais proveitoso quando trabalho num local silencioso. Discordo Totalmente 0 (0,0) 0,00 [0,00;0,00] Discordo 8 (5,1) 4 (2,5) 0,09 [0,01;0,43] Não Concordo nem Discordo ------- 11 (7,0) Concordo 61 (38,6) 45 (28,5) 0,45 [0,34;0,57] Concordo Totalmente 89 (56,3) 98 (62,0) 0,67 [0,57;0,75] Não Respondo II.4 Frequentemente acho mais rentável estudar pelos apontamentos de colegas mais velhos do que pela bibliografia recomendada. 13 (8,2) 0,31[0,12;0,59] 81 (51,3) 56 (65,4) 0.44 [0,34;0,55] 64 (40,5) 73 (46,2) 0,57 [0,46;0,68] 2 (1,3)  II.5 É difícil conciliar as actividades extra-curriculares (por exemplo praticar um desporto, aprender um instrumento musical, etc…) com o trabalho exigido na FMUP 3 (1,9) 0,10 [0,01;0,46] 29 (18,4) 26 (16,5) 0,25 [0,14;0,41] 22 (13,9) 84 (53,2) 58 (36,7) 0,37 [0,27;0,47] 37 (23,4) 47 (29,7) 0,33 [0,22;0,46] Introduction Objective Methods Results Conclusion

VS Conclusion It depends… Freedom of choice Reproducibility Likert Scale and Its Variables - A Comparative Study Freedom of choice VS Introduction Objective Methods Results Conclusion Reproducibility It depends… Conclusion

Likert Scale and Its Variables - A Comparative Study Abel Trzan Ana Fonseca Ana Filipa Ferreira Ana Isabel Maduro Catarina Tinoco Guilherme Amorim Joana Antunes João Dias José Carneiro Maria João Bastos Maria João Costa Pedro Barros Pedro Santos   Faculdade de Medicina da Universidade Do Porto 2º Semestre Março de 2012