Should different countries participating in PISA interpret socioeconomic background in the same way? A measurement invariance approach
DOI:
https://doi.org/10.35362/rie8413981Keywords:
measurement invariance, multi-group confirmatory factor analysis, cultural reproduction theory, Pierre Bourdieu, socio-economic scales, PISAAbstract
It has been claimed that there is a lack of theory-driven constructs and a lack of cross-country comparability in International Large-Scale Assessment (ILSA)’s socio-economic background scales. To address these issues, a new socio-economic background scale was created based on Pierre Bourdieu’s cultural reproduction theory, which distinguishes economic, cultural and social capital. Secondly, measurement invariance of this construct was tested across countries participating in the Programme for International Student Assessment (PISA). After dividing the countries which participated in PISA 2015 into three groups, i.e., Latin American, European, and Asian, a Multi-Group Confirmatory Factor Analysis was carried out in order to examine the measurement invariance of this new socio-economic scale.
The results of this study revealed that this questionnaire, which measures the socio-economic background, was not found to be utterly invariant in the analysis involving all countries. However, when analysing more homogenous groups, measurement invariance was verified at the metric level, except for the group of Latin American countries. Further, implications for policymakers and recommendations for future studies are discussed.
Downloads
References
Abel, T. (2008). Cultural capital and social inequality in health. Journal of Epidemiology & Community Health, 62(7), e13-e13.
Addey, C. & Sellar, S. (2018). Why Do Countries Participate in PISA? Understanding the Role of International Large-Scale Assessments in Global Education Policy. En Verger, A., Altinyelken, H.K. and Novelli, M. (Eds.), Global education policy and international development: New agendas, issues and policies (pp. 98-117). Bloomsbury Publishing.
Addey, C., Sellar, S., Steiner-Khamsi, G., Lingard, B., & Verger,A. (2017). The rise of International large-scale assessments and rationales for participation. Compare: Ajournal of Comparative and International Education, 47(3), 434-452.
Alivernini, F. (2011). Measurement invariance of a reading literacy scale in the Italian Context: a psychometric analysis. Procedia-Social and Behavioral Sciences, 15, 436-441.
Aloisi, C., & Tymms, P. (2017). PISA trends, social changes, and education reforms. Educational Research and Evaluation, 23(5-6), 180-220.
Barone, C. (2006). Cultural capital, ambition and the explanation of inequalities in learning outcomes: A comparative analysis. Sociology, 40(6), 1039-1058.
Bialosiewicz, S., Murphy, K. & Berry, T. (2013). An introduction to measurement invariance testing: Resource packet for participants. Retrieved from https://bit.ly/3iQoCI5
Bodovski, K., Jeon, H., & Byun, S. Y. (2017). Cultural capital and academic achievement in post-socialist Eastern Europe. British Journal of sociology of Education, 38(6), 887-907.
Bourdieu, P. (1986). The forms of capital. In J. Richardson (Ed.), Handbook of theory and research for the sociology of education. New York: Greenwood.
Bourdieu, P. (1997). The Forms of Capital. In Education: Culture, Economy, and Society, edited by A. H. Halsey, H. Lauder, P. Brown, and A. Stuart Wells, 47-58. Oxford: Oxford University Press.
Bornstein, M. C., & Bradley, R. H. (Eds.). (2003). Socioeconomic status, parenting, and child development. Mahwah, NJ: Lawrence Erlbaum.
Bradley, R. H., & Corwyn, R. F. (2002). Socioeconomic status and child development. Annual Review of Psychology, 53, 371-399.
Brown, T. A. (2015). Confirmatory Factor Analysis for Applied Research. New York: Guildford Press.
Byrne, B. M., & Stewart, S. M. (2006). Teacher’s corner: The MACS approach to testing for multigroup invariance of a second-order structure: A walk through the process. Structural equation modeling, 13(2), 287-321.
Byrne, B. M., & Van de Vijver, F. J. R. (2010). Testing for measurement and structural equivalence in large-scale cross-cultural studies: Addressing the issue of nonequivalence. International Journal of Testing, 10, 107-132.
Caro, D. H., & Cortés, D. (2012). Measuring family socioeconomic status: An illustration using data from PIRLS 2006. IERI Monograph Series Issues and Methodologies in Large-Scale Assessments, 5, 9-33.
Caro, D. H., Sandoval-Hernandez, A., & Lüdtke, O. (2014). Cultural, social, and economic capital constructs in international assessments: An evaluation using exploratory structural equation modelling. School Effectiveness and School Improvement, 25(3), 433-450.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural equation modelling, 9(2), 233-255.
Coe, R., & Fitz-Gibbon, C. T. (1998). School Effectiveness Research: Criticisms and Recommendations. Oxford Review of Education, 24(4), 421–438.
Coleman, J. S., Campbell, E. Q., Hobson, C. J., McPartland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1966). Equality of Educational Opportunity. Washington, DC: US Government Printing Office.
Coleman, J. S. (1988). Social capital in the creation of human capital. The American Journal of Sociology, 94(1) Supplement. Organizations and institutions: Sociological and economic approaches to the analysis of social structure, pp. 95-120.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334.
Demir, E. (2017). Testing Measurement Invariance of the Students’ Affective Characteristics Model across Gender Sub-Groups. Educational Sciences: Theory and Practice, 17(1), 47-62.
DiMaggio, P. (1982). Cultural Capital and School Success: The Impact of Status Culture Participation on the Grades of U.S. High School Students. American Sociological Review, 47 (2), 189-201.
Garcia Aracil, A. D. E. L. A., Neira, I., & Albert, C. (2016). Social and Cultural Capital Predictors of Adolescents’ Financial Literacy: Family and School Influences. Revista de Educación, (374), 91-115.
Gordon W. C. & Roger B., R. (2002). Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance, Structural Equation Modeling, 9(2), 233-255.
Goldstein, H. (2017). Measurement and evaluation issues with PISA. En Louis Volante (Eds.), In The PISA effect on global educational governance. New York and London, Routledge.
Gregorich, S. E. (2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44, 78-94.
Hair, J.F., Black, W.C., Babin, B.J., & Anderson, R.E. (2010). Multivariate Data Analysis. Seventh Edition. New Jersey: Prentice Hall, Upper Saddle River.
Hansson, Å., & Gustafsson, J. E. (2013). Measurement invariance of socioeconomic status across migrational background. Scandinavian Journal of Educational Research, 57(2), 148-166.
He. J., Barrera-Pedemonte, F., & Buchholz, J. (2018). Cross-cultural comparability of noncognitive construction in TIMSS and PISA. Assessment in Education: Principles, Policy & Practice, 1-17.
Hobbs, G., & Vignoles, A. (2007). Is free school meal status a valid proxy for socio-economic status (in schools research)? (No. 84). Centre for the Economics of Education, London School of Economics and Political Science.
Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J. A. (2018). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research, 62(3), 333-353.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: a multidisciplinary journal, 6(1), 1-55.
Kalaycioglu, D. B. (2015). The Influence of Socioeconomic Status, Self-Efficacy, and Anxiety on Mathematics Achievement in England, Greece, Hong Kong, the Netherlands, Turkey, and the USA. Educational Sciences: Theory and Practice, 15(5), 1391-1401.
Klieme, E. (2016). TIMSS 2015 and PISA 2015. How are they related on the country level. German Institute for International Educational Research (DIPF). Retrieved from http://www. dipf. de/de/publikationen/pdf-publikationen/Klieme_TIMSS2015andPISA2015. pdf.
Kline, R.B., (2011). Principles and Practices of Structural Equation Modelling. 3rd ed. New York: The Guilford Press.
Lareau (2011). Unequal childhoods: Class, race, and family life (2nd ed.). Berkeley, CA: University of California Press.
Lauder, H., Jamieson, I., & Wikeley, F. (1998). Models of Effective Schools: Limits and Capabilities. In R. Slee, G. Weiner, & S. Tomlinson (Eds.), School Effectiveness for whom?: Challenges to the School Effectiveness and School Improvement Movements. London: Falmer Press.
Lee, S. S. (2019). Longitudinal and Cross-Country Measurement Invariance of The Pisa Home Possessions Scale. Publicly Accessible Penn Dissertations. 3310. https://repository.upenn.edu/edissertations/3310
Lenkeit, J., Caro, D. H., & Strand, S. (2015). Tackling the remaining attainment gap between students with and without immigrant background: An investigation into the equivalence of SES constructs. Educational Research and Evaluation, 21(1), 60-83.
Lietz, P., Cresswell, J. C., Rust, K. F., & Adams, R. J. (2017). Implementation of Large-Scale Education Assessments. En Petra Lietz, John C. Cresswell, Keith F. Rust and Raymond J. Adams (Eds.) Implementation of Large-Scale Education Assessments. John Wiley & Sons.
Liu, Y., Millsap, R. E., West, S. G., Tein, J. Y., Tanaka, R., & Grimm, K. J. (2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological methods, 22(3), 486.
Marteleto, L., & Andrade, F. (2014). The educational achievement of Brazilian adolescents: Cultural capital and the interaction between families and schools. Sociology of Education, 87(1), 16-35.
Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58(4), 525-543.
Milfont, T.L., & Fischer, R. (2015). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3, 111-130.
Millsap, R. E. & Olivera-Aguilar, M. (2012). Investigating measurement invariance using confirmatory factor analysis. In R. H. Hoyle (Ed.), Handbook of structural equation modelling (pp. 380-392). New York, NY: Guilford Press.
Miranda, D. and Castillo, J. C. (2018). Measurement model and invariance testing of scales measuring egalitarian values in ICCS 2009. En A. Sandoval-Hernandez, M. M. Isac and D. Miranda (Eds.) Teaching Tolerance in a Globalized World. Cham: Springer International Publishing
Mullis, I. V. (2002). Background questions in TIMMS and PIRLS: An overview. Paper commissioned by the National Assessment Governing Board, http://www.nagb.org/release/Mullis.doc.
Neff, W. S. (1938). Socioeconomic status and intelligence: A critical survey. Psychological Bulletin, 35(10), 727.
Oberski, D. (2014). lavaan. survey: An R package for complex survey analysis of structural equation models. Journal of Statistical Software, 57(1), 1-27.
OECD (2017). PISA 2015 Technical Report. Paris, France: OECD Publishing.
OECD (2018). PISA 2015 Results in Focus. Available at: https://www.oecd.org/pisa/pisa-2015-results-in-focus.pdf Accessed 9 December 2019.
Oliveri, M. E., & Ercikan, K. (2011). Do different approaches to examining construct comparability in multilanguage assessments lead to similar conclusions?. Applied Measurement in Education, 24(4), 349-366.
Park, H., & Sandefur, G. D. (2006). Families, schools, and reading in Asia and Latin America. In Children’s Lives and Schooling across Societies (pp. 133-162). Emerald Group Publishing Limited.
Pitzalis, M., & Porcu, M. (2017). Cultural capital and educational strategies. Shaping boundaries between groups of students with homologous cultural behaviours. British Journal of Sociology of Education, 38(7), 956-974.
Pokropek, A., Borgonovi, F., & Jakubowski, M. (2015). Socio-economic disparities in academic achievement: A comparative analysis of mechanisms and pathways. Learning and Individual Differences, 42, 10-18.
Pokropek, A., Borgonovi, F., & McCormick, C. (2017). On the Cross-Country Comparability of Indicators of Socioeconomic Resources in PISA. Applied Measurement in Education, 30(4), 243-258.
Puzic, S., Gregurović, M., & Košutić, I. (2016). Cultural capital–a shift in perspective: An analysis of PISA 2009 data for Croatia. British journal of sociology of education, 37(7), 1056-1076.
Puzic, S., Gregurović, M., & Košutić, I. (2018). Cultural Capital and Educational Inequality in Croatia, Germany and Denmark: A Comparative Analysis of the PISA 2009 Data. Revija za socijalnu politiku, 25(2), 133-156.
R Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. Retrieved from https://www.R-project.org/.
Rogošić, S., & Baranović, B. (2016). Social capital and educational achievements: Coleman vs. Bourdieu. Center for Educational Policy Studies Journal, 6(2), 81-100.
Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling and more. Version 0.5–12 (BETA). Journal of statistical software 48(2), 1-36.
Rutkowski, L., & Rutkowski, D. (2010). Getting it ‘better’: the importance of improving background questionnaires in international large-scale assessment. Journal of Curriculum Studies, 42(3), 411-430.
Rutkowski, D., & Rutkowski, L. (2013). Measuring socioeconomic background in PISA: One size might not fit all. Research in Comparative and International Education, 8(3), 259-278.
Rutkowski, L., & Svetina, D. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31-57.
Rutkowski, L., & Rutkowski, D. (2018). Improving the comparability and local usefulness of international assessments: A look back and a way forward. Scandinavian Journal of Educational Research, 62(3), 354-367.
Sandoval-Hernandez, A., Rutkowski, D., Matta, T., & Mirnda, D. (2019). Back to the drawing board: Can we compare socioeconomic background scales?. REVISTA DE EDUCACION, (383), 37-61.
Segeritz, M., & Pant, H. A. (2013). Do they feel the same way about math? Testing measurement invariance of the PISA “students’ approaches to learning” instrument across immigrant groups within Germany. Educational and Psychological Measurement, 73(4), 601-630.
Sirin, S. R. (2005). Socioeconomic status and academic achievement: A meta-analytic review of research. Review of educational research, 75(3), 417-453.
Steiner-Khamsi, G. (2019). Conclusions: What Policy-Makers Do with PISA. En Florian Waldow and Gita Steiner-Khamsi (Eds.) Understanding PISA’s Attractiveness Critical Analyses in Comperative Policy Studies. Bloomsbury Publishing.
Tan, C. Y. (2015). The contribution of cultural capital to students’ mathematics achievement in medium and high socioeconomic gradient economies. British Educational Research Journal, 41(6), 1050-1067.
Thein, L. M., & Ong, M. Y. (2015). Malaysian and Singaporean students’ affective characteristics and mathematics performance: evidence from PISA 2012. SpringerPlus, 4(1), 563.
Tijana, P. B., & Anna, S. (2015). PISA The Experience of Middle-Income Countries Participating in PISA 200-2015. OECD Publishing.
Torney-Purta, J., & Amadeo, J. A. (2013). International large-scale assessments: Challenges in reporting and potentials for secondary analysis. Research in Comparative and International Education, 8(3), 248-258.
UN (2015). Transforming Our World: The 2030 Agenda for Sustainable Development, 2nd August 2015. New York: United Nations.
Uysal, N. K., & Arıkan, Ç. A. (2018). Measurement Invariance of Science Self-Efficacy Scale in PISA. International Journal of Assessment Tools in Education, 5(2), 325-338.
Vanderberg, R. J., & Lance, C. E. (2000). A Review and Synthesis of the Measurement Invariance Literature: Suggestions Practices, and Recommendations for Organizational Research. Organizational Research Methods, 3(1), 4-70.
Van de Vijver, F. J. R., & Leung, K. (1997). Methods and data analysis for cross-cultural research. Thousand Oaks, CA: Sage.
Van de Vijver, F. J. R., & Poortinga, Y. H. (2002). Structural equivalence in multilevel research. Journal of Cross-Cultural Psychology, 33, 141-156.
Van de Vijver, F. J. (2018). Towards an Integrated Framework of Bias in Noncognitive Assessment in International Large-Scale Studies: Challenges and Prospects. Educational Measurement: Issues and Practice, 37(4), 49-56.
White, K. R. (1982). The relationship between socioeconomic status and academic achievement. Psychological Bulletin, 91, 461-481.
Widaman, K. F., Reise, S. P., Bryant, K. J., & Windle, M. (1997). Exploring the measurement invariance of psychological instruments: Applications in substance use domain. Ariel, 165, 220-14.
Wu, A. D., Li, Z., & Zumbo, B. D. (2007). Decoding the meaning of factorial invariance and updating the practice of multi-group confirmatory factor analysis: A demonstration with TIMSS data. Practical Assessment, Research & Evaluation, 12(3), 1-26.
Wu, M. (2010). Comparing the Similarities and Differences of PISA 2003 and TIMSS. OECD Education Working Papers, 32, Paris: OECD Publishing. https://doi.org/10.1787/5km4psnm13nx-en
Yang, Y. (2003). Dimensions of socio-economic status and their relationship to mathematics and science achievement at individual and collective levels. Scandinavian Journal of Educational Research, 47(1), 21-41.
How to Cite
Published
Issue
Section
License
Any authors who publish with this journal accept the following terms:
