Impact Assessment of Stem Initiatives in Improving Educational Outcomes: Research Report from a National Evaluation Conducted to Inform Policy and Practice

2017 Pallavi Amitava Banerjee. All rights reserved.

No part of this book may be reproduced, stored in a retrieval system, or transmitted by any means without the written permission of the author.

Published by AuthorHouse 08/08/2017

ISBN: 978-1-5246-8287-3 (sc)

ISBN: 978-1-5246-8288-0 (e)

Any people depicted in stock imagery provided by Thinkstock are models,

and such images are being used for illustrative purposes only.

Certain stock imagery © Thinkstock.

Because of the dynamic nature of the Internet, any web addresses or links contained in this book may have changed since publication and may no longer be valid. The views expressed in this work are solely those of the author and do not necessarily reflect the views of the publisher, and the publisher hereby disclaims any responsibility for them.

Contents

Preface

Tables

Figures

Abbreviations

Part I Introduction

1 Introduction

Part II The policy context

2 Should STEM education matter?

3 Is widening participation in STEM education important?

Part III The UK scenario

4 The STEM crisis

5 Current concerns for STEM education in the UK

Part IV The way forward

6 Attitudes and aspirations

7 STEM informal education

Part V The evaluation

8 The need for an evaluation

9 Research design and methods

10 Data analysis

Part VI Conclusion

11 Research findings

12 Conclusions and implications

13 Recommendations for practice

Bibliography

Appendix C Number of students achieving A*-C in maths & total entries

Appendix D Number of FSM pupils taking up AS/A levels in STEM subjects

Appendix E Number of Black pupils taking up AS/A levels in STEM subjects

Preface

Economic competitiveness needs a steady supply of skilled workforce. Science, technology, engineering and mathematics (STEM) skills are thus globally valued. Serious concerns over inadequate skilled personnel to meet the predicted labour market demands have led to the introduction of STEM initiatives. These schemes have been introduced and run for over a decade now to raise attainment and improve attitudes of students in school towards science and maths. These schemes call for huge investments of time, money, and resources. There has been a recent world-wide move towards demanding evidence-based policy and practice in education, with policy-makers and practitioners wanting more practical and coherent answers. Funders, schools and teachers often take advantage of the new evidence generated by implementing evidence – based reforms.

However, such evaluations have been woolly for several reasons including continued resistance from activity providers to share data or encourage third party evaluations. This means the funding is disproportionately still being allocated with an unhealthy reliance on repeat business like evaluations – such as testimonials from head-teachers on the websites of activity providers when they were not even attending the events themselves and had no first-hand information of what the activity entailed or how students engaged with it or did it influence their subject choices and learning trajectories.

This book presents innovative methods for the design, conduct, analysis and use of evidence from robust evaluations. The book describes the promises, the problems, the new opportunities of STEM initiatives and their evaluations of improving attainment to political concerns and wider educational impacts. The implications are spelled out for the research community, policy-makers, schools wanting to enrol their students or run their own evaluations, practitioners using evidence and the wider public (who pay for research, these activities and who are the participants in education as parents, teachers, activity providers or students)

The book will appeal to those interested in the substantive area of educational research, or needing to understand the advantages and limitations of evidence based on policy making and practice regarding education. This will include those reading, thinking about conducting robust evaluations and wanting to understand what works and why. It will be a key text for any school wanting to undertake their own evaluations or for research leads trying to pick their way through evidence for use in practice. The research design and method proposals in this book, such as the use of secondary data from government, linkage, estimation of effect sizes and regression modelling have never been presented together as a solution to the kind of impact assessment planned here.

As described in this book, there are problems with the way evaluations are often envisaged by others and then commissioned, conducted, reported and synthesised. Opportunities for real advancements in education research have been missed so many times. I hope that this book will help to decide to act before more money is spent on all initiatives rather than identifying the best ones and building upon them.

I am grateful to the funders of this research output, the Department for Education (UK), to the activity providers, staff and pupils of the thousands of schools taking part in this project.

Tables

Table 3.1 Percentage attainment by disadvantaged groups, 2012/13

Table 5.1 Mainstream science qualifications as function of cohort-size (a)

Table 5.2 Number of 16-18 year olds in England taking GCE A-level sciences

Table 5.3 Percentage of 16-18 year olds taking core sciences and mathematics

Table 5.4 Graduate jobs: Top 10 degree subjects for getting a job

Table 9.1 Overview of research project

Table 9.2 Schools participating in STEM activities from 2007-2012

Table 9.3 Number of students in sub-groups - STEM activity participation

Table 9.4 Initial response rate

Table 9.5 Number of participating schools registered with STEM activity providers

Table 9.6 Number of schools – population and intervention group

Table 10.1 School types in intervention group

Table 10.2 Number of schools – longitudinal intervention and comparator

Table 10.3 KS4 pupils’ date of joining current school

Table 10.4 KS3 pupils’ date of joining current school

Table 10.5 Breakdown of analytical sub-groups

Table 10.6 FSM eligibility – frequency table all pupils

Table 10.7 Major ethnic groups - frequency table for maintained mainstream schools

Table 10.8 Major language groups - frequency table

Table 10.9 Pupil who passed maths GCSE at A*-C

Table 10.10 Achieved two ‘good’ GCSE science GCSEs or equivalent

Table 10.11 Pupil who passed science GCSE at A*-C

Table 10.12 Missing data

Table 10.13 Collinearity statistics for regression analysis for maths test score prediction

Table 10.14 Collinearity statistics for regression analysis for science test score prediction

Table 10.15 Number of cases in intervention sub-groups and comparator

Table 10.16 Qualification routes taken by 16-18 year olds, England

Table 10.17 Trigger variable

Table 10.18 Trigger criteria

Table 11.1 Mean percentage of pupils achieving maths performance indicator

Table 11.2 Annual achievement gap estimation, intervention and comparator schools

Table 11.3 Correlation between attainment and FSM eligible pupils

Table 11.4 Comparison of mean maths performances – longitudinal group

Table 11.5 School FSM intakes and its correlation with attainment

Table 11.6 Pupil maths attainment - A*-C in GCSE maths 2011-12

Table 11.7 FSM versus R – pupil maths attainment

Table 11.8 Percentage students achieving A*-C in maths

Table 11.9 Maths achievement gap – intervention and comparator groups

Table 11.10 Achievement gap estimation between advantaged and disadvantaged groups

Table 11.11 Maths attainment - FSM and non-FSM pupils

Table 11.12 Achievement gap: FSM pupils and non-FSM pupils

Table 11.13 Effect size - ratio of probability of success in treatment versus comparator

Table 11.14 Percentage pupils in various ethnic groups achieving A*-C in maths

Table 11.15 Achievement gap between Chinese and other ethnic groups

Table 11.16 Ratio of probability of success –intervention group & comparator

Table 11.17 Models from multiple regression analysis for predicting of maths test scores

Table 11.18 Logistic regression output for maths attainment

Table 11.19 Mean percentage of pupils achieving science performance indicator

Table 11.20 Annual science achievement gap estimation, intervention and comparator schools

Table 11.21 Correlation and %FSM eligible pupils

Table 11.22 Comparison of mean science performances – longitudinal group

Table 11.23 School FSM intakes and correlation with attainment

Table 11.24 Pupil attainment - A*-C in GCSE science 2011-12

Table 11.25 FSM versus R – pupil science attainment

Table 11.26 Percentage of pupils achieving A*-C in science

Table 11.27 Achievement gap – intervention groups versus comparator

Table 11.28 Achievement gap- advantaged and disadvantaged pupils

Table 11.29 Percentage of FSM pupils attaining A*-C in GCSE science

Table 11.30 Achievement gap between poor pupils and their most affluent peers

Table 11.31 Effect size estimates

Table 11.32 Percentage pupils achieving A*-C in GCSE science by ethnicity

Table 11.33 Achievement gap of various ethnic groups with Chinese pupils

Table 11.34 Effect size estimate – science attainment by ethnicity

Table 11.35 Multiple linear regression analysis models for prediction of science test scores

Table 11.36 Logistic regression output for science attainment

Table 11.37 GCSE attainment by subject

Table 11.38 Percentage pupils achieving five or more GCSE/GNVQs at grades A*- C

Table 11.39 Number of pupils progressing from GCSE to AS/A levels by STEM subject

Table 11.40 pupils progressing from GCSE to AS/A levels

Table 11.41 Post-16 STEM participation – effect sizes

Table 11.42 Progression rates of FSM pupils from GCSE to AS/A levels by STEM subject

Table 11.43 Number of FSM pupils taking up AS/A levels in STEM subjects

Table 11.44 Post-16 STEM participation FSM – effect sizes

Table 11.45 Progression rates of Black pupils from GCSE to AS/A levels

Table 11.46 Number of Black pupils taking up AS/A levels in STEM subjects

Table 11.47 Participation gap and success ratio AS-&A-levels for STEM participation

Figures

Figure 1 How often do visitors come to your school to talk about STEM?

Figure 2 Percentage of pupils achieving A*-C in maths by % FSM pupils in school (Banerjee, 2016, 2017)

Figure 3 Residuals in Regression (Banerjee, 2017)

Abbreviations

CBI - The Confederation of British Industry

DBIS - Department of Business Innovation and Skills

DCSF - The Department for Children, Schools and Families

DfE - Department for Education

EAL - English as an Additional Language

GCSE - General Certificate of Secondary Education

HEFCE - Higher Education Funding Council for England

IDACI - Income Deprivation Affecting Children

LSYPE - Longitudinal Study of Young People in England

NAU - The National Audit Office, UK

NPD - National Pupil Database

NVQ - National Vocational Qualification

QCA - The Qualifications and Curriculum Authority

UPN - Unique Pupil Number

SCORE - Science Community Representing Education

SEN - Special Educational Needs

SES - Socioeconomic Status

STEM - Science, Technology, Engineering and Mathematics

TIMSS - Trends in International Mathematics and Science Study

TISME - The Targeted Initiative on Science and Mathematics Education

UCAS - The Universities and Colleges Admissions Service

Part I

Introduction

1

Introduction

The development of a knowledge intensive workforce has been established as one of the primary sources of advantage (Smith, 2005) in trying to make the economy more competitive globally. Successive governments have considered increasing and widening STEM participation (Conway, 2009) important to enhance the quality of labour (NCC, 2006) and promote the concept of lifelong learning (Report of the Task Force on Lifelong Learning, 2002). The 2020 Vision thus aims to improve a) graduate and employability skills of STEM students, b) student confidence in their employability, c) student awareness of their career options and d) employer engagement with higher education institutions (HEIs). These objectives have seen the growth of several funded STEM schemes and initiatives to raise awareness, attitudes and aspirations towards these subjects and careers. The effectiveness of these schemes are yet to be examined.

There is a need to raise young people’s awareness of the progression routes and career opportunities that can be accessible through studying STEM subjects. Advancing access, STEM careers and resources made available by the National STEM Centre are some of the steps towards this. Accurate and accessible advice is being delivered to young people, relating to subject choice, entry requirements (for higher education) and progression to STEM careers. Several educational programmes are being simultaneously run nationally and locally to enthuse young minds, raise awareness and improve their educational outcomes in science and maths subjects in schools.

This research was motivated by a recognition amongst the STEM community and policy circles regarding the need to understand what works and how successful these initiatives have been? Given the plethora of activities being available how do schools decide which activities should they register their students for? It is certainly not going to work if schools just agreed to register with the provider who makes a direct contact and shares how other local schools have enrolled and benefitted. Where is the evidence and how do we know whether an educational programme which has worked for a certain school and year group will work for others.

Schools and teachers already use evidence in the form of testimonials of participating schools displayed on the activity provider’s website to decide whether an activity would be suitable for their students. Such evidence is very rarely critically evaluated and hence raises questions about the reliability or usefulness of these schemes for diverse groups. Perhaps what works best for one social group or school may not work for another. With the growing demands of evidence based policy the research protocol reported here will help schools, teachers, and even activity providers to carry out self-evaluations. This will also encourage third-party evaluations of STEM enrichment and enhancement programmes being run in different settings and in different countries.

Organisation of the rest of this book

Part II explains why STEM education matters. It explains the need to have a STEM work force comprised of people from various backgrounds and social groups. Part III maps the policy background which led to the development of STEM schemes in the UK. I discuss why governments over the years have felt the need to allocate funds and resources in trying