On Tuesday 23rd February, the University hosted one of a series of regional events, “Everybody Active Everyday: ‘Moving at Scale’”. The day was a joint venture by Public Health England, the BHF National Centre, the NCSEM and the CSP Network and was designed to bring together physical activity professionals from across the region to discuss the evaluation, of ‘what works’ in increasing activity levels at both individual and community levels and the barriers and facilitators to scaling up the effects of the most successful programmes. This article contains some links and notes from the day that we thought were worth sharing.
About 80 delegates attended from across the region, including local health commissioners, county sports partnership employees, voluntary sector leaders, clinicians and academics, all invited as those that have a good understanding of local strategic developments.
The variety of roles that were represented helped to illustrate the various perspectives of different stakeholders on topics such as evaluation and funding and throughout much of the day there was talk about different ways in which participants could explore new collaborations.
Topics Covered
Taking a systemic approach to evaluation
Perspectives on evaluation
Planning your intervention: using evidence, engaging stakeholders and assessing needs
Using innovative approaches to strengthen evaluation study design
Identifying outcomes and selecting measurement tools
Assessing implementation and scaling up interventions
Assessing economic benefits of physical activity interventions
Embedding evaluation of interventions
Themes & notes from the day
Why do evaluation?
Evaluation can be marketing
Product placement
Good evaluation can help translate evidence into practice
Learn and share best practice
Make more efficient use of resources
Recalibration of provision
Considerations in evaluation
We should focus on future decision making processes to help design the evaluation process.
Measure to improve or improve to measure.
The value placed on evaluation can be different for practitioners, academics, commissioners and policy makers.
Rigour v relevance.
Intervention fidelity v real life feasibility – Process evaluation should assess how closely the delivered intervention stuck to the originally intended content. By doing this we can investigate potential reasons for limited intervention outcomes or in contrast judge whether successful intervention outcomes were due to the particular components of any intervention. In addition to maintaining intervention fidelity, we must also be able to allow for adaptation across multiple sites of a complex intervention and this tension between delivering the same intervention and allowing for variation is a challenge that should be reported where present.
Cost effectiveness – funding for interventions was a recurring theme throughout the day with many organisations facing significant funding cuts and having to turn to partners and more innovative interventions in order to make an impact. Many interventions can show that the benefits they deliver can produce cost effective downstream savings in health and wellbeing that suggest they are worth the initial investment. The problems seem to arise as many organisations and departments are responsible for their own budgets (and the relevant cuts) and not only are the health benefits not realised for a number of years but these benefits are then realised by other departments rather than those that make the initial investment. Examples were also given of reduced use of transport having a negative impact on central government tax revenues and the interesting case of an intervention that was cost effective in preventing or delaying significant orthopaedic surgery. This surgery, however, brought in more money to the NHS trust than it cost to deliver, and that surplus income was being used to fund other clinical services that cost more to deliver than the levels of funding were provided. The competing tensions that arise from working in a complex healthcare system were echoed in different ways by a number of those in attendance, it was evident that the existence of silos of practice was problematic and that a whole systems approach to healthcare would be more productive. For more on the consideration of cost effectiveness see ROI tools outlined below.
Study Design and Levels of evidence – examples of different methods of evaluating interventions were discussed and an understanding of the various levels of evidence was promoted. See further reading below for more.
“Do good process evaluations always, do outcome evaluations often” Adrian Bauman
Follow some of those involved in the day:
Nic Cavill - @cavilln
Rob Copeland - @drrobcopeland
Public Health England - @phe_uk
British Heart Foundation National Centre - @BHFactive
The National Centre for Sport & Exercise Medicine England - @NCSEMengland
Tom Wainwright – @twwainwright
Emma Adams - @EmmaJ_Adams
Active Devon - @activedevon
Ben Jane - @benjanefitness - see our slides from the day at the end of this blog
Links and resources from our notes
Andrews, L. M., Allen, H., Sheppard, Z. A., Baylis, G., & Wainwright, T. W. (2015). More than just ticking a box… how patient and public involvement improved the research design and funding application for a project to evaluate a cycling intervention for hip osteoarthritis. Research Involvement and Engagement, 1(1), 1.
BMJ Learning Module: Physical activity in the treatment of long term conditions http://learning.bmj.com/learning/course-intro/physical-activity.html?courseId=10051913
Craig, P., Cooper, C., Gunnell, D., Haw, S., Lawson, K., Macintyre, S., ... & Thompson, S. (2012). Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. Journal of epidemiology and community health, jech-2011. http://jech.bmj.com/content/early/2012/05/10/jech-2011-200375.short
Global Advocacy for Physical Activity (GAPA) the Advocacy Council of the International Society for Physical Activity and Health (ISPAH) . NCD prevention: investments that work for physical activity. Br J Sports Med 2012;46:709–12. [also at www.globalpa.org.uk/investmentsthatwork]
Newton, J. N., Briggs, A. D., Murray, C. J., Dicker, D., Foreman, K. J., Wang, H., ... & Vos, T. (2015). Changes in health in England, with analysis by English regions and areas of deprivation, 1990–2013: a systematic analysis for the Global Burden of Disease Study 2013. The Lancet, 386(10010), 2257-2274.
Ogrinc, G., Mooney, S. E., Estrada, C., Foster, T., Goldmann, D., Hall, L. W., ... & Nelson, W. (2008). The SQUIRE (Standards for QUality Improvement Reporting Excellence) guidelines for quality improvement reporting: explanation and elaboration. Quality and Safety in Health Care, 17(Suppl 1), i13-i32. https://www.med.unc.edu/pediatrics/about/quality/documents/squire%20guidelines-pub.pdf
Plymouth City Council Cycling Projects
Policy Innovation Research Unit (2015) Advice on commissioning external academic evaluations of policy pilots in health and social care: a discussion paper
Public Health England (2015) Evaluation of weight management, physical activity and dietary interventions: an introductory guide
Public Health England (Jan, 2015) Adult physical activity data factsheet https://www.noo.org.uk/securefiles/160226_1226//PA_Factsheet_Adults_Jan%202015_revisions.pdf
Public Health England Briefing Papers http://www.noo.org.uk/NOO_pub/briefing_papers
Public Health England National Obesity (Observatory Resource) Evaluation Guidance
Public Health England. Health Impact of Physical Inactivity http://www.apho.org.uk/resource/view.aspx?RID=123459
Smith, G. C., & Pell, J. P. (2003). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ: British Medical Journal, 327(7429), 1459. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC300808/pdf/32701459.pdf
Varney, J. and Fenton, K. (2015) Moving healthcare professionals at every level https://publichealthmatters.blog.gov.uk/2015/10/19/moving-healthcare-professionals-at-every-level-2/
Return on Investment (ROI) Tools
Public Health England (2015) Guide to online tools for valuing physical activity, sport and obesity programmes http://www.noo.org.uk/securefiles/160226_0947//online_tools_briefing_13011_%20FINAL.PDF
WHO/Europe Health Economic Assessment Tool (HEAT) http://www.heatwalkingcycling.org/index.php
This tool is designed to help you conduct an economic assessment of the health benefits of walking or cycling by estimating the value of reduced mortality that results from specified amounts of walking or cycling
NICE Physical activity return on investment tool
The tool enables the user to evaluate a portfolio of interventions in their geographical area (e.g. region, county or local authority) and models the economic returns that can be expected in different payback timescales. The different interventions included in the tool can be mixed and matched to see which intervention portfolio or package provides the best 'value for money', compared with 'no package of interventions' or any other specified package.
Further Reading:
Bauman, A., & Nutbeam, D. (2013). Evaluation in a nutshell: a practical guide to the evaluation of health promotion programs. McGraw Hill.
Moore, G. F., Audrey, S., Barker, M., Bond, L., Bonell, C., Hardeman, W., ... & Baird, J. (2015). Process evaluation of complex interventions: Medical Research Council guidance. BMJ, 350, h1258. http://www.bmj.com/content/350/bmj.h1258
Study Design and levels of evidence:
The slides from our presentation on the day:
by Ben Jane & Saul Bloxham
No comments:
Post a Comment