From Conception to Delivery and Beyond: Lessons Learned through Evaluation of the Auntie Pam’s Pregnancy Advisory Service
Kirklees Council
LARIA Research Impact Award 2015: Best Use of Social Care or Health Research
Summary:
What is Auntie Pam’s?
Auntie Pam’s is a local pregnancy advisory service in Kirklees (West Yorkshire) developed using a six-stage social marketing process (including an evaluation step). Auntie Pam’s provides volunteer-led peer support for mums and mums-to-be, with the aim of improving maternal health outcomes and the health of women of child-bearing age.
What did we do?
This entry focuses on how the Public Health Intelligence team completed the evaluation stage of the social marketing process for Auntie Pam’s, providing an opportunity to assess the impact of the service, influence its direction at a crucial point in its expansion, and create a general framework for evaluation of other services.
How did we do it?
We identified the types of data available and how these provided evidence for six outcomes. We combined data from several sources, incorporating qualitative and quantitative data. A new database and reporting system were designed to establish baseline and current performance. We reported the findings using a range of infographics, word clouds and written commentary, making this appropriate for a range of audiences.
Who benefitted from this work?
Mum and mums-to-be, the volunteers, our colleagues and maternity stakeholders.
Synopsis:
An evaluation was carried out to determine how well the Auntie Pam’s service was meeting its objectives, including assessment of performance in six outcome areas:
- Number of women who access the Auntie Pam’s service (single and repeat visits).
- Increased resilience to poor health behaviours and negative lifestyle circumstances.
- Improved emotional wellbeing, self-esteem and motivation.
- Number of women who successfully train as peer support volunteers.
- Improved and/or increased access to appropriate and timely services required by clients.
- Clients and volunteers involved in and influencing children’s and maternity services planning, development and delivery.
The evaluation process began by identifying which types of data were available and how these provided evidence for the six outcomes. To build this picture we combined data from several sources, incorporating qualitative and quantitative data. A new database and reporting system were designed to establish baseline and current performance.
An evaluation report was produced, which included a range of innovative presentation methods such as infographics, charts, word clouds and written commentary, making this appropriate for a range of audiences. The report included data collected routinely along with quotes and case studies from clients who have used the service. This report was structured around each of the six outcome areas and discussed why it was important, how it was measured, what we found and what this tells us. The report is available on the council’s Involve tool which can be accessed by colleagues, partners and the public. Volunteers from the service were consulted throughout the process and they fed back their knowledge on what should be collected and how this could be collected. We have received positive feedback from key stakeholders in maternity services who have said how useful the report has been for them.
The evaluation process provided a number of challenges, including acquisition and interpretation of quantitative and qualitative data, and assessing the impact of the service on volunteers, clients and the wider community. Valuable lessons were learned such as who benefits from the service, identification of support topics, referrals to other services and understanding the client profile.
Through the process of evaluating Auntie Pam’s we were able to evidence the range of short, medium and long-term health and wellbeing outcomes for the women who use the service, the women who give up their time to volunteer and the wider impact on the local Dewsbury community. Improvements made through the evaluation process have been replicated within the recently launched Auntie Pam’s service in Huddersfield. At the end of the evaluation process we were able to provide evidence of the impact of the service on the local community and provide suggestions for improvement, if required, for each of the six outcome areas which were then used to inform the new service.
Through this process a new evaluation framework was created by the Public Health Intelligence team, underpinned by Outcomes-Based Accountability (OBA), which may be generally applied to evaluation of other commissioned services. Importantly, a robust and sustainable evaluation model is now in place that supports the principles of service co-design.
What should LARIA members learn from this award entry?
Evaluation is essential:
Service evaluation is an essential aspect of the commissioning cycle. Ideally, evaluation methodology should be built into the project at the planning stage to ensure that supporting evidence for defined outcomes may be acquired.
Quality data is the secret ingredient:
It is vital to ensure that every service has an appropriate system in place for collecting and analysing the right sort of data. Appropriate indicators need to be measured to provide evidence of the outcomes the service is trying to achieve.
Develop your ideas by combining methods:
The use of mixed methods and both qualitative and quantitative data allowed us to gain a more complete picture of the service and the support it offered to local people.
Demonstrate change by collecting baseline data:
Through completing this evaluation, it highlighted the difficulty of measuring some of the outcomes and the importance of collecting baseline indicators to ensure that we can demonstrate what has changed since the intervention was put in place.
Photo Credit: ‘Castle Hill with Victoria Tower’ by Jerzy Kociatkiewicz