Best use of local area research 2013 sponsored by ORS

Manchester City Council

The RCT methodology in Manchester has given partners the confidence to support new and potentially ‘untested’ processes, where they have invested resources to support the trialling of the NDM. The upfront investment in a strong and agreed evaluation methodology has proved as powerful as actual evidence.

Summary

In 2010, Manchester City Council was looking for an evidenced based intervention that would meet the needs of the city’s complex families. Individual interventions already existed in Families First (FF), Family Intervention Project (FIP), and Complex Families Parenting Team (CFPT), however no single ‘complex families’ intervention existed.

Therefore, Manchester City Council worked in partnership with the University of Manchester to develop a robust Randomised Control Trial (RCT) methodology. This RCT would test how a new delivery model (NDM) focusing on integration, prioritisation & sequencing could improve the effectiveness of the individual existing/business as usual (BAU) interventions.

The RCT began in April 2011, when the first families were allocated to the new delivery model, and ends in 2013/14 when a target cohort of at least 480 families (240 NDM and 240 BAU) would be reached.

The final results of the RCT will not be concluded until the full cohort of families has completed their interventions. However, a series of mid-point reviews, alongside the confidence the RCT methodology has provided to partners, have already provided strong evidence to support the refinement of processes and maximisation of outcomes.

Synopsis

The RCT in Manchester uses a research technique more commonly used to test the effectiveness of products, to assess the social and fiscal impact of a new service. This is an innovated approach to filling an evidence void, and the first of its kind in local government.

Manchester City Council worked in partnership with the University of Manchester to develop an RCT methodology and the University’s involvement as a ‘critical friend’ continues throughout the RCT. The RCT methodology needed the support of senior management, the interventions and partner agencies, as its implementation would require resource and ethical decisions to be taken on individual cases. For example, a high-need BAU case may not able to access certain services – a risk for managers, a frustration for interventions and a resource issue for partners.

A minimum cohort of 480 families was agreed as part of the methodology so that significance testing is within a reasonable range, ensuring that there is not ambiguity in final analysis of the results.

The cohort has been established through a ‘taxi-rank’ allocation system, linked into the referral process. Referrals come directly from partner agencies, who consider families against the minimum criteria for the interventions, and submit a standard referral/consent form. This approach ensures that cases are submitted randomly with no interference from the City Council, the taxi-rank randomisation then allocates alternate cases to the NDM and BAU groups.

Once randomised the referrals then follow two separate paths. The first being the BAU cases, where these referrals follow a standard BAU process with no further support from the research team. The second are the NDM cases, where referrals are taken to an allocation panel to discuss the best combination of interventions to work with the family. Once allocated the research team provide support to case workers to collate monthly performance monitoring at a case level, so that partners have a transparent process which they are able to add value to through greater integration.

Confidence has been placed in the research team, so that they ensure that model fidelity is maintained and issues of contamination / quality control mitigated.

The final findings from the RCT will not be available until the last of the families have completed their interventions, however outcomes are reviewed on a 6-monthly basis. These ‘mid-point’ reviews have enabled the City Council and its partners to take confidence from the NDM processes, and ensure on-going commitment to what is a 2/3 year pilot period.

In addition the intelligence gathered as part of the RCT enables Manchester to respond quickly to new Government initiatives. When the national Troubled Families Programme was introduced, Manchester already understood the complex characteristics of these families and the support they require.

The RCT, rather than testing the impact of a single service or product is testing the impact integration, priorisation and sequencing of services, has on outcomes. The confidence this robust approach has already given is already showing itself in the form of upfront investment of resources to pilot these new approaches.

Conclusions

A strong evaluation methodology provides confidence which can enable support for new and potentially ‘untested’ processes. The evaluation methodology itself helps to mitigate some of the risk of these new processes, and makes a clear link between robust evaluation and potential investment.

The implementation of an RCT approach is not an straightforward one and requires not only the support and rigor of researchers, but also the buy-in from senior managers, frontline workers and investment partners. Without there trust in the research team, then maintaining model fidelity and mitigating contamination issues would be near impossible.

Agree a suitable period to run the trial over, and don’t rush the results. Ensure that you review progress on a regular cycle to maintain buy-in and support, but be clear that evaluation takes time and outcomes are not always realised immediately.

Link the findings directly into the programme, immerse the research work in the programme and ensure that findings feed other processes (i.e. Investment decisions / Cost Benefit Analysis)

There are not results available at present from the RCT as the trial is still running, however some background information on the process is available at the Manchester Partnership Website

Photo credit: Manchester City Council winning the 2013 award