Julia Salado-Rasmussen is a Doctoral Researcher, Centre for Labour Market Research (CARMA) & Research Centre for Evaluation (FCE), Aalborg University, Denmark and was a Visiting Researcher at CERIC in May 2015.
Active labour market policies (ALMPs) continue to be a hot topic in Danish politics, as well as in many other countries. Despite a substantial amount of resources spent on ALMPs there is still a lack of results, especially when it comes to the disadvantaged unemployed. The disadvantaged unemployed have problems besides being without a job (e.g. health, economic or social problems) and are often long-term unemployed. This blog examines the results of ALMPs and discusses the challenges of evaluating ALMPs. I also present my PhD research design, which sets out how I will evaluate ALMPs by using realistic evaluation techniques.
Across OECD countries, Denmark has the highest level of public expenditure on ALMPs. Denmark spent 2.3 per cent of GDP on ALMPs in 2011. In comparison, the second biggest spender is Belgium (spending 1.6 per cent of GDP), followed by the Netherlands and Sweden (both spending 1.1 per cent of GDP in 2011). The total amount spent on ALMPs in Denmark was around DKK 13.5 billion in 2013 (around £1.3 billion). It has been pointed out in several papers that the practice of evaluating ALMPs is much less developed in Europe than in the United States. In light of the relatively high amounts spent on ALMPs in Denmark, and in Europe in general, compared to the US, this seems rather paradoxical.
Evaluations of ALMPs in Denmark show little consensus about what approaches seem to work. There is particular lack of knowledge about the disadvantaged unemployed as a group, something which is mirrored in the wider international literature. The studies undertaken in Denmark indicate that there is an overall positive effect of ALMPs. However, a number of programmes have no effects, or have directly negative effects during recessionary periods. The Danish Economic Council goes as far as to conclude that all ALMPs prolong unemployment under recession due to ‘locking-in effects’. Locking-in effects mean that in the period when the unemployed participate in a programme, they may seek jobs less actively, because they want to complete the programme they are attending or because they have less time to search for a job. Nonetheless it seems that programmes that take place at real companies, using private wage-subsidies and internships (virksomhedspraktik), work for the disadvantaged unemployed as well as for other groups. Employers receive up to 50 per cent of wage costs when they hire an unemployed worker via the wage-subsidy scheme, and they get the labour for free when they hire an unemployed worker via an internship, since the unemployed person continues to receive their income through benefits from the Jobcentre.
Research shows that results depend on the state of the economy (boom or recession), the target group, the area of implementation and also, crucially, the evaluation method and design (e.g. whether the ‘motivation effect’, which refers to the pattern of increased job search activity and employment just before the unemployed are forced into a programme), is included or excluded). Furthermore there is a tendency to rank evaluations in a hierarchy, where studies based on randomized controlled trials (RCTs) are considered the ’gold standard’. The uncompromising and typically narrow focus of studies based on RCTs means that evaluations based on other methods are often excluded from meta-analyses, thus leaving out potentially fruitful knowledge. RCTs are suitable for explaining whether an intervention leads to an outcome or not, but typically cannot explain the causal relations between interventions and outcomes, which then become a “black box”. Why did the intervention work and how? Moreover, to make experimental studies possible programmes are often pooled into larger categories, thus obscuring the nuances between different programmes. All these circumstances make it hard to transfer the results into political action.
The black box approach makes it hard to replicate successful programmes. Since there is limited evidence on why the programme works, programmes which seem to work may be copied in their entirety to ensure no essential parts are left out, which in practice can be difficult. This issue is especially interesting in a British context, and the current Work Programme’s ‘black box’ commissioning approach, where providers can personalize the support for each individual and are paid by results. At some point one would expect that politicians would be interested in knowing what the high-performing providers which help more unemployed people into sustained work do and what types of programmes they offer.
In my PhD project I advocate for a more open-minded approach to evaluating ALMPs. The project applies realistic evaluation and evaluates a number of selected programmes directed towards the disadvantaged unemployed. Working within the framework of realistic evaluation, the research project incorporates a mixed-method approach that allows for an opening of the ’black-box’ and thus deals not only with ‘what works’ but also for whom, why and in what context. As Ray Pawson writes in his latest book from 2013: “Why does a programme work in Wigan on a wet Wednesday and why does it fail in Frinton on a foggy Friday?”
The research design is based on qualitative data including interviews with the unemployed and practitioners and quantitative data on employment from the DREAM-register system held by the Danish Ministry of Employment. The aim is to focus on the underlying programme theory (assumptions) about how ALMPs work. What mechanisms make the programme work? When the programme theory is the unit of analysis, instead of the active labour market programme as such, it becomes easier to generate learning from one programme to another. Thus the same programme theory often repeats itself across different types of active labour market programmes and national settings. By using realistic evaluation it will hopefully be possible to get a deeper understanding of what works in ALMPs.