DCSF required ongoing evaluation of Connexions Direct (CXD) and its operation: examining interaction (many variables as per operational and process guidelines) with new target groups to implement changes to increase uptake. Research needed to be covert and ethical, while replicating real-life enquiries. Fieldwork was required from 8am-2am 7 days a week for six-week periods.

DCSF specified a mystery shopping methodology. However, our approach exceeded expectations: recruiting, and training young people (13-19) rather than 18+ to increase realism of interactions (vocal/textual) with advisers – acknowledging that service expectations vary greatly by age. Using mystery shoppers from the target age-group ensured understanding and evaluation of the process and interactions from the user’s perspective.

We recommended involving young people and Connexions Advisers in the design of data collection and evaluation tools: further capacity building. Assessment factors from previous exercises were built upon for more sophisticated assessments. We introduced audio recording (covered by CXD advisers’ existing employment contracts for quality monitoring) and text-capture, to allow desk-based review of interactions by researchers and a ‘control’ panel of young people and high-street Connexions Advisers. School-aged mystery shoppers were only used within early evening and weekends, under local authority work permits.

To meet the needs of DCSF Research Series publications, a 33 page report was produced with additional appendices detailing findings for each of the 15 scenario topics explored. The main report focussed on the process of contacting CXD and interacting with an adviser in order to receive information and/or advice.

Three-page Executive Summaries were also published as ‘Research Briefs’.

The mystery shopping (process evaluation) was published by the DCSF (http://www.dcsf.gov.uk/research/data/uploadfiles/DCSF-RW021.pdf) as was the CXD website evaluation (http://www.dcsf.gov.uk/research/data/uploadfiles/DCSF-RW025.pdf).

DCSF specified a mystery shopping methodology. However, our approach exceeded expectations: recruiting, and training young people (13-19) rather than 18+ to increase realism of interactions (vocal/textual) with advisers – acknowledging that service expectations vary greatly by age. Using mystery shoppers from the target age-group ensured understanding and evaluation of the process and interactions from the user’s perspective.

We recommended involving young people and Connexions Advisers in the design of data collection and evaluation tools: further capacity building. Assessment factors from previous exercises were built upon for more sophisticated assessments. We introduced audio recording (covered by CXD advisers’ existing employment contracts for quality monitoring) and text-capture, to allow desk-based review of interactions by researchers and a ‘control’ panel of young people and high-street Connexions Advisers. School-aged mystery shoppers were only used within early evening and weekends, under local authority work permits.

Although the mystery shopping methodology tends to be seen as a customer satisfaction approach when applied to ‘how it feels’ to receive/use a service, it is also a powerful tool for monitoring adherence to prescribed procedures and can underpin reviews of the appropriateness of processes used. Within this three-phase exercise, our team recorded and analysed the success rate for contacting CXD by four methods (and by time-of-day/week); content and warmth of greeting; verbal/written communication styles; listening/probing; information provision; and closing interactions. Analysis and recommendations made reference to the CXD contract and internal operational guidance.

To meet the needs of DCSF Research Series publications, a 33 page report was produced with additional appendices detailing findings for each of the 15 scenario topics explored. The main report focussed on the process of contacting CXD and interacting with an adviser in order to receive information and/or advice.

Three-page Executive Summaries were also published as ‘Research Briefs’.

The mystery shopping evaluation was published by the DCSF (http://www.dcsf.gov.uk/research/data/uploadfiles/DCSF-RW021.pdf) as was the CXD website evaluation (http://www.dcsf.gov.uk/research/data/uploadfiles/DCSF-RW025.pdf).