Determining the causal impact of the policy

7The first evaluation difficulty lies in determining a causal relationship between a policy and a result. Let us suppose, for example, that we wished to evaluate the impact of healthcare expenditure on the level of health of a population. The simple correlation, within the population, between healthcare expenditure and level of health is negative, since the individuals who spend the most are generally the least healthy. This is a case of reverse causality, from level of health to expenditure, which does not provide us with any information on the impact of expenditure (Figure 1).

1

An example of reverse causality

An example of reverse causality

8The identification of a causal relationship between healthcare expenditure and level of health also comes up against interference from external factors, such as living standard, which have an influence on both expenditure and level of health: comfortably off individuals spend more on their health and are generally speaking more healthy, amongst other reasons because they carry out lower risk activities. The correlation between healthcare expenditure and level of health does not therefore correspond to any causal relationship (Figure 2).

2

An example of correlation without causality

An example of correlation without causality

9In order to identify the impact of healthcare expenditure on the level of health of individuals, we would need to compare not the level of health of the “major” consumers of healthcare with that of “minor” consumers, but the level of health of the same person individually depending on their consumption of healthcare. Since the same individual cannot consume both a high and a low level of healthcare, our approach must be based on a large number of individuals in respect of whom all of the characteristics liable to influence their level of health can be closely monitored independently of healthcare expenditure. In estimating healthcare expenditure and requirements, Martin et al. (2008) [2] were able to demonstrate a positive causal impact: increasing healthcare expenditure to fight against cancer and cardiovascular disease saves lives. Estimates reveal that on average cancer care can add one year of life for 13,100 pounds sterling and for cardiovascular disease one year can be added for 8,000 pounds.

10The same types of problems are encountered when we seek to evaluate return to work accompaniment policies. Unless caution is exercised, we may find that those who received accompaniment took longer than other unemployed persons to find a job. But we must take into account the fact that the Job Centre personnel do not necessarily allocate this accompaniment randomly: they may direct accompaniment to those individuals most disadvantaged in terms of employability or inversely, upon those persons who are the closest to employment (particularly where centre employees receive a bonus for each unemployed person who finds work). This is referred to as selection bias: accompanied persons are not drawn randomly from the unemployed population; by the same token, in the previous example, individuals who spend a lot on their health are not drawn at random from the population (they are generally less healthy to begin with).

The incidence issue

11The second issue encountered in public policy evaluation is the issue of incidence: the ultimate beneficiary of the policy is not necessarily the individual targeted. This second issue is frequent in the case of taxation or subsidies/transfers. Tax incidence theory reveals that the burden of taxation may not ultimately fall upon the person who writes the cheque for it: the parties taxed may pass on the cost of the tax to others; inversely, individuals not originally targeted by a subsidy may indirectly find that they are its beneficiaries.

12An illustration of this is provided by housing benefits. In 2009 this rental subsidy constituted a quarter of all benefits paid to French households. If the households officially receiving these benefits were the actual beneficiaries, housing benefit would account for over one-fifth of the reduction in inequality in living standards achieved by the welfare system. [3] However, this redistributive action is questioned by Fack (2005) [4] who considers that between 50% and 80% of these benefits in fact benefit lessors through increased rents as a result of market mechanisms (housing supply and demand): because the rental offer does not change very much in the short and medium term, demand subsidies raise rents, such that the support intended for disadvantaged tenants is in part received by the owners of rental accommodation. This example reveals that a policy of this type cannot be evaluated simply by limiting our enquiry to means or to the number of households receiving the benefit. Any analysis of housing benefit that fails to take into account its impact on rents would lead to an overly optimistic assessment (Figure 3).

3

The need to take incidence into account: the example of housing benefits

The need to take incidence into account: the example of housing benefits

13Furthermore, the effect of the policy can result detrimental to those persons not benefiting from the policy. This is the case with lessees who do not receive housing benefit (they are affected by increased rents) – not to mention the additional taxation required to finance the policy.

Multiplicity of effects

14The third difficulty in evaluation resides in the multiplicity of effects of a single public policy. Multiple effects may firstly arise in the same public policy field. For example, introducing a medical excess can be effective in combatting over-medication but at the same time result in a disturbing failure to access care (Box 1). In other cases, multiple effects appear in different public policy fields. For example an increase in the taxation of company profits may result in a decrease in the remuneration not of investors, but of the work-force. [5]

15Where the adverse effect of a policy occurs outside the field of investigation of the evaluator, or outside the variables relevant to their study, or even their disciplinary field, there is a high risk that it will be neglected. This is why the plurality of evaluators, in terms of discipline as much as sensitivities, is primordial in order to understand all of the consequences of a public policy without being restricted to those closely linked to the initially targeted objective.

Practical implementation

16Finally, the issue of practical implementation can have a strong influence over the efficiency of a policy. In certain cases, it may be tempting to reject a measure that has merely been applied under inappropriate conditions. Accordingly, the activity-based hospital tariff (T2A) probably did not yield the hoped for results in France not because of any error in its design, but rather because of an excessively detailed classification of hospital activities and an unfavourable interaction with other measures. [6] To resolve these problems, it may prove pertinent to carry out, where possible, a small scale preliminary experiment in order to determine the operating parameters that are crucial for a public policy to be successful (see Box 2). Such an experiment in no way dispenses with the need subsequently to accurately evaluate the reform in the aggregate. In fact, prior experimentation may favour the creation of a protocol for a subsequent evaluation of policies. Specifically, there is a need to harmonise legal and administrative constraints with compliance with the economic mechanisms targeted, by leading administrations to enter into discussion with experts (and not, as is currently the case, by sequentially introducing the various stages of the decision).

Random experimentation

19As we have seen, a major difficulty in evaluation is linked to the fact that individuals or businesses targeted by a public policy are not taken randomly from the population: for example, they possess below average health, or are of below average employability. One way of getting around this problem is by conducting a random experiment: a group of individuals or businesses are selected by random drawings and applied a policy, whilst another group constitutes the control group. The randomised selection from a sufficiently large population ensures that the control and treatment groups are comparable: policy access is not dependent upon individual characteristics.

20The experiment set out in Box 1 is a random experiment: the researchers offered free insurance to a large panel of Californian households. The participants were not free to choose the type of insurance policy offered to them. The policy, and particularly whether or not it provided full cover or required a personal contribution, was selected by random drawings.

21A random experiment may prove costly, although the accuracy of the results and the budgetary savings that they may enable to be made often make them a profitable investment. The cost and complexity of random experimentation are matched by the results that they are able to yield. Accordingly, the random experiment set out in Box 1, conducted in the 1970s, serves as a reference today even though behaviours have changed since that time.

22Moreover, random experiments often raise ethical issues in certain ca ses. [8] Certain fields are ill-suited to random experiments for reasons of fairness. For example, it would be inconceivable (and unconstitutional) to study the effect of a tax reform by subjecting various contributors at random to different taxes; others are unsuitable because they may place vulnerable subjects at risk.

23Although a purely random experiment can turn out to be difficult and sometimes costly, some substitutes may also yield very satisfactory results. At the point at which a public policy is decided upon, this consists in not implementing it in a single stage throughout the territory but staggering its implementation in a number of waves, for example by groups of départements (administrative areas). The départements of each wave then need to be chosen such that each is as closely comparable as possible to the others.

24This type of experimentation had been envisaged for the replacement of French Minimum Income (Revenu minimum d’insertion, RMI) by an Earned Income Supplement (Revenu de solidarité active, RSA). The measure was first applied in the departement of Eure, and then in 25, 34 and finally 40 départements before being fully rolled out across the nation. However, the experiment was not conducted with sufficient consistency and evaluation was not of the calibre that had been anticipated.

25Random experiments can yield very reliable results provided that they are prepared in advance, either by defining the test groups, or by sequential implementation of the policy. Where this is not the case, other evaluation methods must be envisaged.

Natural experimentation

26A “natural” [9] experiment consists in comparing groups of individuals (or businesses) which are found to be unintentionally separated in terms of access to the policy under consideration. Individuals excluded from benefiting from the public policy act as a “counterfactual” against the actual beneficiaries.

27The difficulty in evaluating a natural experiment is the validity of the “counterfactual” component, i.e. the comparability of the experimentation and control groups: an apparent similarity between the groups compared does not exclude the presence of bias in the evaluation. A number of econometric techniques have consequently been developed to ensure the comparability of the experimentation and control groups. Box 3 sets out some of these techniques.

The limits of random and natural experiments

28The techniques set out above are relatively recent and their ability to provide explanations may be relied upon. However, their predictive abilities have been disputed on the grounds that behaviours depend upon a constantly changing socioeconomic environment. Accordingly, behaviours may differ between the reaction to a small-scale experiment and the reaction to implementation of the actual policy, owing to the fact that the aforesaid policy more radically alters the economic context. For example, the random experiment of the RAND Corporation (Box 1) examines the effect of insurance on the consumption of care in a small number of subjects. If we consider a public insurance policy of greater scope such as Medicare, [10] we may observe a much greater increase in spending on care: the fact that the insurance encompasses more individuals and substantially increases the financial resources of the sector leads to a supply-side care reaction, the construction of new hospitals and an increase in medical research. Methods combining estimation in natural experimentation and more general models are currently being developed in order to correct this handicap. [11]

The need for reliable data

29To conduct evaluations, the availability of databases that are both exhaustive and reliable is imperative. Researchers rarely have the means to conduct surveys themselves. Fortunately, the required data already exists, mostly in various government databases. It is therefore important to set in place institutions and procedures that enable researchers to make use of this data whilst safeguarding the rights of individuals and businesses whose information is stored in these databases. Secure protocols already exist in France, such as Secure Data Access (SDA), requiring agreement for each study from the Statistics Confidentiality Commission (Commission du secret statistique). However, two lacunae are still acutely felt.

30The first is that much data remains inaccessible, in particular health insurance and tax data. Yet, this data is essential for a number of evaluations, for several reasons. Firstly, there are many tax policies and these have been little studied due to a lack of data. Access to tax data would enable great progress to be made with these types of policies. Also, because tax data is so rich, it can be deployed in the evaluation of non-tax policies. Therefore, evaluators need to be allowed to work with these databases whilst ensuring confidentiality for taxpayers. This would take the form of safeguards such as SDA and anonymisation of the databases, whilst keeping observations codified to ensure that it is possible to make up panels. This is technically straightforward and inexpensive.

31The second lacuna concerns the possibility of comparing government files or surveys. In effect, even if the information contained in the databases that are accessible to evaluators were to expand, it would not always be usable. To study the behaviour of mothers in terms of the labour supply, for example, information is needed on a woman’s children, which is found in one database, and on her participation in the employment market, which is found in another. If we are unable to match the databases, then the infor m ation contained in each database becomes useless. Simple, reliable and inexpensive ways exist of matching data whilst complying with data anonymity.

How can different public policies be compared?

32The evaluation of a public policy may lead to a clear-cut conclusion: the policy is inefficient as regards the stated objective, and is even counterproductive. However, often the assessment is more nuanced: the policy is effective but appears costly in relation to the results obtained. To conduct a comparison with other public policies, particularly those active in different public policy fields, the benefits must be converted into a metric that makes them comparable with both the costs and the benefits of other public policies. In practice, a monetary value must be ascribed to non-monetary benefits such as air quality, longevity or health.

33On the face of it, this may appear shocking, but it is the only means of making the criteria used for public decision-making explicit. These monetary values may be defined protectively, as has often been the case in the domain of road safety. However, it is preferable to seek to identify the preferences of individuals from surveys in which they express their willingness to pay for example for an improvement in water quality. [12]

Technical expertise and administrative expertise

35Although technical expertise is indispensable in order to avoid the evaluation pitfalls set out above, we could not do without administrative expertise on the practical implementation of policies and the operation of the public sector or governmental institutions that manage them. Administrative expertise not only enables an identifying strategy and counterfactual scenarios to be established, but also allows us to discern, within the results, what arises from the policy’s general principal and what owes itself to its practical implementation These two types of expertise – technical and administrative– must collaborate not only during the evaluation phase itself but equally and particularly upstream of it, if possible prior to implementation of the policy to be evaluated. This must enable us to adapt certain legal mechanisms to make it possible to evaluate it more accurately. It is also during this a priori coordination phase that the decision may be taken, depending on the legal options, to stagger implementation of the measure to construct a pertinent counterfactual component prior to evaluation. Finally this a priori coordination could enable failings in implementation to be avoided that would be liable to render ineffectual a public policy that is in principal beneficial.

Independence of evaluators

36No matter how thorough it may be, an evaluation remains subject to scientific uncertainty: the results are conditional upon the validity of the methods (choice of counterfactual component, generalisation of results, etc.). Yet, in order for the public policy evaluation to be useful, it is important that these results are credible: that the hypotheses are presented transparently, without leading one to suspect that some of them have been concealed. Transparency and credibility require the independence of the evaluators. The difficulty then lies in getting institutional and scientific partners to work together whilst safeguarding the independence of the evaluation. Obvious conflicts of interest exist where the evaluation is conducted by administrations, ministries, directorates or public institutions tasked with designing or applying a public policy. The same institution cannot be both judge and judged. This is not the only independence issue, however, and care must be taken not to create de facto dependency during the process of appointment of evaluators, or by blocking the publication of results.

37The evaluation timeframe is not the same as the policy timeframe. This divergence in the timeframe is brought about by a number of factors. Firstly, most evaluation methods require longitudinal data. We therefore need to wait until sufficient data has been compiled in order to ensure the validity of the evaluation. Moreover, the evaluation itself takes time, from the appointment of evaluators to the discussion of results, and also the choice of methodologies and the statistical work. In this field, haste is often the enemy of precision and exhaustiveness. In particular, hypotheses must be subjected to in-depth peer review. This an ex post validation, which is the truly scientific method, must not be neglected in the case of public policy evaluation. [13] Rushing an evaluation results in a reduction in its credibility. This makes public decision-making more difficult and potentially longer.

Dissemination of results

38The freedom to disseminate results is a key condition for the independence of the evaluators. Specifically, access to data must not be subject to monitoring rights by the administration providing the data. Any practice that runs counter to this freedom to disseminate, by placing pressure on the evaluator in terms of their results, would contravene the independence requirement for this latter, in addition to the fact that it would exclude any scientific debate on the method and results.

39The dissemination of results must be accompanied by a cross-comparison with other evaluations emerging, where applicable, from other disciplinary fields (see below). This cross-comparison must take the form both of the publication of the different results and their critiques, and also the organisation of debates, and even consensus conferences. This is to enable citizens to be better informed and to better understand the various effects of a public policy. To this end, methods for hierarchising the robustness of the results may be brought into play (scientifically established proof, scientific assumption, low level of proof).

Plurality of evaluators and interdisciplinarity

40Since a single public policy often has multiple effects, it is necessary to have at our disposal a number of evaluations corresponding to different approaches, disciplines or sensitivities. The French Competitiveness and Employment Tax Cred it (Crédit d’impôt compétitivité emploi, CICE, Law no 2012-1510 of 29 December 2012) serves as an example in this regard. When evaluating this measure, a number of relevant variables may be envisaged. Firstly, an impact study on the commercial position of France in relation to foreign countries would seem to be an obvious requirement, as well as an impact study on employment in quantitative terms. However, other evaluators might take an interest in other issues, such as the impact of the policy on the structure of qualifications, of careers within industry or on employment conditions. We might also evaluate the impact on the financing of social security from a more general standpoint, and on its acceptability.

41This example illustrates the importance of pluralism not only in the a posteriori evaluation, but also in its a priori preparation. The upstream coordination phase must enable relevant variables to be defined, changes in which we wish to measure in view of the policy under study, and enable each of the consequences of the measure to be evaluated. It is therefore important that this ex ante coordination be pluralist in terms of methods, disciplines and sensitivities. This ex ante pluralism is more complex to set in place than ex post pluralism, namely because this latter may be obtained through the juxtaposition of evaluations issuing from different expert groups. The initial coordination phase being unique, even more sustained attention must be given from this stage onwards to ensuring the existence of pluralism.

The evaluation triptych

42Contrary to other developed countries (see Box 4), France has little experience in public policy evaluation within the meaning defined in this Note. Sound evaluation should be based on the triptych form of coordinator, the administrations concerned and independent experts:

• coordination of the evaluation should be provided by an institution that is external to the executive. Democratic logic dictates that it should be Parliament that is tasked with commissioning these evaluations. This would mean vesting it with the technical ability, i.e. the personnel, to genuinely coordinate implementation of evaluation, its appraisal and its dissemination to members of parliament and the general public. The Cour des comptes is another candidate, with sufficient institutional weight to commission independent evaluations. [14] Whatever the choice of institutional commissioner, it must coordinate preparation of the evaluation, ensure compliance with a plurality of approaches, and check that all measures likely to facilitate evaluations have been taken (particularly data access). It must also organise discussion and the dissemination of results. Evaluatory bodies should be selected on the basis of public tenders;
• the administrations concerned should provide their institutional expertise. Collaboration with evaluators must be without pressure, under the oversight of the commissioning bodies. Ministerial statistical departments possess technical skills in the area of evaluation, in addition to institutional skills. They can therefore organise parallel evaluations to the independent evaluations and participate usefully in discussions on the results. However, they must facilitate execution of the independent evaluations, particularly by providing full and informed access to the data;
• experts must provide their scientific skills as evaluators. Their independence must be safeguarded amongst other measures, by rotation, so as to prevent any skewing based on the previous results of evaluations. Experts must be bound by statistical confidentiality constraints and be transparent as regards any ancillary activities that may cause conflicts of interest to arise. It is vitally important that they work together with other disciplines, particularly during the preparatory phases and those of results dissemination.

4

The evaluation triptych

The evaluation triptych