The right tool for the job? A case study on the choice of model for policy analysis
Cambridge Econometrics and E3M have recently done modelling for the European Commission DG Energy exploring how policy conclusions differ if the same set of scenarios are estimated with two different models: E3ME and GEM-E3-FiT. This kind of exercise could be problematic. If the modelling results align, then why did we bother using different models? If they differ, then which should we believe?
This article argues for the importance of such exercises, as models and theories have different strengths and weaknesses representing the complexity of the economic system. Having different conclusions helps us make fully informed policy decisions, as long as the key mechanisms behind the differences are well understood.
What kind of model to use?
All macroeconomic models commonly used for policy analysis fall somewhere on the spectrum from purely theoretical abstractions to entirely data-driven empirical constructs. However, it is difficult to judge where exactly they fall. The two models used in this analysis are also different in this regard: GEM-E3-FiT is a theoretical Computable General Equilibrium (CGE) model with an optimization framework at heart, while E3ME has a data-driven macro-econometric core.
On the other hand, GEM-E3-FiT uses empirics to produce some of the model parameters, and the specifications of E3ME’s econometric equations have a robust theoretical underpinning. E3ME has also incorporated technology diffusion modules building on differential equations instead of empirical estimates to better capture technological change in power generation and other sectors (FTT models).
Models can also differ in their assumptions about equilibrium, the optimal allocation of resources in the economy. Pure CGE models assume that the economy moves towards achieving a balanced growth path. Solving for the long-term equilibrium by eliminating disequilibrium behaviours makes them potent tools for long-run projection. E3ME does not require the optimal use of resources on any timeframe, but accepts that inefficiencies and spare capacities can and do exist.
E3ME does not require the optimal use of resources on any timeframe, but accepts that inefficiencies and spare capacities can and do exist. Policies resolving those inefficiencies can boost growth.
Policies resolving those inefficiencies can boost growth. E3ME solves yearly, thus its long-run projections are path dependent and short-term shocks can have lasting impacts on variable trajectories. The model’s two-stage econometric framework enables both short and long-term modelling.
Accepting disequilibrium is a key strength of E3ME in modelling the labour market and capturing that not all unemployment is voluntary. Over the years, GEM-E3-Fit has deviated from pure CGE assumptions and incorporated disequilibrium behaviours as well, a representation of spare capacity in the labour market.
Supply and demand orientation can also be a defining feature of models. Traditionally, CGE models and GEM-E3-FiT are more supply oriented, as capacity constraints of resources are the key determinants of the output achievable in the long-run. E3ME is a more demand oriented model, building on input-output relationships and estimations of demand equations. However, E3ME has also incorporated price equations and maximum achievable capacities (normal output) as supply side constraints of demand.
There are many more contrasts that can be drawn between model features, such as general or partial models, top-down or bottom-up, stochastic or deterministic, endogenous or exogenous technology distinctions. However, classifying models based on different features may not increase our understanding. Models evolve over time. Large scale models which have been able to stay at the frontier for a long time, were able to do so by learning from each other. Incorporating new modules and building on each others’ strengths over time made them better, but made them far more difficult to categorise in simple terms. With all models becoming hybrids to some extent, the classifications are not as clear cut or informative anymore. Rather than rely on these misleadingly simple terms, we need a deeper understanding of the specific mechanisms of each model.
Large scale models which have been able to stay at the frontier for a long time, were able to do so by learning from each other.
How much does the model type matter?
The differences and similarities between E3ME and GEM-E3-Fit can be illustrated through the results of a modelling exercise done for the European Commission, DG Energy. In the project a series of scenarios were estimated with the two models, capturing how different trends in technological change, globalisation, demographics and resource use affect the economy of the EU and the low-carbon transition. The aim of the project was to gain a better understanding of the impacts of these ‘megatrends’ from two different angles. For more details see my previous blog on how global megatrends impact climate scenario modelling.
The figure above illustrates the difference in the results modelled by E3ME and GEM-E3-FiT for sixteen different scenarios with various assumptions. It shows the EU28 cumulative GDP impact by 2050 estimated with the two models. The chart reveals that, in general, the difference between the model results depends more on the scenario type than the choice of model. The divergence between the results of the two models also varies according to the scenario: in some cases, the models offer identical results, while in others they have opposite signs and over 10% difference in their predicted impacts.
For instance, in the case of scenarios T1-T5 E3ME always finds negative impacts, while GEM-E3 results are positive. These scenarios differ in the assumed pace of automation. In E3ME, job losses dominate the total impact. Increasing unemployment reduces average household incomes, which lowers consumption and results in a negative GDP impact. Conversely, in GEM-E3-FiT the productivity enhancement effect of automation outweighs the negative effect of job losses and the scenarios find positive GDP effects.
The broad model categories that these tools belong to do not give a clear picture of the dynamics of these mechanisms, or the particular strength of the different channels in each individual model. Today’s large-scale models are too complex for such simplification.
Today’s large-scale models are too complex for such simplification.
It is also difficult to tell a priori which is the better representation of automation impacts in the long-run. As both projections can be considered as best guesses based on reasonable assumptions and theory, the results can be thought to characterize a range within the actual impact could fall. This is admittedly a less convenient policy conclusion than what perfectly aligned results would offer. However, revealing both sides of the story is key for making a fully informed policy choice in a complex world.
This article builds on the modelling results of the project ‘European Commission, 2020 (forthcoming). Macroeconomics of the Energy Union: Policy Report for Modelling Megatrends’.
Cover image is by Jan Vašek.