Simulation output database: data_full

This section presents the format of the outputs from a given simulation

Storing the results of a simulation

Simulations are ran through using the following line in the Main.R file.

data_full <- run_simulations(configuration = config)

Executing this command line will thus run the configured simulation and store the results in a data.frame object called data_full in the global environment. In addition, the results will be stored on the hard drive in the data/output folder.

Reaggregation of sectors and commodities (ThreeME model specific option)

For simulations using the ThreeME model, and depending on the configuration option selected, 1 to 4 versions of the database may be created. These versions differ in their sector and commodity distribution.

The simulations will always run based on the classification specified in the config_input file under the classification argument. However, a reaggregation of the sectors and/or commodities can be specified, in which case it will be performed ex-post.

The bridge and codenames files

To control the reaggregation, open the file src/bridges_bridge_cXX_sXX.R where cXX_sXX is the classification specified in the classification argument of the config_input file.

Bridges are files that dictates how sectors and commodities may be combined. They are R scripts that define two R lists objects: bridge_sectors and bridge_commodities. They are named lists built with the following structure:

  • Groupings (the lists’ elements) are vectors of the sectors or commodities codes to be grouped
  • Each grouping of sectors has a name thats starts with S followed by three digits
  • Each grouping of commodities has a name thats starts with C followed by three digits
  • Each sector and commodity code of the classification must be cited in their correponding list.

Groupings can be given explicit labels by adding them to the corresponding codenames_cXX_sXX.R file in the same folder.

Below are examples of bridges and codenames files:

bridge_c17_s17.R 
bridge_c17_s17.R
bridge_sectors <- list(
  "S001" = c("sazx"),
  "S002" = c("sdex", "sc2x"),
  "S003" = c("sc1x", "sc3x", "sc4x", "sc5x"),
  "S004" = c("sfzx"),
  "S005" = c("sgzx", "shzx", "sizx", "sjzx", "skzx", "slzx", "smnx", "srux"),
  "S006" = c("soqx")
)

bridge_commodities <- list(
  "C001" = c("cazx"),
  "C002" = c("cdex", "cc2x"),
  "C003" = c("cc1x", "cc3x", "cc4x", "cc5x"),
  "C004" = c("cfzx"),
  "C005" = c("cgzx", "chzx", "cizx", "cjzx", "ckzx", "clzx", "cmnx", "crux"),
  "C006" = c("coqx")
)
codenames_c17_s17.R 
codenames_c17_s17.R
### Commodities: c17
names_commodities <- rbind(
c('Agriculture','cazx'),
c('Energie, eau, déchets','cdex'),
c('Industrie agroalimentaire','cc1x'),
c('Cokéfaction et raffinage','cc2x'),
c('Biens d\'équipement','cc3x'),
c('Matériels de transport','cc4x'),
c('Autres branches industrielles','cc5x'),
c('Construction','cfzx'),
c('Commerce','cgzx'),
c('Transport','chzx'),
c('Hébergement et restauration','cizx'),
c('Information et communication','cjzx'),
c('Services financiers','ckzx'),
c('Services immobiliers','clzx'),
c('Services aux entreprises','cmnx'),
c('Services non marchands','coqx'),
c('Services aux ménages','crux'),
c('Agriculture','C001'),
c('Energie (y.c. C2)','C002'),
c('Industrie manufacturière (hors C2)','C003'),
c('Construction','C004'),
c('Services marchands (hors construction)','C005'),
c('Services non marchands ','C006')
) %>%
  as.data.frame() %>% rename(name = V1,code = V2)

### Sectors: s17
names_sectors <- rbind(
c('Agriculture','sazx'),
c('Energie, eau, déchets','sdex'),
c('Industrie agroalimentaire','sc1x'),
c('Cokéfaction et raffinage','sc2x'),
c('Biens d\'équipement','sc3x'),
c('Matériels de transport','sc4x'),
c('Autres branches industrielles','sc5x'),
c('Construction','sfzx'),
c('Commerce','sgzx'),
c('Transport','shzx'),
c('Hébergement et restauration','sizx'),
c('Information et communication','sjzx'),
c('Services financiers','skzx'),
c('Services immobiliers','slzx'),
c('Services aux entreprises','smnx'),
c('Services non marchands','soqx'),
c('Services aux ménages','srux'),
c('Agriculture','S001'),
c('Energie (y.c. C2)','S002'),
c('Industrie manufacturière (hors C2)','S003'),
c('Construction','S004'),
c('Services marchands (hors construction)','S005'),
c('Services non marchands ','S006')
) %>%
  as.data.frame() %>% rename(name = V1,code = V2)

Producing the reaggregated data

In order to be able to produce reaggregated versions of the simulations output, the output_saved option must be correctly configured. Multiple options are possible:

  • output_saved = c() : Specifying an empty vector means no reaggregation will be performed
  • output_saved = c("com","sec","sec_com") : four types of databases will be stored : the normal disaggregated version, a version with only commodities reaggregated (com), a version with only sectors reaggregated (sec), a version with both sectors and commodities reaggregated (sec_com).
  • output_saved = c("sec_com") : two types of databases will be stored: the normal disaggregated version, and a version with both sectors and commodities reaggregated (sec_com).
  • output_saved = c("sec") : two types of databases will be stored: the normal disaggregated version, and a version with only sectors reaggregated (sec)
  • etc..

Each version of the simulation output will be stored in the data/output folder as .rds file named after the project_name argument from the config_input file and will bear the corresponding suffix :

  • _sec: sectoral reaggregation only
  • _com: commodity reaggregation only
  • _sec_com: sectoral and commodity reaggregation
  • no suffix: no reaggregation

Note that the reaggregated databases follow the same structure as the normal output database data_full. Unlike data_full, they are not loaded in the environment upon completion, only saved as .rds files.

The data_full structure

The structure of output simulations is standardised as a simple data.frame object with standardised column names in order to facilitate the usage of functions and templates on the outputs.

Below is a random sample of the data_full database created from the run_simulations() command.

data_full (random sample) 
bridge_c17_s17.R
slice_sample(.data = data_full,n = 10)
   variable year sector commodity   values_ref scenario       values index_scen
1         I 2021   <NA>      <NA> 5.867187e+05      ct1 5.860104e+05          0
2        PX 2045   <NA>      <NA> 1.673418e+00      ct1 1.705873e+00          0
3        PM 2043   <NA>      <NA> 1.608437e+00 baseline 1.608437e+00          1
4        PM 2027   <NA>      <NA> 1.171659e+00 baseline 1.171659e+00          1
5         I 2020   <NA>      <NA> 5.794610e+05 baseline 5.794610e+05          1
6       PCI 2046   <NA>      <NA> 1.706886e+00      ct1 1.742476e+00          0
7        LF 2021   <NA>      <NA> 2.961753e+04 baseline 2.961753e+04          1
8       PVA 2022   <NA>      <NA> 1.061208e+00 baseline 1.061208e+00          1
9         W 2044   <NA>      <NA> 9.294308e+01      ct1 9.468705e+01          0
10       PY 2032   <NA>      <NA> 1.293607e+00 baseline 1.293607e+00          1

By default the table is sorted by variable, year and scenario in this order.

data_full (default ordered sample) 
bridge_c17_s17.R
head(data_full)
  variable year sector commodity values_ref scenario   values index_scen
1      C_L 2019   <NA>      <NA>   55.09293 baseline 55.09293          1
2      C_L 2019   <NA>      <NA>   55.09293      ct1 55.09293          0
3      C_L 2020   <NA>      <NA>   56.75674 baseline 56.75674          1
4      C_L 2020   <NA>      <NA>   56.75674      ct1 56.75674          0
5      C_L 2021   <NA>      <NA>   58.47079 baseline 58.47079          1
6      C_L 2021   <NA>      <NA>   58.47079      ct1 58.45938          0

Data columns

data_full is made up of 8 columns:

  • variable: type [string]. This column contains the name of the variables. The variables names are always alphanumerical characters (including underscores) in UPPERCASE.
  • year: type [integer]. This column contains the year of the observation. It serves as the time series identifier, which does not necessarily have to be a calendar year. It could be identified by sequence of integer in any range (ie 1 to 10). The name year is a mainly a legacy feature.
  • sector: type [string]. This is a ThreeME specific item that can be used by any model using a similar list system to identify sectors. This is only a label variable. Its value is by default NA unless the variable is considered to be a sector specific variable, in which case the value reported will be either the sector code or the sector name if a codenames file has been properly configured (see Section 1.1.1) .
  • commodity: type [string]. This is a ThreeME specific item that can be used by any model using a similar list system to identify commodities. This is only a label variable. Its value is by default NA unless the variable is considered to be a commodity specific variable, in which case the value reported will be either the commodity code or the commodity name if a codenames file has been properly configured (see Section 1.1.1) .
  • scenario: type [string]. This is the (short) name of the scenario, as specified in the scenario argument of the config_input file.
  • values_ref: type [numeric]. This column contains the chosen reference scenario value. The reference scenario is the one specified in the scenario_baseline argument in the config_input.1
  • values: type [numeric]. This column contains simulated value for the observation. This is where the main outputs are found.
  • index_scen: type [dummy]. This variable takes the value 1 if the scenario is baseline or 0 otherwise. It’s main purpose is to quickly filter out the baseline scenario from the database. The information given for this variable is essentially redundant with the information from the scenario column.

Understanding the observations (rows of data_full)

data_full is organised as a long-form data.frame, that is all the model variables are listed in a single columns, as well as all the observations values.

The minimum observation identifier variables are : the variable, the year and the scenario

Switching from long to wide format (and back to long)

Examples of simple data manipulations

Most of the data manipulations can be made using the dplyr and the tidyverse packages.

Extracting a given scenario

A simple filter function

Reducing the dataset to a set of variables

For variables sets,

Selecting a chosen timespan

Computing values (relative) difference with the baseline

Composite variable computations

Making a simple ggplot graph

The main advantage of using a long format is that it makes the process of creating ggplot2 graphs straightforward

Making a simple gt table to display results

Notes de bas de page

  1. On an unmodified data_full from the simulations, for the baseline scenario the values in values and values_ref are equal.↩︎