=======================
sampo_ps aggregate_eval
=======================

.. contents:: Contents
    :local:

Overview
========
.. warning::

   sampo_ps aggregate_eval command is deprecated.

**sampo_ps aggregate_eval** is used to view the evaluation results for the predictions of multiple processes/components at the same time. The aggregated results are output to a csv file.

|

Synopsis
========
See sampo_ps command help::

    $ sampo_ps aggregate_eval --help

|

Examples
========
The following examples are assumed to be executed with the ProcessStore shown below::

    /var/process_store_storage
    ├── my_process1
    │   ├── attr_metadata
    │   ├── components
    │   │   ├── dl
    │   │   ├── rg
    │   │   └── std
    │   ├── spd
    │   └── src
    ├── my_process2
    │   ├── attr_metadata
    │   ├── components
    │   │   ├── dl
    │   │   ├── rg2
    │   │   └── std
    │   ├── spd
    │   └── src
    └── my_process3
        ├── attr_metadata
        ├── components
        │   ├── cl
        │   ├── dl
        │   └── std
        ├── spd
        └── src

|

* Aggregating evaluations of a specific process and component.

  The evaluation result of the component **rg** in the process **my_process1** is output.

  * Command::

      $ sampo_ps aggregate_eval -l "my_process1/rg" -s file:///var/process_store_storage -o output_dir/

  * Output::

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,102

|

* Aggregating evaluations of multiple processes and arbitrary components.

  Shell-style wild cards are available. The evaluation results of all components in **my_process1** and **my_process2** are output.

  * Command::

      $ sampo_ps aggregate_eval -l "my_process[1-2]/*" -s file:///var/process_store_storage -o output_dir/

  * Output::

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,102
      my_process2,LEARN,0:00:01.290844,rg2,FABHMEBernGateLinearRgComponent,None,4.961700e-14,0.000000e+00,5.413659e-15,0.000000e+00,4.547474e-13,1.566328e-17,0.000000e+00,1.708955e-18,0.000000e+00,1.446397e-16,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.662566e-16,1.979245e-18,0.000000e+00,0.000000e+00,0,3.349495e-16,2.220446e-16,2.240271e-16,2.220446e-16,1.332268e-15,2.675211e-16,1.721786e-16,2.147418e-16,1.721418e-16,7.412143e-16,4.177831e-16,2.220446e-16,3.081563e-16,2.220446e-16,1.332268e-15,1.431702e-16,5.551115e-17,1.003078e-16,5.551115e-17,2.220446e-16,-3.699083e+00,-5.092890e-01,-6.165138e-01,-1.127294e-01,102

  .. note::

     A component which doesn't have an evaluation result (like **DataLoader** component) outputs nothing in *aggregated_eval.csv*

|

* Aggregating evaluations of all components in all processes in a ProcessStore.

  If you don't specify target_list (or specify **'\*/\*'**), the evaluation results of all components in a ProcessStore are output.

  * Command::

      $ sampo_ps aggregate_eval -s file:///var/process_store_storage -o output_dir/

  * Output::

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num,true_positive,false_positive,true_negative,false_negative,accuracy,classification_error,precision,recall,specificity,false_positive_rate,false_negative_rate,f_measure,auc,area_under_precision_recall
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,?
      my_process2,LEARN,0:00:01.290844,rg2,FABHMEBernGateLinearRgComponent,None,4.961700e-14,0.000000e+00,5.413659e-15,0.000000e+00,4.547474e-13,1.566328e-17,0.000000e+00,1.708955e-18,0.000000e+00,1.446397e-16,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.662566e-16,1.979245e-18,0.000000e+00,0.000000e+00,0.000000e+00,3.349495e-16,2.220446e-16,2.240271e-16,2.220446e-16,1.332268e-15,2.675211e-16,1.721786e-16,2.147418e-16,1.721418e-16,7.412143e-16,4.177831e-16,2.220446e-16,3.081563e-16,2.220446e-16,1.332268e-15,1.431702e-16,5.551115e-17,1.003078e-16,5.551115e-17,2.220446e-16,-3.699083e+00,-5.092890e-01,-6.165138e-01,-1.127294e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,?
      my_process3,LEARN,0:00:01.537659,cl,FABHMEBernGateLinearClComponent,None,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,1.090000e+02,7.000000e+00,4.100000e+01,1.100000e+01,8.928571e-01,1.071429e-01,9.396552e-01,9.083333e-01,8.541667e-01,1.458333e-01,9.166667e-02,9.237288e-01,9.439236e-01,9.808705e-01

  .. note::

     **The question marks ('?')** in the `aggregated_eval.csv` mean that the evaluation index (column) is not applied to the component (line). 

|

* Specifying multiple targets in the **target list** argument.

  You can specify multiple targets in the **target list** argument. In the following example, the evaluation results of all components named **rg** OR **cl** in all processes are output.

  * Command::

      $ sampo_ps aggregate_eval -l "*/rg, */cl" -s file:///var/process_store_storage -o output_dir/

  * Output::

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num,true_positive,false_positive,true_negative,false_negative,accuracy,classification_error,precision,recall,specificity,false_positive_rate,false_negative_rate,f_measure,auc,area_under_precision_recall
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,?
      my_process3,LEARN,0:00:01.537659,cl,FABHMEBernGateLinearClComponent,None,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,1.090000e+02,7.000000e+00,4.100000e+01,1.100000e+01,8.928571e-01,1.071429e-01,9.396552e-01,9.083333e-01,8.541667e-01,1.458333e-01,9.166667e-02,9.237288e-01,9.439236e-01,9.808705e-01

|

Output Format
=============
The aggregated evaluation file format is described below.

* The file name is *aggregated_eval.csv*
* The first six columns are the properties of each component.

 +-------------------------------------------+------------------------------------------------------------------------+
 |Property                                   |Description                                                             |
 +===========================================+========================================================================+
 |proc_name                                  |The name of the process in which the component is.                      |
 +-------------------------------------------+------------------------------------------------------------------------+
 |ptype                                      |Process type: LEARN or PREDICT                                          |
 +-------------------------------------------+------------------------------------------------------------------------+
 |proc_running_time                          |Running time of the process.                                            |
 +-------------------------------------------+------------------------------------------------------------------------+
 |component_id                               |The component ID.                                                       |
 +-------------------------------------------+------------------------------------------------------------------------+
 |ctype                                      |The class name of the component                                         |
 +-------------------------------------------+------------------------------------------------------------------------+
 |comp_running_time                          |Running time of the component.                                          |
 +-------------------------------------------+------------------------------------------------------------------------+

* The rest of the columns are evaluation indices. Each index has a component-specific format. See each component's specification.
