sampo_ps aggregate_eval

Overview

Warning

sampo_ps aggregate_eval command is deprecated.

sampo_ps aggregate_eval is used to view the evaluation results for the predictions of multiple processes/components at the same time. The aggregated results are output to a csv file.


Synopsis

See sampo_ps command help:

$ sampo_ps aggregate_eval --help

Examples

The following examples are assumed to be executed with the ProcessStore shown below:

/var/process_store_storage
├── my_process1
│   ├── attr_metadata
│   ├── components
│   │   ├── dl
│   │   ├── rg
│   │   └── std
│   ├── spd
│   └── src
├── my_process2
│   ├── attr_metadata
│   ├── components
│   │   ├── dl
│   │   ├── rg2
│   │   └── std
│   ├── spd
│   └── src
└── my_process3
    ├── attr_metadata
    ├── components
    │   ├── cl
    │   ├── dl
    │   └── std
    ├── spd
    └── src

  • Aggregating evaluations of a specific process and component.

    The evaluation result of the component rg in the process my_process1 is output.

    • Command:

      $ sampo_ps aggregate_eval -l "my_process1/rg" -s file:///var/process_store_storage -o output_dir/
      
    • Output:

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,102
      

  • Aggregating evaluations of multiple processes and arbitrary components.

    Shell-style wild cards are available. The evaluation results of all components in my_process1 and my_process2 are output.

    • Command:

      $ sampo_ps aggregate_eval -l "my_process[1-2]/*" -s file:///var/process_store_storage -o output_dir/
      
    • Output:

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,102
      my_process2,LEARN,0:00:01.290844,rg2,FABHMEBernGateLinearRgComponent,None,4.961700e-14,0.000000e+00,5.413659e-15,0.000000e+00,4.547474e-13,1.566328e-17,0.000000e+00,1.708955e-18,0.000000e+00,1.446397e-16,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.662566e-16,1.979245e-18,0.000000e+00,0.000000e+00,0,3.349495e-16,2.220446e-16,2.240271e-16,2.220446e-16,1.332268e-15,2.675211e-16,1.721786e-16,2.147418e-16,1.721418e-16,7.412143e-16,4.177831e-16,2.220446e-16,3.081563e-16,2.220446e-16,1.332268e-15,1.431702e-16,5.551115e-17,1.003078e-16,5.551115e-17,2.220446e-16,-3.699083e+00,-5.092890e-01,-6.165138e-01,-1.127294e-01,102
      

    Note

    A component which doesn’t have an evaluation result (like DataLoader component) outputs nothing in aggregated_eval.csv


  • Aggregating evaluations of all components in all processes in a ProcessStore.

    If you don’t specify target_list (or specify ‘*/*’), the evaluation results of all components in a ProcessStore are output.

    • Command:

      $ sampo_ps aggregate_eval -s file:///var/process_store_storage -o output_dir/
      
    • Output:

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num,true_positive,false_positive,true_negative,false_negative,accuracy,classification_error,precision,recall,specificity,false_positive_rate,false_negative_rate,f_measure,auc,area_under_precision_recall
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,?
      my_process2,LEARN,0:00:01.290844,rg2,FABHMEBernGateLinearRgComponent,None,4.961700e-14,0.000000e+00,5.413659e-15,0.000000e+00,4.547474e-13,1.566328e-17,0.000000e+00,1.708955e-18,0.000000e+00,1.446397e-16,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.662566e-16,1.979245e-18,0.000000e+00,0.000000e+00,0.000000e+00,3.349495e-16,2.220446e-16,2.240271e-16,2.220446e-16,1.332268e-15,2.675211e-16,1.721786e-16,2.147418e-16,1.721418e-16,7.412143e-16,4.177831e-16,2.220446e-16,3.081563e-16,2.220446e-16,1.332268e-15,1.431702e-16,5.551115e-17,1.003078e-16,5.551115e-17,2.220446e-16,-3.699083e+00,-5.092890e-01,-6.165138e-01,-1.127294e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,?
      my_process3,LEARN,0:00:01.537659,cl,FABHMEBernGateLinearClComponent,None,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,1.090000e+02,7.000000e+00,4.100000e+01,1.100000e+01,8.928571e-01,1.071429e-01,9.396552e-01,9.083333e-01,8.541667e-01,1.458333e-01,9.166667e-02,9.237288e-01,9.439236e-01,9.808705e-01
      

    Note

    The question marks (‘?’) in the aggregated_eval.csv mean that the evaluation index (column) is not applied to the component (line).


  • Specifying multiple targets in the target list argument.

    You can specify multiple targets in the target list argument. In the following example, the evaluation results of all components named rg OR cl in all processes are output.

    • Command:

      $ sampo_ps aggregate_eval -l "*/rg, */cl" -s file:///var/process_store_storage -o output_dir/
      
    • Output:

      proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num,true_positive,false_positive,true_negative,false_negative,accuracy,classification_error,precision,recall,specificity,false_positive_rate,false_negative_rate,f_measure,auc,area_under_precision_recall
      my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,?
      my_process3,LEARN,0:00:01.537659,cl,FABHMEBernGateLinearClComponent,None,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,1.090000e+02,7.000000e+00,4.100000e+01,1.100000e+01,8.928571e-01,1.071429e-01,9.396552e-01,9.083333e-01,8.541667e-01,1.458333e-01,9.166667e-02,9.237288e-01,9.439236e-01,9.808705e-01
      

Output Format

The aggregated evaluation file format is described below.

  • The file name is aggregated_eval.csv

  • The first six columns are the properties of each component.

Property

Description

proc_name

The name of the process in which the component is.

ptype

Process type: LEARN or PREDICT

proc_running_time

Running time of the process.

component_id

The component ID.

ctype

The class name of the component

comp_running_time

Running time of the component.

  • The rest of the columns are evaluation indices. Each index has a component-specific format. See each component’s specification.