sampo_ps aggregate_eval¶
Contents
Overview¶
Warning
sampo_ps aggregate_eval command is deprecated.
sampo_ps aggregate_eval is used to view the evaluation results for the predictions of multiple processes/components at the same time. The aggregated results are output to a csv file.
Examples¶
The following examples are assumed to be executed with the ProcessStore shown below:
/var/process_store_storage
├── my_process1
│ ├── attr_metadata
│ ├── components
│ │ ├── dl
│ │ ├── rg
│ │ └── std
│ ├── spd
│ └── src
├── my_process2
│ ├── attr_metadata
│ ├── components
│ │ ├── dl
│ │ ├── rg2
│ │ └── std
│ ├── spd
│ └── src
└── my_process3
├── attr_metadata
├── components
│ ├── cl
│ ├── dl
│ └── std
├── spd
└── src
Aggregating evaluations of a specific process and component.
The evaluation result of the component rg in the process my_process1 is output.
Command:
$ sampo_ps aggregate_eval -l "my_process1/rg" -s file:///var/process_store_storage -o output_dir/
Output:
proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,102
Aggregating evaluations of multiple processes and arbitrary components.
Shell-style wild cards are available. The evaluation results of all components in my_process1 and my_process2 are output.
Command:
$ sampo_ps aggregate_eval -l "my_process[1-2]/*" -s file:///var/process_store_storage -o output_dir/
Output:
proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,102 my_process2,LEARN,0:00:01.290844,rg2,FABHMEBernGateLinearRgComponent,None,4.961700e-14,0.000000e+00,5.413659e-15,0.000000e+00,4.547474e-13,1.566328e-17,0.000000e+00,1.708955e-18,0.000000e+00,1.446397e-16,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.662566e-16,1.979245e-18,0.000000e+00,0.000000e+00,0,3.349495e-16,2.220446e-16,2.240271e-16,2.220446e-16,1.332268e-15,2.675211e-16,1.721786e-16,2.147418e-16,1.721418e-16,7.412143e-16,4.177831e-16,2.220446e-16,3.081563e-16,2.220446e-16,1.332268e-15,1.431702e-16,5.551115e-17,1.003078e-16,5.551115e-17,2.220446e-16,-3.699083e+00,-5.092890e-01,-6.165138e-01,-1.127294e-01,102
Note
A component which doesn’t have an evaluation result (like DataLoader component) outputs nothing in aggregated_eval.csv
Aggregating evaluations of all components in all processes in a ProcessStore.
If you don’t specify target_list (or specify ‘*/*’), the evaluation results of all components in a ProcessStore are output.
Command:
$ sampo_ps aggregate_eval -s file:///var/process_store_storage -o output_dir/
Output:
proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num,true_positive,false_positive,true_negative,false_negative,accuracy,classification_error,precision,recall,specificity,false_positive_rate,false_negative_rate,f_measure,auc,area_under_precision_recall my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,? my_process2,LEARN,0:00:01.290844,rg2,FABHMEBernGateLinearRgComponent,None,4.961700e-14,0.000000e+00,5.413659e-15,0.000000e+00,4.547474e-13,1.566328e-17,0.000000e+00,1.708955e-18,0.000000e+00,1.446397e-16,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,4.547474e-13,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.662566e-16,1.979245e-18,0.000000e+00,0.000000e+00,0.000000e+00,3.349495e-16,2.220446e-16,2.240271e-16,2.220446e-16,1.332268e-15,2.675211e-16,1.721786e-16,2.147418e-16,1.721418e-16,7.412143e-16,4.177831e-16,2.220446e-16,3.081563e-16,2.220446e-16,1.332268e-15,1.431702e-16,5.551115e-17,1.003078e-16,5.551115e-17,2.220446e-16,-3.699083e+00,-5.092890e-01,-6.165138e-01,-1.127294e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,? my_process3,LEARN,0:00:01.537659,cl,FABHMEBernGateLinearClComponent,None,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,1.090000e+02,7.000000e+00,4.100000e+01,1.100000e+01,8.928571e-01,1.071429e-01,9.396552e-01,9.083333e-01,8.541667e-01,1.458333e-01,9.166667e-02,9.237288e-01,9.439236e-01,9.808705e-01
Note
The question marks (‘?’) in the aggregated_eval.csv mean that the evaluation index (column) is not applied to the component (line).
Specifying multiple targets in the target list argument.
You can specify multiple targets in the target list argument. In the following example, the evaluation results of all components named rg OR cl in all processes are output.
Command:
$ sampo_ps aggregate_eval -l "*/rg, */cl" -s file:///var/process_store_storage -o output_dir/
Output:
proc_name,ptype,proc_running_time,component_id,ctype,comp_running_time,root_mean_squared_error,root_median_squared_error,mean_abs_error,median_abs_error,max_abs_error,relative_root_mean_squared_error,relative_root_median_squared_error,relative_mean_abs_error,relative_median_abs_error,relative_max_abs_error,positive_side_root_mean_squared_error,positive_side_root_median_squared_error,positive_side_mean_abs_error,positive_side_median_abs_error,positive_side_max_abs_error,negative_side_root_mean_squared_error,negative_side_root_median_squared_error,negative_side_mean_abs_error,negative_side_median_abs_error,negative_side_max_abs_error,max_upside_err_mean_obs,mean_upside_err_mean_obs,max_downside_err_mean_obs,mean_downside_err_mean_obs,negative_pred_num,std_root_mean_squared_error,std_root_median_squared_error,std_mean_abs_error,std_median_abs_error,std_max_abs_error,std_relative_root_mean_squared_error,std_relative_root_median_squared_error,std_relative_mean_abs_error,std_relative_median_abs_error,std_relative_max_abs_error,std_positive_side_root_mean_squared_error,std_positive_side_root_median_squared_error,std_positive_side_mean_abs_error,std_positive_side_median_abs_error,std_positive_side_max_abs_error,std_negative_side_root_mean_squared_error,std_negative_side_root_median_squared_error,std_negative_side_mean_abs_error,std_negative_side_median_abs_error,std_negative_side_max_abs_error,std_max_upside_err_mean_obs,std_mean_upside_err_mean_obs,std_max_downside_err_mean_obs,std_mean_downside_err_mean_obs,std_negative_pred_num,true_positive,false_positive,true_negative,false_negative,accuracy,classification_error,precision,recall,specificity,false_positive_rate,false_negative_rate,f_measure,auc,area_under_precision_recall my_process1,LEARN,0:00:01.095488,rg,FABHMEBernGateLinearRgComponent,None,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,?,?,?,?,?,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,0.000000e+00,1.831357e-16,2.220446e-16,1.493514e-16,2.220446e-16,3.330669e-16,7.911276e-16,2.053621e-16,3.625999e-16,2.053621e-16,4.420926e-15,1.909562e-16,1.665335e-16,1.788693e-16,1.665335e-16,2.775558e-16,1.815998e-16,2.220446e-16,1.436991e-16,2.220446e-16,3.330669e-16,-7.706422e-01,-7.981651e-02,-9.247706e-01,-3.348624e-01,1.020000e+02,?,?,?,?,?,?,?,?,?,?,?,?,?,? my_process3,LEARN,0:00:01.537659,cl,FABHMEBernGateLinearClComponent,None,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,1.090000e+02,7.000000e+00,4.100000e+01,1.100000e+01,8.928571e-01,1.071429e-01,9.396552e-01,9.083333e-01,8.541667e-01,1.458333e-01,9.166667e-02,9.237288e-01,9.439236e-01,9.808705e-01
Output Format¶
The aggregated evaluation file format is described below.
The file name is aggregated_eval.csv
The first six columns are the properties of each component.
Property
Description
proc_name
The name of the process in which the component is.
ptype
Process type: LEARN or PREDICT
proc_running_time
Running time of the process.
component_id
The component ID.
ctype
The class name of the component
comp_running_time
Running time of the component.
The rest of the columns are evaluation indices. Each index has a component-specific format. See each component’s specification.