Required headers¶
In [1]:
#include "TRInterface.h"
#include "TMVA/MethodC50.h"
#include "TMVA/MethodRSNNS.h"
#include "TMVA/MethodRXGB.h"
Declare Factory¶
In [2]:
TMVA::Tools::Instance();
auto inputFile = TFile::Open("https://raw.githubusercontent.com/iml-wg/tmvatutorials/master/inputdata.root");
auto outputFile = TFile::Open("TMVAOutputCV.root", "RECREATE");
TMVA::Factory factory("TMVAClassification", outputFile,
"!V:ROC:!Correlations:!Silent:Color:!DrawProgressBar:AnalysisType=Classification" );
--- Factory : You are running ROOT Version: 6.07/07, Apr 1, 2016 --- Factory : --- Factory : _/_/_/_/_/ _| _| _| _| _|_| --- Factory : _/ _|_| _|_| _| _| _| _| --- Factory : _/ _| _| _| _| _| _|_|_|_| --- Factory : _/ _| _| _| _| _| _| --- Factory : _/ _| _| _| _| _| --- Factory : --- Factory : ___________TMVA Version 4.2.1, Feb 5, 2015 --- Factory :
Declare DataLoader¶
In [3]:
TMVA::DataLoader loader("dataset");
//adding variables to dataset
loader.AddVariable("var1");
loader.AddVariable("var2");
loader.AddVariable("var3");
loader.AddVariable("var4");
Setting up Dataset¶
In [4]:
TTree *tsignal, *tbackground;
inputFile->GetObject("Sig", tsignal);
inputFile->GetObject("Bkg", tbackground);
TCut mycuts, mycutb;
loader.AddSignalTree (tsignal, 1); //signal weight = 1
loader.AddBackgroundTree (tbackground, 1); //background weight = 1
loader.PrepareTrainingAndTestTree(mycuts, mycutb,
"nTrain_Signal=1000:nTrain_Background=1000:SplitMode=Random:NormMode=NumEvents:!V");
--- DataSetInfo : Dataset[dataset] : Added class "Signal" with internal class number 0 --- dataset : Add Tree Sig of type Signal with 6000 events --- DataSetInfo : Dataset[dataset] : Added class "Background" with internal class number 1 --- dataset : Add Tree Bkg of type Background with 6000 events --- dataset : Preparing trees for training and testing...
Booking methods¶
The available Booking methods with options for RMVA are:
- C50 Boosted Decision Trees http://oproject.org/tiki-index.php?page=RMVA#C50Booking
- RMLP Neural Networks http://oproject.org/tiki-index.php?page=RMVA#RSNNSMLP
- Extreme Gradient Boosted (RXGB) Decision Trees http://oproject.org/tiki-index.php?page=RMVA#RXGBBooking
In [5]:
//C50 Boosted Decision Trees (BDTs)
factory.BookMethod(&loader, TMVA::Types::kC50, "C50",
"!H:NTrials=5:Rules=kTRUE:ControlSubSet=kFALSE:ControlBands=10:ControlWinnow=kFALSE:ControlNoGlobalPruning=kTRUE:ControlCF=0.25:ControlMinCases=2:ControlFuzzyThreshold=kTRUE:ControlSample=0:ControlEarlyStopping=kTRUE:!V" );
//Neural Networks using RSNNS package
factory.BookMethod(&loader, TMVA::Types::kRSNNS, "RMLP",
"!H:VarTransform=N:Size=c(5):Maxit=10:InitFunc=Randomize_Weights:LearnFunc=Std_Backpropagation:LearnFuncParams=c(0.2,0):!V" );
//eXtreme Gradient Boosted XGB Decision Trees
factory.BookMethod(&loader, TMVA::Types::kRXGB, "RXGB","!V:NRounds=20:MaxDepth=2:Eta=1" );
//TMVA BDTs
factory.BookMethod(&loader,TMVA::Types::kBDT, "BDT",
"!V:NTrees=50:MinNodeSize=2.5%:MaxDepth=2:BoostType=AdaBoost:AdaBoostBeta=0.5:UseBaggedBoost:BaggedSampleFraction=0.5:SeparationType=GiniIndex:nCuts=20" );
--- Factory : Booking method: C50 DataSet Name: dataset --- DataSetFactory : Dataset[dataset] : Splitmode is: "RANDOM" the mixmode is: "SAMEASSPLITMODE" --- DataSetFactory : Dataset[dataset] : Create training and testing trees -- looping over class "Signal" ... --- DataSetFactory : Dataset[dataset] : Weight expression for class 'Signal': "" --- DataSetFactory : Dataset[dataset] : Create training and testing trees -- looping over class "Background" ... --- DataSetFactory : Dataset[dataset] : Weight expression for class 'Background': "" --- DataSetFactory : Dataset[dataset] : Number of events in input trees (after possible flattening of arrays): --- DataSetFactory : Dataset[dataset] : Signal -- number of events : 6000 / sum of weights: 6000 --- DataSetFactory : Dataset[dataset] : Background -- number of events : 6000 / sum of weights: 6000 --- DataSetFactory : Dataset[dataset] : Signal tree -- total number of entries: 6000 --- DataSetFactory : Dataset[dataset] : Background tree -- total number of entries: 6000 --- DataSetFactory : Dataset[dataset] : Preselection: (will NOT affect number of requested training and testing events) --- DataSetFactory : Dataset[dataset] : No preselection cuts applied on event classes --- DataSetFactory : Dataset[dataset] : Weight renormalisation mode: "NumEvents": renormalises all event classes --- DataSetFactory : Dataset[dataset] : such that the effective (weighted) number of events in each class equals the respective --- DataSetFactory : Dataset[dataset] : number of events (entries) that you demanded in PrepareTrainingAndTestTree("","nTrain_Signal=.. ) --- DataSetFactory : Dataset[dataset] : ... i.e. such that Sum[i=1..N_j]{w_i} = N_j, j=0,1,2... --- DataSetFactory : Dataset[dataset] : ... (note that N_j is the sum of TRAINING events (nTrain_j...with j=Signal,Background.. --- DataSetFactory : Dataset[dataset] : ..... Testing events are not renormalised nor included in the renormalisation factor! ) --- DataSetFactory : Dataset[dataset] : --> Rescale Signal event weights by factor: 1 --- DataSetFactory : Dataset[dataset] : --> Rescale Background event weights by factor: 1 --- DataSetFactory : Dataset[dataset] : Number of training and testing events after rescaling: --- DataSetFactory : Dataset[dataset] : --------------------------------------------------------------------------- --- DataSetFactory : Dataset[dataset] : Signal -- training events : 1000 (sum of weights: 1000) - requested were 1000 events --- DataSetFactory : Dataset[dataset] : Signal -- testing events : 5000 (sum of weights: 5000) - requested were 0 events --- DataSetFactory : Dataset[dataset] : Signal -- training and testing events: 6000 (sum of weights: 6000) --- DataSetFactory : Dataset[dataset] : Background -- training events : 1000 (sum of weights: 1000) - requested were 1000 events --- DataSetFactory : Dataset[dataset] : Background -- testing events : 5000 (sum of weights: 5000) - requested were 0 events --- DataSetFactory : Dataset[dataset] : Background -- training and testing events: 6000 (sum of weights: 6000) --- DataSetFactory : Dataset[dataset] : Create internal training tree --- DataSetFactory : Dataset[dataset] : Create internal testing tree --- DataSetInfo : Dataset[dataset] : Correlation matrix (Signal): --- DataSetInfo : ---------------------------------------- --- DataSetInfo : var1 var2 var3 var4 --- DataSetInfo : var1: +1.000 +0.386 +0.597 +0.808 --- DataSetInfo : var2: +0.386 +1.000 +0.696 +0.743 --- DataSetInfo : var3: +0.597 +0.696 +1.000 +0.860 --- DataSetInfo : var4: +0.808 +0.743 +0.860 +1.000 --- DataSetInfo : ---------------------------------------- --- DataSetInfo : Dataset[dataset] : Correlation matrix (Background): --- DataSetInfo : ---------------------------------------- --- DataSetInfo : var1 var2 var3 var4 --- DataSetInfo : var1: +1.000 +0.856 +0.914 +0.964 --- DataSetInfo : var2: +0.856 +1.000 +0.927 +0.937 --- DataSetInfo : var3: +0.914 +0.927 +1.000 +0.971 --- DataSetInfo : var4: +0.964 +0.937 +0.971 +1.000 --- DataSetInfo : ---------------------------------------- --- DataSetFactory : Dataset[dataset] : --- Factory : Booking method: RMLP DataSet Name: dataset --- RMLP : Dataset[dataset] : Create Transformation "N" with events from all classes. --- Norm : Transformation, Variable selection : --- Norm : Input : variable 'var1' (index=0). <---> Output : variable 'var1' (index=0). --- Norm : Input : variable 'var2' (index=1). <---> Output : variable 'var2' (index=1). --- Norm : Input : variable 'var3' (index=2). <---> Output : variable 'var3' (index=2). --- Norm : Input : variable 'var4' (index=3). <---> Output : variable 'var4' (index=3). --- Factory : Booking method: RXGB DataSet Name: dataset --- Factory : Booking method: BDT DataSet Name: dataset
Training the Methods¶
In [6]:
factory.TrainAllMethods();
--- Factory : --- Factory : Train all methods for Classification ... --- Factory : --- Factory : current transformation string: 'I' --- Factory : Dataset[dataset] : Create Transformation "I" with events from all classes. --- Id : Transformation, Variable selection : --- Id : Input : variable 'var1' (index=0). <---> Output : variable 'var1' (index=0). --- Id : Input : variable 'var2' (index=1). <---> Output : variable 'var2' (index=1). --- Id : Input : variable 'var3' (index=2). <---> Output : variable 'var3' (index=2). --- Id : Input : variable 'var4' (index=3). <---> Output : variable 'var4' (index=3). --- Id : Preparing the Identity transformation... --- TFHandler_Factory : ----------------------------------------------------------- --- TFHandler_Factory : Variable Mean RMS [ Min Max ] --- TFHandler_Factory : ----------------------------------------------------------- --- TFHandler_Factory : var1: 0.00077102 1.6695 [ -5.8991 4.7639 ] --- TFHandler_Factory : var2: -0.0063164 1.5765 [ -5.2454 4.8300 ] --- TFHandler_Factory : var3: -0.010870 1.7365 [ -5.3563 4.6430 ] --- TFHandler_Factory : var4: 0.14557 2.1608 [ -6.9675 5.0307 ] --- TFHandler_Factory : ----------------------------------------------------------- --- TFHandler_Factory : Plot event variables for Id --- TFHandler_Factory : Create scatter and profile plots in target-file directory: --- TFHandler_Factory : TMVAOutputCV.root:/dataset/InputVariables_Id/CorrelationPlots --- TFHandler_Factory : --- TFHandler_Factory : Ranking input variables (method unspecific)... --- IdTransformation : Ranking result (top variable is best ranked) --- IdTransformation : ----------------------------- --- IdTransformation : Rank : Variable : Separation --- IdTransformation : ----------------------------- --- IdTransformation : 1 : var4 : 3.458e-01 --- IdTransformation : 2 : var3 : 2.817e-01 --- IdTransformation : 3 : var1 : 2.640e-01 --- IdTransformation : 4 : var2 : 2.173e-01 --- IdTransformation : ----------------------------- --- Factory : Train method: C50 for Classification --- C50 : Dataset[dataset] : Begin training --- C50 : --- C50 : --- Saving State File In:weights/C50Model.RData --- C50 : --- C50 : Dataset[dataset] : End of training --- C50 : Dataset[dataset] : Elapsed time for training with 2000 events: 0.873 sec --- C50 : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample --- C50 : Dataset[dataset] : Evaluation of C50 on training sample (2000 events) --- C50 : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.0748 sec --- C50 : Dataset[dataset] : Creating weight file in xml format: weights/TMVAClassification_C50.weights.xml --- Factory : Training finished --- Factory : Train method: RMLP for Classification --- Norm : Preparing the transformation. --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : Variable Mean RMS [ Min Max ] --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : var1: 0.085752 0.35735 [ -1.0000 1.0000 ] --- TFHandler_RMLP : var2: 0.10321 0.36589 [ -1.0000 1.0000 ] --- TFHandler_RMLP : var3: 0.10411 0.37276 [ -1.0000 1.0000 ] --- TFHandler_RMLP : var4: 0.17623 0.40650 [ -1.0000 1.0000 ] --- TFHandler_RMLP : ----------------------------------------------------------- --- RMLP : Dataset[dataset] : Begin training --- RMLP : --- RMLP : --- Saving State File In:weights/RMLPModel.RData --- RMLP : --- RMLP : Dataset[dataset] : End of training --- RMLP : Dataset[dataset] : Elapsed time for training with 2000 events: 1.07 sec --- RMLP : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample --- RMLP : Dataset[dataset] : Evaluation of RMLP on training sample (2000 events) --- RMLP : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.739 sec --- RMLP : Dataset[dataset] : Creating weight file in xml format: weights/TMVAClassification_RMLP.weights.xml --- RMLP : Dataset[dataset] : Creating standalone response class: weights/TMVAClassification_RMLP.class.C --- Factory : Training finished --- Factory : Train method: RXGB for Classification --- RXGB : Dataset[dataset] : Begin training [0] train-rmse:0.393636 [1] train-rmse:0.383795 [2] train-rmse:0.371803 [3] train-rmse:0.362196 [4] train-rmse:0.357333 [5] train-rmse:0.351344 [6] train-rmse:0.346084 [7] train-rmse:0.340683 [8] train-rmse:0.332887 [9] train-rmse:0.328419 [10] train-rmse:0.326714 [11] train-rmse:0.324244 [12] train-rmse:0.321295 [13] train-rmse:0.318570 [14] train-rmse:0.315405 [15] train-rmse:0.312603 [16] train-rmse:0.310186 [17] train-rmse:0.309424 [18] train-rmse:0.306099 [19] train-rmse:0.304363 --- RXGB : --- RXGB : --- Saving State File In:weights/RXGBModel.RData --- RXGB : --- RXGB : Dataset[dataset] : End of training --- RXGB : Dataset[dataset] : Elapsed time for training with 2000 events: 0.234 sec --- RXGB : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample --- RXGB : Dataset[dataset] : Evaluation of RXGB on training sample (2000 events) --- RXGB : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.00485 sec --- RXGB : Dataset[dataset] : Creating weight file in xml format: weights/TMVAClassification_RXGB.weights.xml --- Factory : Training finished --- Factory : Train method: BDT for Classification --- BDT : Dataset[dataset] : Begin training --- BDT : found and suggest the following possible pre-selection cuts --- BDT : as option DoPreselection was not used, these cuts however will not be performed, but the training will see the full sample --- BDT : found cut: Bkg if var 0 < -2.99038 --- BDT : found cut: Bkg if var 2 < -2.88493 --- BDT : found cut: Bkg if var 3 < -2.54088 --- BDT : <InitEventSample> For classification trees, --- BDT : the effective number of backgrounds is scaled to match --- BDT : the signal. Othersise the first boosting step would do 'just that'! --- BDT : re-normlise events such that Sig and Bkg have respective sum of weights = 1 --- BDT : sig->sig*1ev. bkg->bkg*1ev. --- BDT : #events: (reweighted) sig: 1000 bkg: 1000 --- BDT : #events: (unweighted) sig: 1000 bkg: 1000 --- BDT : Training 50 Decision Trees ... patience please --- BinaryTree : The minimal node size MinNodeSize=2.5 fMinNodeSize=2.5% is translated to an actual number of events = 25.7 for the training sample size of 1028 --- BinaryTree : Note: This number will be taken as absolute minimum in the node, --- BinaryTree : in terms of 'weighted events' and unweighted ones !! --- BDT : <Train> elapsed time: 0.0508 sec --- BDT : <Train> average number of nodes (w/o pruning) : 4 --- BDT : Dataset[dataset] : End of training --- BDT : Dataset[dataset] : Elapsed time for training with 2000 events: 0.061 sec --- BDT : Dataset[dataset] : Create MVA output for Dataset[dataset] : classification on training sample --- BDT : Dataset[dataset] : Evaluation of BDT on training sample (2000 events) --- BDT : Dataset[dataset] : Elapsed time for evaluation of 2000 events: 0.00845 sec --- BDT : Dataset[dataset] : Creating weight file in xml format: dataset/weights/TMVAClassification_BDT.weights.xml --- BDT : Dataset[dataset] : Creating standalone response class: dataset/weights/TMVAClassification_BDT.class.C --- BDT : Write monitoring histograms to file: TMVAOutputCV.root:/dataset/Method_BDT/BDT --- Factory : Training finished --- Factory : --- Factory : Ranking input variables (method specific)... --- Factory : No variable ranking supplied by classifier: C50 --- Factory : No variable ranking supplied by classifier: RMLP --- Factory : No variable ranking supplied by classifier: RXGB --- BDT : Ranking result (top variable is best ranked) --- BDT : -------------------------------------- --- BDT : Rank : Variable : Variable Importance --- BDT : -------------------------------------- --- BDT : 1 : var4 : 4.608e-01 --- BDT : 2 : var1 : 2.923e-01 --- BDT : 3 : var2 : 1.457e-01 --- BDT : 4 : var3 : 1.012e-01 --- BDT : -------------------------------------- --- Factory : --- Factory : === Destroy and recreate all methods via weight files for testing === --- Factory : --- MethodBase : Dataset[dataset] : Reading weight file: weights/TMVAClassification_C50.weights.xml --- C50 : Dataset[dataset] : Read method "C50" of type "C50" --- C50 : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1 --- C50 : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07 --- MethodBase : Dataset[dataset] : Reading weight file: weights/TMVAClassification_RMLP.weights.xml --- RMLP : Dataset[dataset] : Read method "RMLP" of type "RSNNS" --- RMLP : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1 --- RMLP : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07 --- MethodBase : Dataset[dataset] : Reading weight file: weights/TMVAClassification_RXGB.weights.xml --- RXGB : Dataset[dataset] : Read method "RXGB" of type "RXGB" --- RXGB : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1 --- RXGB : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07 --- MethodBase : Dataset[dataset] : Reading weight file: dataset/weights/TMVAClassification_BDT.weights.xml --- BDT : Dataset[dataset] : Read method "BDT" of type "BDT" --- BDT : Dataset[dataset] : MVA method was trained with TMVA Version: 4.2.1 --- BDT : Dataset[dataset] : MVA method was trained with ROOT Version: 6.07/07
Testing and Evaluating the data¶
In [7]:
factory.TestAllMethods();
factory.EvaluateAllMethods();
--- Factory : Test all methods... --- Factory : Test method: C50 for Classification performance --- C50 : Dataset[dataset] : Evaluation of C50 on testing sample (10000 events) --- C50 : --- C50 : --- Loading State File From:weights/C50Model.RData --- C50 : --- C50 : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 0.546 sec --- Factory : Test method: RMLP for Classification performance --- RMLP : Dataset[dataset] : Evaluation of RMLP on testing sample (10000 events) --- RMLP : --- RMLP : --- Loading State File From:weights/RMLPModel.RData --- RMLP : --- RMLP : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 3.49 sec --- Factory : Test method: RXGB for Classification performance --- RXGB : Dataset[dataset] : Evaluation of RXGB on testing sample (10000 events) --- RXGB : --- RXGB : --- Loading State File From:weights/RXGBModel.RData --- RXGB : --- RXGB : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 0.0286 sec --- Factory : Test method: BDT for Classification performance --- BDT : Dataset[dataset] : Evaluation of BDT on testing sample (10000 events) --- BDT : Dataset[dataset] : Elapsed time for evaluation of 10000 events: 0.0235 sec --- Factory : Evaluate all methods... --- Factory : Evaluate classifier: C50 --- C50 : Testing Classification C50 METHOD --- C50 : Dataset[dataset] : Loop over test events and fill histograms with classifier response... --- Factory : Write evaluation histograms to file --- TFHandler_C50 : Plot event variables for C50 --- TFHandler_C50 : ----------------------------------------------------------- --- TFHandler_C50 : Variable Mean RMS [ Min Max ] --- TFHandler_C50 : ----------------------------------------------------------- --- TFHandler_C50 : var1: 0.00077102 1.6695 [ -5.8991 4.7639 ] --- TFHandler_C50 : var2: -0.0063164 1.5765 [ -5.2454 4.8300 ] --- TFHandler_C50 : var3: -0.010870 1.7365 [ -5.3563 4.6430 ] --- TFHandler_C50 : var4: 0.14557 2.1608 [ -6.9675 5.0307 ] --- TFHandler_C50 : ----------------------------------------------------------- --- TFHandler_C50 : Create scatter and profile plots in target-file directory: --- TFHandler_C50 : TMVAOutputCV.root:/dataset/Method_C50/C50/CorrelationPlots --- Factory : Evaluate classifier: RMLP --- RMLP : Testing Classification RMLP METHOD --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : Variable Mean RMS [ Min Max ] --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : var1: 0.066774 0.35913 [ -1.2024 1.0914 ] --- TFHandler_RMLP : var2: 0.079492 0.36669 [ -1.1391 1.2044 ] --- TFHandler_RMLP : var3: 0.079125 0.37282 [ -1.0685 1.0783 ] --- TFHandler_RMLP : var4: 0.15120 0.40805 [ -1.1921 1.0737 ] --- TFHandler_RMLP : ----------------------------------------------------------- --- RMLP : Dataset[dataset] : Loop over test events and fill histograms with classifier response... --- Factory : Write evaluation histograms to file --- TFHandler_RMLP : Plot event variables for RMLP --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : Variable Mean RMS [ Min Max ] --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : var1: 0.066774 0.35913 [ -1.2024 1.0914 ] --- TFHandler_RMLP : var2: 0.079492 0.36669 [ -1.1391 1.2044 ] --- TFHandler_RMLP : var3: 0.079125 0.37282 [ -1.0685 1.0783 ] --- TFHandler_RMLP : var4: 0.15120 0.40805 [ -1.1921 1.0737 ] --- TFHandler_RMLP : ----------------------------------------------------------- --- TFHandler_RMLP : Create scatter and profile plots in target-file directory: --- TFHandler_RMLP : TMVAOutputCV.root:/dataset/Method_RMLP/RMLP/CorrelationPlots --- Factory : Evaluate classifier: RXGB --- RXGB : Testing Classification RXGB METHOD --- RXGB : Dataset[dataset] : Loop over test events and fill histograms with classifier response... --- Factory : Write evaluation histograms to file --- TFHandler_RXGB : Plot event variables for RXGB --- TFHandler_RXGB : ----------------------------------------------------------- --- TFHandler_RXGB : Variable Mean RMS [ Min Max ] --- TFHandler_RXGB : ----------------------------------------------------------- --- TFHandler_RXGB : var1: 0.00077102 1.6695 [ -5.8991 4.7639 ] --- TFHandler_RXGB : var2: -0.0063164 1.5765 [ -5.2454 4.8300 ] --- TFHandler_RXGB : var3: -0.010870 1.7365 [ -5.3563 4.6430 ] --- TFHandler_RXGB : var4: 0.14557 2.1608 [ -6.9675 5.0307 ] --- TFHandler_RXGB : ----------------------------------------------------------- --- TFHandler_RXGB : Create scatter and profile plots in target-file directory: --- TFHandler_RXGB : TMVAOutputCV.root:/dataset/Method_RXGB/RXGB/CorrelationPlots --- Factory : Evaluate classifier: BDT --- BDT : Dataset[dataset] : Loop over test events and fill histograms with classifier response... --- Factory : Write evaluation histograms to file --- TFHandler_BDT : Plot event variables for BDT --- TFHandler_BDT : ----------------------------------------------------------- --- TFHandler_BDT : Variable Mean RMS [ Min Max ] --- TFHandler_BDT : ----------------------------------------------------------- --- TFHandler_BDT : var1: 0.00077102 1.6695 [ -5.8991 4.7639 ] --- TFHandler_BDT : var2: -0.0063164 1.5765 [ -5.2454 4.8300 ] --- TFHandler_BDT : var3: -0.010870 1.7365 [ -5.3563 4.6430 ] --- TFHandler_BDT : var4: 0.14557 2.1608 [ -6.9675 5.0307 ] --- TFHandler_BDT : ----------------------------------------------------------- --- TFHandler_BDT : Create scatter and profile plots in target-file directory: --- TFHandler_BDT : TMVAOutputCV.root:/dataset/Method_BDT/BDT/CorrelationPlots --- Factory : --- Factory : Evaluation results ranked by best signal efficiency and purity (area) --- Factory : ------------------------------------------------------------------------------------------------------------------- --- Factory : DataSet MVA Signal efficiency at bkg eff.(error): | Sepa- Signifi- --- Factory : Name: Method: @B=0.01 @B=0.10 @B=0.30 ROC-integ ROCCurve| ration: cance: --- Factory : ------------------------------------------------------------------------------------------------------------------- --- Factory : dataset RMLP : 0.343(06) 0.764(06) 0.954(02) 0.930 0.929 | 0.578 1.647 --- Factory : dataset RXGB : 0.208(05) 0.693(06) 0.921(03) 0.901 0.902 | 0.507 1.387 --- Factory : dataset BDT : 0.263(06) 0.642(06) 0.900(04) 0.894 0.894 | 0.483 1.248 --- Factory : dataset C50 : 0.000(00) 0.689(06) 0.926(03) 0.892 0.898 | 0.521 1.411 --- Factory : ------------------------------------------------------------------------------------------------------------------- --- Factory : --- Factory : Testing efficiency compared to training efficiency (overtraining check) --- Factory : ------------------------------------------------------------------------------------------------------------------- --- Factory : DataSet MVA Signal efficiency: from test sample (from training sample) --- Factory : Name: Method: @B=0.01 @B=0.10 @B=0.30 --- Factory : ------------------------------------------------------------------------------------------------------------------- --- Factory : dataset RMLP : 0.343 (0.365) 0.764 (0.784) 0.954 (0.955) --- Factory : dataset RXGB : 0.208 (0.375) 0.693 (0.802) 0.921 (0.944) --- Factory : dataset BDT : 0.263 (0.225) 0.642 (0.671) 0.900 (0.902) --- Factory : dataset C50 : 0.000 (0.474) 0.689 (0.848) 0.926 (0.943) --- Factory : ------------------------------------------------------------------------------------------------------------------- --- Factory : --- Dataset:dataset : Dataset[dataset] : Created tree 'TestTree' with 10000 events --- Dataset:dataset : Dataset[dataset] : Created tree 'TrainTree' with 2000 events --- Factory : --- Factory : Thank you for using TMVA! --- Factory : For citation information, please visit: http://tmva.sf.net/citeTMVA.html
Ploting ROC Curve¶
We enable the ROOT JavaScript interactive visualisation.
In [8]:
%jsroot on
auto c = factory.GetROCCurve(&loader);
c->Draw();