Detect which classes are difficult to predict

[Glavin001]

If you look at the scores you can see that certain classes are very difficult to predict and should be ignored.

Example

For example, take https://github.com/Glavin001/atom-beautify

Before

We correctly predicted 573 of 835 labelled issues, obtaining a score of 68.62275449101797%!

                               precision    recall  f1-score   support

               add-beautifier       0.97      1.00      0.99        37
        add-beautifier-option       1.00      0.93      0.97        30
                 add-language       1.00      1.00      1.00       144
                          bug       1.00      0.94      0.97       103
          deprecated-atom-api       0.93      0.74      0.82        19
                  enhancement       1.00      1.00      1.00       134
                      feature       1.00      1.00      1.00        58
                high priority       1.00      0.32      0.48        95
                  in-progress       1.00      0.36      0.53        11
                      invalid       1.00      0.56      0.72        66
                          mac       1.00      0.50      0.67         4
                    published       1.00      0.15      0.26       164
                     question       1.00      0.93      0.97       168
                   quick-todo       0.00      0.00      0.00        12
              third-party bug       0.96      0.99      0.98       139
update third-party dependency       1.00      0.31      0.47        13
             user-update-atom       0.00      0.00      0.00         4
 waiting-for-user-information       0.81      0.53      0.64        32
                      windows       0.93      1.00      0.96        25
                      wontfix       0.00      0.00      0.00         7

                  avg / total       0.97      0.75      0.80      1265

After

Ignore the following labels to achieve 97%

ignore_labels = ['quick-todo', 'wontfix', 'user-update-atom', 'in-progress', 'pending-publication', 'published', 'waiting-for-user-information', 'high priority']

                               precision    recall  f1-score   support

               add-beautifier       0.97      1.00      0.99        37
        add-beautifier-option       1.00      0.93      0.97        30
                 add-language       1.00      1.00      1.00       144
                          bug       1.00      0.97      0.99       103
          deprecated-atom-api       1.00      0.84      0.91        19
                  enhancement       1.00      1.00      1.00       134
                      feature       1.00      1.00      1.00        58
                      invalid       1.00      0.53      0.69        66
                          mac       1.00      0.50      0.67         4
                     question       1.00      0.95      0.98       168
              third-party bug       0.97      0.99      0.98       139
update third-party dependency       1.00      0.31      0.47        13
                      windows       0.93      1.00      0.96        25

                  avg / total       0.99      0.94      0.96       940

---

Resources:

• http://stats.stackexchange.com/questions/117654/what-does-the-numbers-in-the-classification-report-of-sklearn-mean
• http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html