Compute TPR0 and TPR10

[milesxu]

@linan7788626

Please search for code which compute TPR0 and TPR10

[linan7788626]

Hi @milesxu

Since the code for calculating TPR0 and TPR10 is short, I paste it below instead of making a pull request. Should you have any questions, please feel free to let me know.

Cheers,
Nan

import numpy as np
import pylab as pl

data = np.load('./tpr_result.npz', 'rb')

def eval_outputs(p_in, labels_in):
    '''
    calculate FPR and TPR according to the 
    output probability and original labels.
        - <p_in> is of the outputs of the probability 
          of the images.
        - <labels_in> is of the original labels of 
          the input images. 
    '''
    p_index_sorted = np.argsort(p_in)
    p_in_sorted=p_in[p_index_sorted]
    labels_in_sorted = labels_in[p_index_sorted]
    fpr_out = p_in*0.0
    tpr_out = p_in*0.0
    nNonLenses = len(labels_in[np.where(labels_in==0)])
    nLenses = len(labels_in[np.where(labels_in==1)]) 
    for i in range(len(p_in)):
        masksNonLenses = labels_in_sorted[i:]==0
        masksLenses = labels_in_sorted[i:]==1
        fpr_out[i] = len(labels_in_sorted[i:][masksNonLenses])/nNonLenses
        tpr_out[i] = len(labels_in_sorted[i:][masksLenses])/nLenses
    return fpr_out, tpr_out

def tprN(nFP, p_in, labels_in):
    '''
    Calculate TPR0, TPR10, and beyond. 
        - <nFP> is the threshold of False positives, 
          e.g., for tpr0, nFP = 1, 
          for tpr10, nFP = 10.
        - <p_in> is the outputs of the probability 
          of each image.
        - <labels_in> is of the original labels of 
          the input images. 
    '''
    p_index_sorted = np.argsort(p_in)
    p_in_sorted = p_in[p_index_sorted]
    labels_in_sorted = labels_in[p_index_sorted]
    nLenses = len(labels_in[np.where(labels_in==1)])
    for i in range(1,len(p_in)):
        idx = labels_in_sorted[-i:]<1
        nFPs = len(labels_in_sorted[-i:][idx])
        if nFPs>=nFP:
            break
        else:
            p_th = p_in_sorted[-i]
            nTPs = len(labels_in_sorted[-i:][~idx])
            tprN = nTPs/nLenses
    return tprN

if __name__ == "__main__":
    #-------------------------------------------------------
    # Calculating tpr0 and tpr10.
    # Note: the definitions of tpr0 and tpr10 are for the 
    # testing set of 100k images only, hence, 
    # the tpr0 and tpr10 for the dataset you 
    # provided is not comparable to the numbers 
    # in the paper of lens Finding Challenge. 
    # If we want to evaluate the rank of our 
    # approach in lens finding challenge, we 
    # have to apply this function to the outputs
    # of the testing set of 100k images from
    # lens finding challenge.
    #
    
    tpr0  = tprN(1, data['prob'], data['labels'])
    tpr10 = tprN(10, data['prob'], data['labels'])
    print(tpr0, tpr10)

    #-------------------------------------------------------
    # Comparing to the calculation using sklearn
    #
    import sklearn.metrics as skm
    fpr_skm, tpr_skm, pth_skm = skm.roc_curve(data['labels'], data['prob'])
    fpr_arr, tpr_arr = eval_outputs(data['prob'], data['labels'])

    masks1 = data['fpr'] < 5e-2
    masks2 = fpr_arr < 5e-2
    masks3 = fpr_skm < 5e-2

    pl.figure()
    pl.plot(data['fpr'][masks1], data['tpr'][masks1], '.', label='Your Results')
    pl.plot(fpr_arr[masks2], tpr_arr[masks2], '-', label='this function')
    pl.plot(fpr_skm[masks3], tpr_skm[masks3], '-', label='sklearn')
    pl.legend()
    pl.show()

[milesxu]

I have drew the plot by code above. I am not sure this is what you want.
All your code has merged to the repo.

[linan7788626]

Hi @milesxu, the figure you posted is correct but with a limited range of FPR. I set the limitation because I wanted to check the difference between the methods we used in the range of [0.0, 5e-2]. If you change the number (5e-2) in the below screenshot to 1.0, you will make a plot with a ROC from 0.0 to 1.0. Cheers.

[milesxu]

The Picture is here. All the changes are pushed to github.