Evaluation toolkit of WILDTRACK

[TTTREE]

Thanks your project!
Is there any Evaluation toolkit of WILDTRACK for estimating the MTMC tracker on WILDTRACK dataset? I can find MOTChallenge devkit for MOT. However, it seems no Evaluation toolkit for WILDTRACK or MTMC on MOTA , IDF1.
Looking forward to your reply.

[sramakrishnan247]

@TTTREE @Chavdarova
Any updates on this or any links that might be helpful?

[TTTREE]

@TTTREE @Chavdarova
Any updates on this or any links that might be helpful?

No, I am not working on it.

[sramakrishnan247]

@TTTREE
How did you proceed with this task?
Please let me know if you have any resources that might be helpful. Thanks!

[JohnPekl]

I do not really know what exactly the authors did but I have tried to build my own evaluation and hope it helps someone.

Since Wildtrack only provides ground truth location restricted to the ground plane, we only have (x, y) location (I guess it is the person's feet center) of each person without z, width, and height. Here is an example of ground truth at frame #0.

Here is an example of Ground Truth projection to image plane (camera index #3, (1, 2, 3, 4, 5, 6, 7)).

I use py-motmetrics to build this evaluation.

import motmetrics as mm
import numpy as np

def euclid_distance(x, y):
    return np.sqrt((x[0] - y[0]) ** 2 + (x[1] - y[1]) ** 2)


# Compute the distance between every pair of points
def clear_mot(gt_list, est_list, dataset_list, max_iou=0.5):
    acc = [None for i in gt_list]
    for idx, (gt, est) in enumerate(zip(gt_list, est_list)):
        num_frame = int(max(gt[:, 0]))
        acc[idx] = mm.MOTAccumulator(auto_id=True)

        for frame_idx in range(num_frame):
            targets_gt = gt[gt[:, 0] == (frame_idx + 1)]
            gt_ids = targets_gt[:, 1]
            targets_est = est[est[:, 0] == (frame_idx + 1)]
            trk_ids = targets_est[:, 1]
            iou = np.zeros((len(targets_gt), len(targets_est)))
            for gt_idx, tt_gt in enumerate(targets_gt):
                # bbox3d_gt = centerxyz_3dbbox(tt_gt[7:])  # CMC
                for est_idx, tt_est in enumerate(targets_est):
                    # bbox3d_est = centerxyz_3dbbox(tt_est[7:])  # CMC
                    # iou[gt_idx, est_idx] = bbox3d_distance(bbox3d_gt, bbox3d_est)  # CMC
                    iou[gt_idx, est_idx] = euclid_distance(tt_gt[7:9], tt_est[7:9]) # Wildtrack
            # Object / hypothesis points with larger distance are set to np.nan signalling do-not-pair.
            iou = np.where(iou > max_iou, np.nan, iou)
            acc[idx].update(gt_ids, trk_ids, iou )

    metrics = mm.metrics.motchallenge_metrics
    mh = mm.metrics.create()
    summary = mh.compute_many(
        acc,
        metrics=metrics,
        names=dataset_list,
        generate_overall=True
    )
    strsummary = mm.io.render_summary(
        summary,
        formatters=mh.formatters,
        namemap=mm.io.motchallenge_metric_names
    )
    return strsummary

How to use it?
np_gt and np_est are NumPy array and follow MOT Challenge format

Note from their paper: "For evaluation, we used the devkit provided with [31], and we similarly report metrics for radius r=1m".

{frame}, {id}, -1, -1, -1, -1, -1, {x}, {y}, -1, -1, -1, -1

strsummary = clear_mot([np_gt], [np_est], ['wildtrack'], 1)
print(strsummary)