Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark of algorithms #17

Open
keunwoochoi opened this issue Dec 18, 2015 · 4 comments
Open

Benchmark of algorithms #17

keunwoochoi opened this issue Dec 18, 2015 · 4 comments

Comments

@keunwoochoi
Copy link
Contributor

Hi,
do you have any benchmark result of the algorithms? Accuracy(or any other measure), complexity, or whatever.
If not, could you recommend me one? I'd like to get segmentations of 10k songs, and I can accept some degradation of performance for significantly low complexity.

@urinieto
Copy link
Owner

Hi, you can refer to my dissertation for some detailed scores (especially in the Appendices) for the existing algorithms in MSAF.

Generally, I would recommend Structural Features (sf) for boundaries, and 2D-Fourier Magnitude Coefficients (fmc2d) for labels.

Regardless, convolutional neural networks should outperform any of the existing algorithms in MSAF right now (this is the latest publication about it, I believe). I might include an implementation in MSAF sometime in the future (right, @f0k? :)).

@f0k
Copy link

f0k commented Dec 18, 2015

I might include an implementation in MSAF sometime in the future

Yeah, thanks for the reminder! I'll still have to figure out what's possible. Maybe the most realistic setting would be us posting the implementation from MIREX 2014 under a license not for commercial use. That would mean it would live outside of MSAF, but it could produce outputs in a compatible format so you can still use MSAF for labelling -- I think I remember it was relatively straightforward to use MSAF to label existing segmentations? Alternatively, MSAF could probably import and use it.

I can accept some degradation of performance for significantly low complexity.

Our CNNs are not tuned towards low complexity -- if you need high throughput, a simple hand-crafted algorithm may be better. A complex hand-crafted algorithm will probably be slower than the CNNs. Would be interesting to do a direct comparison some time.

@keunwoochoi
Copy link
Contributor Author

On MacPro 2014 (no multi processing) with 10 wave files. CNMF took 1555 seconds and Spectral clustering: 211 seconds. Posting for someone who might be also interested, performance was not evaluated though.

@urinieto
Copy link
Owner

Thanks! Yeah, C-NMF can be demanding. Probably the Checkerboard kernel (foote) is the fastest at finding boundaries. And 2D-FMC (fmc2d) the fastest at labeling the segments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants