[PRE REVIEW]: Ipyannotator: the infinitely hackable annotation framework

[editorialbot]

Submitting author: @itepifanio (Ítalo Epifânio)
Repository: https://github.com/palaimon/ipyannotator/
Branch with paper.md (empty if default branch): joss
Version: 0.8.1
Editor: @danielskatz
Reviewers: @csadorf, @matthewfeickert
Managing EiC: Arfon Smith

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5306e204788fd469104d77321223d7a1"><img src="https://joss.theoj.org/papers/5306e204788fd469104d77321223d7a1/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5306e204788fd469104d77321223d7a1/status.svg)](https://joss.theoj.org/papers/5306e204788fd469104d77321223d7a1)

Author instructions

Thanks for submitting your paper to JOSS @itepifanio. Currently, there isn't an JOSS editor assigned to your paper.

@itepifanio if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSS submission bot @editorialbot is here to help you find and assign reviewers and start the main review. To find out what @editorialbot can do for you type:

@editorialbot commands

[editorialbot]

Hello human, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.03 s (141.4 files/s, 19162.7 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
SVG                              2              0              0            286
Markdown                         1             50              0            105
TeX                              1              9              0             92
-------------------------------------------------------------------------------
SUM:                             4             59              0            483
-------------------------------------------------------------------------------


gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1790

[editorialbot]

Failed to discover a valid open source license

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/3359141 is OK

MISSING DOIs

- 10.1111/cgf.12574 may be a valid DOI for title: Smartannotator an interactive tool for annotating indoor rgbd images
- 10.1109/icip42928.2021.9506683 may be a valid DOI for title: BRIMA: low-overhead BRowser-only IMage Annotation tool (Preprint)
- 10.1145/3343031.3350535 may be a valid DOI for title: The VIA annotation software for images, audio and video
- 10.1007/s11548-018-1864-x may be a valid DOI for title: GTCreator: a flexible annotation tool for image-based datasets

INVALID DOIs

- None

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[arfon]

Output of cloc for actual software:

github.com/AlDanial/cloc v 1.86  T=0.13 s (684.0 files/s, 205871.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Jupyter Notebook                35              0          11385           7388
Python                          29           1277            805           4789
Markdown                         4            146              0            267
YAML                             4             23              3            184
Bourne Shell                     5             25             13             60
Dockerfile                       1             20             10             59
JSON                             2              0              0             59
INI                              1              8              0             50
TOML                             1              5              2             50
DOS Batch                        1              8              1             26
make                             1             11              0             26
HTML                             2              0              0             18
reStructuredText                 3             11             45             14
-------------------------------------------------------------------------------
SUM:                            89           1534          12264          12990
-------------------------------------------------------------------------------

[itepifanio]

Potential reviewers: galessiorob, emdupre, jmschrei

[itepifanio]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[arfon]

@editorialbot invite @danielskatz as editor

👋 @danielskatz – would you be able to take this submission on?

[editorialbot]

Invitation to edit this submission sent!

[danielskatz]

@editorialbot assign me as editor

[editorialbot]

Assigned! @danielskatz is now the editor

[danielskatz]

👋 @itepifanio - I'll be your editor. As we try to find reviewers, you could work on the possibly missing DOIs that editorialbot suggests, but note that some may be incorrect. Please feel free to make changes to your .bib file, then use the command @editorialbot check references to check again, and the command @editorialbot generate pdf when the references are right to make a new PDF. editorialbot commands need to be the first entry in a new comment.

[danielskatz]

In addition, if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSS and may be suitable for this submission (please start at the bottom of the list). The suggestions you made above are editors, and we generally don't ask editors to also be reviewers.

[itepifanio]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/3359141 is OK
- 10.1145/3359141 is OK
- 10.1109/icip42928.2021.9506683 is OK
- 10.1145/3343031.3350535 is OK
- 10.1007/s11548-018-1864-x is OK

MISSING DOIs

- None

INVALID DOIs

- None

[itepifanio]

@editorialbot generate pdf

[itepifanio]

Potential reviewers: aminmed, Shahzaibkhalid25, Athene-ai

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[itepifanio]

@danielskatz thanks for the suggestions! my last commands should cover them all

[danielskatz]

👋 @aminmed, @Shahzaibkhalid25, @Athene-ai - Would ideally two of you be willing to review this submission?

[Athene-ai]

@danielskatz I would like to review this submission

[danielskatz]

Great, thanks - I will add you to the system, but we won't start the review until we also get a second reviewer

[danielskatz]

@editorialbot add @Athene-ai as reviewer

[editorialbot]

@Athene-ai added to the reviewers list!

[danielskatz]

@Athene-ai - As a JOSS policy, as I wrote above:

we prefer to start new reviewers off with just one review initially

Once your initial review is complete, we can assign you more

[Athene-ai]

@Athene-ai - As a JOSS policy, as I wrote above:

we prefer to start new reviewers off with just one review initially

Once your initial review is complete, we can assign you more

Oh I see :-) So I can not wait to start and complete my initial review in order to contribute to review more

[ibayer]

@matthewfeickert

if Ipyannotator meets the scholarly effort criteria for JOSS

Thanks for bringing this up. In addition to @itepifanio comment above I want to mention that we have been following a Private Development, Public Releases workflow[0].
I guess this makes an effort estimate based on number of commits difficult. Please find below a screenshot from our private development branch. I hope the number of commits are more in line with your expectations.

Also now that the project matures we are planning to also develop more code in the open.

[0] https://www.braintreepayments.com/blog/our-git-workflow/

[matthewfeickert]

Palaimon's uses Ipyannotator internally and since we work with RnD projects our iterations are pretty quickly with ideas that are not always being added to the library core. To keep the iterations flexible we use a private GitLab repository and release all changes at once on Github, that's why we have just a few commits.

@itepifanio @ibayer Thanks for clarifying. I didn't understand from the Issue or from the docs that this is actually a project from a company that is being made incrementally open. While different then most projects submitted, there's nothing wrong with this approach. 👍

Regarding the documentation, we always try the tutorial-first approach

That's fine for your team, but it appears it isn't "tutorial-first" but rather "tutorial-only". As it stands, your project has no API reference documentation (the Internal Components page is essentially just a view of the source code) and the docs/tutorial webpage is unversioned, so in terms of the JOSS review criteria for API documentation I would have to say this is not passing.

To be clear, I think user guides are great, but I think API documentation is necessary.

[ibayer]

@matthewfeickert

To be clear, I think user guides are great, but I think API documentation is necessary.

I understand your concern. ipyannotator is build using https://nbdev.fast.ai/ which is somewhat popular (3.2k stars) and is also being used to build the popular https://github.com/fastai/fastai library (22k stars). As is, nbdev is a bit special literate programming style and doesn't produce the typical sphinx type API docs. To be clear, I'm not opposing your view. I'm just trying to provide background to avoid the impression that the missing API docs are viewed as being an indication of us being lazy or sloppy. ;)

We'll discuss internally what we can propose to accommodate the need for generated API docs.

[itepifanio]

@matthewfeickert we could provide an "API" page reference like the adaptr (one of the latest papers accepted).

doesn't produce the typical sphinx type API docs

Our API doesn't change much, so a manual reference could be an option.

[matthewfeickert]

I'm just trying to provide background to avoid the impression that the missing API docs are viewed as being an indication of us being lazy or sloppy

Yup. I fully appreciate that and I in no way view your team as lazy as I'm quite familiar with both literate programming and nbdev. fast.ai is great and Jeremy Howard is very skilled at API design — I use ghapi regularly, which I'll note does have API reference docs.

Our API doesn't change much, so a manual reference could be an option.

Please don't. That's a bad option for everyone — it creates busy work for you and it will only lead to wrong APIs in the future.

[itepifanio]

@matthewfeickert thanks for the reference, this could be a very helpful example for us. We'll take a look at the ghapi API doc generation to see if we can provide something similar. I'll provide feedback as soon as I design a POC.

[itepifanio]

@matthewfeickert can you check Ipyannotator docs again? The new release contains the changes that you suggested.

[danielskatz]

👋 @matthewfeickert - Any thoughts on this?

[itepifanio]

@danielskatz I was wondering if there's something we can do to keep the review process rolling.

[danielskatz]

sorry, this is my fault - I had a very busy week.

👋 @matthewfeickert - if you can do this review (based on your interest and the discussion so far), it would be very helpful.

[danielskatz]

👋 @bnriiitb, @rewbenio - Would one of you be willing to review this JOSS submission?

[danielskatz]

👋 @matthewfeickert - congratulations on your new job!! Also, repeating, if you can do this review (based on your interest and the discussion so far), it would be very helpful.

[danielskatz]

👋 @lepisma - Would you be willing to review this JOSS submission?

[matthewfeickert]

Sorry for going into radio silence — I had GitHub notifications turned off for awhile. I've got them back on again and will be following up here later today.

[danielskatz]

@matthewfeickert - what do you think? (not trying to bug you too much, just want to get this review going...)

[ibayer]

@danielskatz Thanks for staying on top of things. Please let us know if we can do anything else to pass the [PRE REVIEW] stage and get going. We are very much looking forward to getting more feedback that helps us to improve ipyannotator. :)

[matthewfeickert]

Hi. I am currently traveling in a remote part of the US with terrible internet. @danielskatz I will try to read through all of this and give some useful response by Sunday night (I will be home and have stable internet on Monday).

edit: Flight delays mean that I got back on Monday afternoon, so will look on Tuesday.

[danielskatz]

Thanks @matthewfeickert

[matthewfeickert]

Thanks for everyone's extreme patience here.

can you check Ipyannotator docs again? The new release contains the changes that you suggested.

Thanks @itepifanio. As Annotator only exposes 3 public methods this is a pretty good start, so we can discuss any further bits in the review proper.

Also, repeating, if you can do this review (based on your interest and the discussion so far), it would be very helpful.

I'll support moving ipyannotator into review status. I'm willing to do the review, and I hope it can be done quickly, but I also know that things are going to be pretty hectic between now and the last week of July. I can fully appreciate that both the editors and the authors might (very reasonably!) require a more reasonable iteration speed then I can provide (2 weeks should generally more than enough time in most situations). So while I'm ready to move forward I also understand if the authors would be more comfortable with another reviewer and won't be offended if they request that I be replaced if I fail to stick to a reasonable time in the review. Everyone's time is valuable here and I don't want the authors time abused.

Okay, with that out of the way, onward to the review if @csadorf is still game! 🚀

@danielskatz Also thanks very much for pinging me here. I really do appreciate it, especially when I'm fighting to keep any notifications on the board. Hopefully you won't have to keep doing it, but thanks for not giving up!

congratulations on your new job!!

Thanks! I truly appreciate it. :) (One of the reasons things have been much busier than normal)

[danielskatz]

Thanks @matthewfeickert - I'll add you and start the review

[danielskatz]

@editorialbot add @matthewfeickert as reviewer

[editorialbot]

@matthewfeickert added to the reviewers list!

[danielskatz]

@editorialbot start review

[editorialbot]

OK, I've started the review over in #4480.