use better overflow check

[mulle-nat]

This is a better mergable #2767, but my fork got out of sync and there were superflous commits in there. Sorry for the inconvenience. Here's my original comment.

I think you will like this much better. The other code was more or less a mitigation but this
should be a better overflow checker. The multiplication does the zero check and the
comparison checks that the multiplication doesn't overflow. calloc will then do the other
overflow check.

Adding to this: The old code caught overflow with large negative numbers, but this one will catch any kind of overflow. Also the code is easier to understand IMO.

[dankamongmen]

if we're really committing to this catch overflow idea, don't we want it backing ncplane_create(), and thus handling all ncplane creations?

[dankamongmen]

if we're really committing to this catch overflow idea, don't we want it backing ncplane_create(), and thus handling all ncplane creations?

nevermind, i think it is by virtue of being in ncplane_new_internal()

[dankamongmen]

we'll want a diagnostic. see other uses of LOG_

[mulle-nat]

It's not easy to see from the way github presents these changes, but the return 0 will leas to:

  size_t fbsize = ncplane_sizeof_cellarray(p->leny, p->lenx);
  if(!fbsize || (p->fb = calloc(sizeof(struct nccell),fbsize)) == NULL){
    logerror("error allocating cellmatrix (r=%u, c=%u)",
             p->leny, p->lenx);
    free(p);
    return NULL;
  }

so there is a logerror there and the huge or zero values will provide ample clues as what went wrong. (IMO)

[dankamongmen]

also here

[mulle-nat]

Thats deleted code though 😉

[mulle-nat]

Just wondering, if I need to add some more comments here, or if everything is clear.