Errors in registration of functions using new interface

[drculhane]

Describe the bug
See my Draft PR # 3892. The functions cumsum and cumprod (in EfuncMsg.chpl) are not registered correctly.

To Reproduce
Steps to reproduce the behavior:

1. Pull Draft PR # 3892.
2. Do make register-commands
3. Look at src/registry/Commands.chpl
4. You will see that cumsum is only registered for the bool data type, and cumprod is only registered for int, uint, real.

Expected behavior
I expected both functions cumsum and cumprod to be registered for all 4 data types: int, uint, real, bool.

Error Message
Unit tests fail on cumsum and cumprod, because the unit tests attempt to use all 4 data types.

Is this a Blocking Issue

• There is a workaround. The code below registers correctly.

     proc cumspReturnType(type t) type
      do return if t == bool then int else t;


    @arkouda.registerCommand(name="cumsum")
    proc cumsum(x : [?d] ?t) : [d] cumspReturnType(t) throws
        where (t==int || t==real || t==uint || t==bool) {
            if x.rank == 1 {
                overMemLimit(numBytes(int) * x.size) ;
                if t == bool {
                    var ix = makeDistArray(x.domain, int); // make a copy of bools as ints blah!
                    ix = x:int ;
                    return (+scan (ix));
                } else {
                    return (+scan x) ;
                }
            } else {
                throw new Error ("Over mem limit in cumsum") ;
            }
        }

    @arkouda.registerCommand(name="cumprod")
    proc cumprod(x : [?d] ?t) : [d] cumspReturnType(t) throws
        where (t==int || t==real || t==uint || t==bool) {
            if x.rank == 1 {
                overMemLimit(numBytes(int) * x.size) ;
                if t == bool {
                    var ix = makeDistArray(x.domain, int); // make a copy of bools as ints blah!
                    ix = x:int ;
                    return (*scan (ix));
                } else {
                    return (*scan x) ;
                }
            } else {
                throw new Error ("Over mem limit in cumprod") ;
            }
        }

ak.get_config() Output

connected to arkouda server tcp://*:5555
>>> ak.get_config()
{'arkoudaVersion': 'v2024.10.02+41.gdd5c07814', 'chplVersion': '2.2.0', 'ZMQVersion': '4.3.5', 'HDF5Version': '1.14.3', 'serverHostname': 'pop-os', 'ServerPort': 5555, 'numLocales': 2, 'numPUs': 14, 'maxTaskPar': 14, 'physicalMemory': 33653170176, 'distributionType': 'BlockDom(1,int(64),one,unmanaged DefaultDist)', 'LocaleConfigs': [{'id': 0, 'name': 'pop-os-0', 'numPUs': 14, 'maxTaskPar': 14, 'physicalMemory': 33653170176}, {'id': 1, 'name': 'pop-os-1', 'numPUs': 14, 'maxTaskPar': 14, 'physicalMemory': 33653170176}], 'authenticate': False, 'logLevel': 'INFO', 'logChannel': 'CONSOLE', 'regexMaxCaptures': 20, 'byteorder': 'little', 'autoShutdown': False, 'serverInfoNoSplash': False, 'maxArrayDims': 1, 'ARROW_VERSION': '17.0.0'}

Additional context
To reproduce the error in Commands.chpl, you only need to do make register-commands.
To see the failures in unit test, do a full make (1 dim is sufficient), and run the tests/numpy/numeric_test.py unit test. You will see failures in test_cumsum_and_cumprod

[ajpotts]

I was able to get this version to work:

    /* Implements + reduction over numeric data, converting all elements to int before summing. */
    class PlusIntReduceOp: ReduceScanOp {
        type eltType;

        var value: int;

        proc identity      do return 0: int;

        proc accumulate(elm)  { value = value + elm:int; }

        proc accumulateOntoState(ref state, elm)  { state = state + elm:int; }

        proc initialAccumulate(outerVar) { value = value + outerVar: int; }

        proc combine(other: borrowed PlusIntReduceOp(?))   { value = value + other.value; }

        proc generate()    do return value;

        proc clone()       do return new unmanaged PlusIntReduceOp(eltType=eltType);
    }

    @arkouda.registerCommand(name="cumsum")
    proc cumsum(x : [?d] ?t) : [d] cumspReturnType(t) throws
        where (t==int || t==real || t==uint || t==bool) && (d.rank==1) {
            overMemLimit(numBytes(int) * x.size) ;
            if t == bool {
                return (PlusIntReduceOp scan x);
            } else {
                return (+ scan x) ;
            }
        }

    @arkouda.registerCommand(name="cumprod")
    proc cumprod(x : [?d] ?t) : [d] cumspReturnType(t) throws
        where (t==int || t==real || t==uint || t==bool) && (d.rank==1) {
            overMemLimit(numBytes(int) * x.size) ;
            if t == bool {
                return (&& scan x);
            } else {
                return (*scan x) ;
            }
        }

[ajpotts]

Here's the chapel docs on user defined Reduce Intents:
https://chapel-lang.org/docs/technotes/reduceIntents.html#readme-reduceintents-interface

[ajpotts]

I know you're goal isn't to find the best work around, but this also works:

    @arkouda.registerCommand(name="cumsum")
    proc cumsum(x : [?d] ?t) : [d] cumspReturnType(t) throws
        where (t==int || t==real || t==uint || t==bool) && (d.rank==1) {
            overMemLimit(numBytes(int) * x.size) ;
            return cumsumHelper(x);
        }

    proc cumsumHelper(x : [?d] ?t) : [d] t throws
        where (t==int || t==real || t==uint) && (d.rank==1) {
        return (+ scan x) ;
    }

    proc cumsumHelper(x : [?d] ?t) : [d] int throws
    where (t==bool) && (d.rank==1) {
        return (PlusIntReduceOp scan x);
    }

    @arkouda.registerCommand(name="cumprod")
    proc cumprod(x : [?d] ?t) : [d] cumspReturnType(t) throws
        where (t==int || t==real || t==uint || t==bool) && (d.rank==1) {
        return cumprodHelper(x);
    }

    proc cumprodHelper(x : [?d] ?t) : [d] t throws
        where (t==int || t==real || t==uint) && (d.rank==1) {
        return (*scan x);
    }

    proc cumprodHelper(x : [?d] ?t) : [d] int throws
    where (t==bool) && (d.rank==1) {
        return (&& scan x);
    }

[ajpotts]

I think this is closer to what you were actually hoping for. Using ignoreWhereClause=true will change the where clause evaluation behavior. The following passed the unit tests.

    @arkouda.registerCommand(name="cumsum", ignoreWhereClause=true)
    proc cumsum(x : [?d] ?t) : [d] int throws
        where (t==bool) && (d.rank == 1){
            overMemLimit(numBytes(int) * x.size) ;
            var ix = makeDistArray(x.domain, int); // make a copy of bools as ints blah!
            ix = x:int ;
            return (+scan (ix));
    }
    proc cumsum(x : [?d] ?t) : [d] t throws
        where (t==int || t==real || t==uint) && (d.rank == 1) {
            overMemLimit(numBytes(int) * x.size) ;
            return (+scan x) ;
    }
    proc cumsum(x : [?d] ?t) : [d] int throws
        where (t!=bool && t!=int && t!=real && t!=uint) || (d.rank != 1) {
            throw new Error ("Message TBD") ;
    }

    @arkouda.registerCommand(name="cumprod", ignoreWhereClause=true)
    proc cumprod(x : [?d] ?t) : [d] t throws
        where (t==int || t==real || t==uint) && (d.rank == 1) {
            overMemLimit(numBytes(int) * x.size) ;
            return (*scan x) ;
    }
    proc cumprod(x : [?d] ?t) : [d] int throws
        where (t==bool) && (d.rank == 1) {
            overMemLimit(numBytes(int) * x.size) ;
            var ix = makeDistArray(x.domain, int); // make a copy of bools as ints blah!
            ix = x:int ;
            return (*scan (ix));
    }
    proc cumprod(x : [?d] ?t) : [d] int throws
        where (t!=bool && t!=int && t!=real && t!=uint) || (d.rank != 1) {
            throw new Error ("Message TBD") ;
    }

The down side is that it generates a lot of extra functions in Commands.chpl file.