Division bug in Programmer Calculator DEC mode

[korenkonder]

Describe the bug
There's weird bug when you divide 4294967296 (0x100000000) by 255 (0xFF) in DEC mode. The output is 0
Also happens with:
4294967297 (0x100000001)
4294967298 (0x100000002)
4294967299 (0x100000003)
4294967300 (0x100000004)
4294967301 (0x100000005)
4294967302 (0x100000006)
4294967303 (0x100000007)

Steps To Reproduce
DEC way

1. Go to Programmer Calculator
2. Click on DEC
3. Enter 4294967296
4. Press / (Divide)
5. Enter 255
6. Press = (Result)
7. See error

HEX way

1. Go to Programmer Calculator
2. Click on HEX
3. Enter 100000000
4. Press / (Divide)
5. Enter FF
6. Click on DEC
7. Press = (Result)
8. See error

Expected behavior
Result should be 16843009 (0x1010101)

Screenshots
Screenshots may be not enough so I had recorded video
https://www.youtube.com/watch?v=J4If2A0BnQo

Device and Application Information

• OS Build: 10.0.19043.0
• Architecture: x64
• Application Version: 10.2103.8.0
• Region: ru-RU
• Dev Version Installed: False

Requested Assignment
I'm just reporting this problem. I don't want to fix it.

[MicrosoftIssueBot]

This is your friendly Microsoft Issue Bot. I've seen this issue come in and have gone to tell a human about it.

[randy408]

I can confirm the same issue with Calculator 10.2103.8.0.

[bahlea]

#1788
I reported another one of those number combinations before stumbling accross this one.

[rogitech]

I noticed the remainder of pret after remrat(&pret, rat_one) is not in the same scale as px, in this case, so subrat(px, pret, precision); will produce a wrong result.
By doing another flatrat(pret, radix, precision); solves the issue (and #1788 as well). The only problem is this fix breaks one of the unit tests - UnitTest: CalculatorManagerTestScientific2 (commands21). (And I guess it's not ideal to change intrat()).

void intrat(Inout PRAT* px, uint32_t radix, int32_t precision)
{
// Only do the intrat operation if number is nonzero.
// and only if the bottom part is not one.
if (!zernum((*px)->pp) && !equnum((*px)->pq, num_one))
{
flatrat(*px, radix, precision);

    // Subtract the fractional part of the rational
    PRAT pret = nullptr;
    DUPRAT(pret, *px);
    remrat(&pret, rat_one);

    // Flattern the remainder in case it's not aligned with px after % operation
    flatrat(pret, radix, precision);

    subrat(px, pret, precision);
    destroyrat(pret);

    // Simplify the value if possible to resolve rounding errors
    flatrat(*px, radix, precision);
}

}

[ishuah]

I wrote a fix for this issue built on @rogitech's comment - PR #2300.

Instead of flattening pret on every call, only flatten it when it's not in the same scale as px.

        // Flatten pret in case it's not aligned with px after remrat operation
        if (!equnum((*px)->pq, pret->pq))
        {
            flatrat(pret, radix, precision);
        }