Feature/enable and disable rules

[MariusWirtz]

Successor of #1103

[vmitsenko]

Not all tests passed

[vmitsenko]

Consider adding a RunTimeError when attempting to disable feeders in a rule without feeders

[MariusWirtz]

As a TM1py user, I would prefer not to receive an error in that scenario. If we did raise it, I would have to handle it on each write function just in case that feeders might be removed at some point in the cube.

But that's just my opinion

[vmitsenko]

I'm fine with that, just noticed that in this scenario we get
ValueError: 'feeders;' is not in list
which is still easy to interpret

[MariusWirtz]

Just pushed another commit. Neither empty rules nor empty feeders will raise an error.

I think that's the most convenient from a user perspective. Especially if we introduce this as an optional argument on the write function.

[MariusWirtz]

@vmitsenko, @nicolasbisurgi
All tests pass.

Are you OK to merge?

[vmitsenko]

I'm fine with that, just noticed that in this scenario we get
ValueError: 'feeders;' is not in list
which is still easy to interpret

[MariusWirtz]

The below script attempts to quantify performance improvements in this feature. It is self contained and should work with any environment.

With MTFeeders set to True, I get a 10% performance improvement in a scenario with heavy feeding rules (each written cell feeds 1000 other cells).

@vmitsenko, @nicolasbisurgi
Do you have a more or less optimal scenario in your environment?

from datetime import datetime

import numpy as np
import pandas as pd

from TM1py import TM1Service
from TM1py.Objects.Cube import Cube

tm1params = {...}

CUBE_NAME = "Py Cube"
DIMENSION_NAMES = ["Py Version", "Py Time", "Py Product", "Py Customer", "Py Measure"]
ELEMENTS = [f"Element {str(n).zfill(3)}" for n in range(1, 1001)]
VERSION_VALUE = "Element 001"
NUM_RECORDS = 1_000

with TM1Service(**tm1params) as tm1:
    for dimension_name in DIMENSION_NAMES:
        df = pd.DataFrame(
            columns=[dimension_name],
            data=ELEMENTS)
        df["level000"] = "TOTAL"
        tm1.hierarchies.update_or_create_hierarchy_from_dataframe(
            dimension_name=dimension_name,
            hierarchy_name=dimension_name,
            df=df)

    rules = """
    SKIPCHECK;
    
    FEEDERS;
    ['Py Version':'Element 001'] => ['Py Version':'TOTAL'];
    """
    cube = Cube(CUBE_NAME, dimensions=DIMENSION_NAMES, rules=rules)
    if tm1.cubes.exists(CUBE_NAME):
        tm1.cubes.delete(CUBE_NAME)
    tm1.cubes.create(cube)

    # Creating the DataFrame
    df = pd.DataFrame({
        "Py Version": [VERSION_VALUE] * NUM_RECORDS,
        "Py Time": np.random.choice(ELEMENTS, size=NUM_RECORDS),
        "Py Product": np.random.choice(ELEMENTS, size=NUM_RECORDS),
        "Py Customer": np.random.choice(ELEMENTS, size=NUM_RECORDS),
        "Py Measure": np.random.choice(ELEMENTS, size=NUM_RECORDS),
        "Value": [1000] * NUM_RECORDS
    })

    before = datetime.now()
    tm1.cells.write_dataframe(cube_name=CUBE_NAME, data=df, use_blob=True)
    elapsed_time = datetime.now() - before
    print("Elapsed time with feeders: ", elapsed_time)

    if tm1.cubes.exists(CUBE_NAME):
        tm1.cubes.delete(CUBE_NAME)
    tm1.cubes.create(cube)

    before = datetime.now()
    tm1.cubes.disable_feeders(CUBE_NAME)
    tm1.cells.write_dataframe(cube_name=CUBE_NAME, data=df, use_blob=True)
    tm1.cubes.enable_feeders(CUBE_NAME)
    elapsed_time = datetime.now() - before
    print("Elapsed time without feeders: ", elapsed_time)

Output

Elapsed time with feeders:  0:07:11.574673
Elapsed time without feeders:  0:05:52.537822

[MariusWirtz]

Closed: Subject to Future Re-evaluation