Add client side ExperimentDefinition

[panahiparham]

Addresses #7

[andnp]

We will want some single source-of-truth for this function. Most of it can be refactored out into a utility, I think:

def get_experiment_name():
  import __main__
  return __main__.__file__.split('/')[-2]

def get_results_path(base_path: str, name: str):
  return os.path.join(
    base_path,
    name,
    get_experiment_name(),
  )

Then both sides can reference this (the def'n creator side and the client side).

[panahiparham]

Agreed! But this change breaks stubbed_DefinitionPart in tests/test_ExperimentDefinition.py as this class stubs the get_results_path method. For now I added a property that points to the 'single source of truth` function that we can override in tests.

from ml_experiment._utils.path import get_results_path

class ExperimentDefinition:
    def __init__(self, ...):
        self.get_results_path = get_results_path
        ...

[andnp]

It might be worth pushing the assertion down into the:

self.table.get_configuration(cur, config_id)

code instead. This way, we can be more specific with our error message:

raise ValueError(f"config_id <{config_id}> is not in table <{self.get_table_name()}>")

[panahiparham]

Makes sense!

[andnp]

This is a great spot for the tmp_path fixture, which produces (and cleans up) a temporary file path for storing test results and guaranteeing there is no overlapping global state.

def test_ExperimentDefinition(tmp_path):
  ...
  part = stubbed_DefinitionPart(exp_name, part_name, base=str(tmp_path))
  ...

[panahiparham]

Sorcery!

[andnp]

This does make me wonder if we should consider making some guarantees about ordering. I'm leaning towards "yes", since that's also what PyExpUtils does.

We can make that an issue, then do the guaranteed ordering in another PR.

[panahiparham]

Created #11 for this feature!

[andnp]

Looks great! I would go ahead and do the dict(zip) change and whatever git cleanup you'd like, then it's good to merge!

[panahiparham]

After clean up this pr has 5 commits and resolves issue #7 by adding an experiment_definition class that handles the client side interaction with the experiment after it is created and stored in a metadata database.

Commit 1 and 2 implement the experiment_definition class and its corresponding test. Commit 3 and 4 implement a utility for getting experiment path and commit 5 adds an init file to allow pytest to import this package.