Extending extension to support notebook cell clearing, processing

[psychemedia]

Hi

We have a use case in which we produce Jupyter notebooks for use by students as part of a distance education course, and are looking to automate how we test and package notebooks in weekly zip files for students.

For example:

• running authored notebooks to populate all cells to act as a gold master for testing using nbval;
• testing notebooks using nbval;
• cleaning notebook output cells using nbconvert;
• zipping folders with checks that we have removed hidden folders and files (such as checkpointed files containing assessment answers!).

Forgive my scruffy code, example here of a notebook processor function (without nbval test mode) that can be called as part of the zipper:

def notebookProcessor(notebook, mode=None, outpath=None, outfile=None, inplace=True):
    ''' Clear notebook output cells.
    
        Process a single notebook, clearing cell outputs running cells until
        a warning, or running all cells despite warnings.
        
        Processed notebooks can be written to a specified directory or rendered inplace.
    '''
    
    if mode is None: return (-1, 'Mode not specified.')
    
    if outpath is not None and not os.path.exists(outpath):
        os.makedirs(outpath)
    
    if outfile is not None:
        outpath = '/'.join([outpath,outfile]) if outpath is not None else outfile
    
    cmd='jupyter nbconvert --to notebook'
    
    if mode in ['clearOutput', 'clearOutputTest' ]:
        cmd = '{cmd} --ClearOutputPreprocessor.enabled=True'.format(cmd=cmd)
    elif mode == 'run':
        cmd = '{cmd} --execute'.format(cmd=cmd)
    elif mode == 'runWithErrors':
        cmd = '{cmd} --ExecutePreprocessor.allow_errors=True --execute'.format(cmd=cmd)
    else: return (-1, 'Mode not specified correctly.')
    
    if outpath is None and inplace:
        cmd='{cmd} --inplace'.format(cmd=cmd)

    #Select file
    cmd='{cmd} "{notebook}"'.format(cmd=cmd,notebook=notebook)
    
    #If output path not set, and --inplace is not set,
    #  nbformat will create a new file with same name ending: .nbformat.ipynb
    if outpath is not None:
        cmd ='{cmd} --output-dir "{outpath}"'.format(cmd=cmd, outpath=outpath)
        
    return cli_command(cmd)

I wonder if/how the UI of nbzip could be modified to allow users to:

• download zipped directories using either .tar.gz or .zip (we have a requirement on the latter);
• require output cells in notebooks to be cleared before zipping; or
• require notebooks to be run (until errors occur) before zipping; or
• require notebooks to be run even in the presence of errors before zipping; or
• run notebook tests on all notebooks in the folder.

It also occurs to me that there could be different modalities here:

1. running all notebooks in the folder in a particular way using nbformat or nbval;
2. zipping all notebooks in the folder;
3. running then zipping all the notebooks in the folder.