Skip to content
This repository has been archived by the owner on Dec 6, 2022. It is now read-only.

Error in rule convert_nb #2

Open
Sofie8 opened this issue Jan 16, 2021 · 2 comments
Open

Error in rule convert_nb #2

Sofie8 opened this issue Jan 16, 2021 · 2 comments

Comments

@Sofie8
Copy link

Sofie8 commented Jan 16, 2021

Hi Silas,

I was trying out atlas_analyze. So far so good, I have only one error at the second last step in rule convert_nb.
Here is the error line:

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       all
        1       convert_nb
        2
Select jobs to execute...

[Sat Jan 16 16:07:43 2021]
rule convert_nb:
    input: Results/Code.ipynb
    output: Results/Summary.html
    jobid: 1

[NbConvertApp] Converting notebook Results/Code.ipynb to html
Traceback (most recent call last):
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/reader.py", line 14, in parse_json
    nb_dict = json.loads(s, **kwargs)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/__init__.py", line 143, in read
    return reads(buf, as_version, **kwargs)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/__init__.py", line 73, in reads
    nb = reader.reads(s, **kwargs)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/reader.py", line 58, in reads
    nb_dict = parse_json(s, **kwargs)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbformat/reader.py", line 17, in parse_json
    raise NotJSONError(("Notebook does not appear to be JSON: %r" % s)[:77] + "...") from e
nbformat.reader.NotJSONError: Notebook does not appear to be JSON: ''...
[Sat Jan 16 16:07:49 2021]
Error in rule convert_nb:
    jobid: 1
    output: Results/Summary.html
    shell:
        jupyter nbconvert --output Summary --to=html --TemplateExporter.exclude_input=True Results/Code.ipynb
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/.snakemake/log/2021-01-16T160743.880488.snakemake.log
Traceback (most recent call last):
  File "./analyze.py", line 21, in <module>
    "snakemake "
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/shell.py", line 213, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail;  snakemake -d /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena -j 1 -s ./Snakefile' returned non-zero exit status 1.

And this was the previous error:

[Sat Jan 16 16:05:35 2021]
localrule analyze:
    input: Results/taxonomy.tsv, Results/mapping_rate.tsv, Results/genome_completeness.tsv, Results/counts/raw_counts_genomes.tsv, Results/counts/median_coverage_genomes.tsv, Results/annotations/KO.tsv, Results/annotations/CAZy.tsv, Results/annotations/KO.tsv, Results/annotations/CAZy.tsv
    log: Results/Code.ipynb
    jobid: 2

[NbConvertApp] ERROR | Notebook JSON is invalid: Additional properties are not allowed ('id' was unexpected)

Failed validating 'additionalProperties' in code_cell:

On instance['cells'][0]:
{'cell_type': 'code',
 'execution_count': None,
 'id': 'hourly-liberal',
 'metadata': {'tags': ['snakemake-job-properties']},
 'outputs': ['...0 outputs...'],
 'source': '\n'
           '######## snakemake preamble start (automatically inserted, do '
           'n...'}
Traceback (most recent call last):
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/jupyter_core/application.py", line 254, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/traitlets/config/application.py", line 845, in launch_instance
    app.start()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 350, in start
    self.convert_notebooks()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 524, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 489, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/nbconvertapp.py", line 418, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 181, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 199, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/notebook.py", line 32, in from_notebook_node
    nb_copy, resources = super().from_notebook_node(nb, resources, **kw)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 143, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/exporters/exporter.py", line 318, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 79, in preprocess
    self.execute()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/client.py", line 541, in async_execute
    cell, index, execution_count=self.code_cells_executed + 1
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 123, in async_execute_cell
    cell, resources = self.preprocess_cell(cell, self.resources, cell_index)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbconvert/preprocessors/execute.py", line 146, in preprocess_cell
    cell = run_sync(NotebookClient.async_execute_cell)(self, cell, index, store_history=self.store_history)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 74, in wrapped
    return just_run(coro(*args, **kwargs))
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/util.py", line 53, in just_run
    return loop.run_until_complete(coro)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/asyncio/futures.py", line 181, in result
    raise self._exception
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/asyncio/tasks.py", line 249, in __step
    result = coro.send(None)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/client.py", line 832, in async_execute_cell
    self._check_raise_for_error(cell, exec_reply)
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/nbclient/client.py", line 740, in _check_raise_for_error
    raise CellExecutionError.from_cell_and_msg(cell, exec_reply['content'])
nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:
------------------
#CAZy
CAZy_annotations_genome= pd.read_table('Results/annotations/CAZy.tsv',index_col=0)
CAZy_presence= (CAZy_annotations_genome>0).astype(int)
CAZy_presence.head()


function_relab = relab @ CAZy_presence

sns.clustermap(function_relab)

function_relab.head()
------------------

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-87471da32f9f> in <module>
      7 function_relab = relab @ CAZy_presence
      8
----> 9 sns.clustermap(function_relab)
     10
     11 function_relab.head()

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/_decorators.py in inner_f(*args, **kwargs)
     44             )
     45         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 46         return f(**kwargs)
     47     return inner_f
     48

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in clustermap(data, pivot_kws, method, metric, z_score, standard_scale, figsize, cbar_kws, row_cluster, col_cluster, row_linkage, col_linkage, row_colors, col_colors, mask, dendrogram_ratio, colors_ratio, cbar_pos, tree_kws, **kwargs)
   1410                         row_cluster=row_cluster, col_cluster=col_cluster,
   1411                         row_linkage=row_linkage, col_linkage=col_linkage,
-> 1412                         tree_kws=tree_kws, **kwargs)

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in plot(self, metric, method, colorbar_kws, row_cluster, col_cluster, row_linkage, col_linkage, tree_kws, **kws)
   1221         self.plot_dendrograms(row_cluster, col_cluster, metric, method,
   1222                               row_linkage=row_linkage, col_linkage=col_linkage,
-> 1223                               tree_kws=tree_kws)
   1224         try:
   1225             xind = self.dendrogram_col.reordered_ind

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in plot_dendrograms(self, row_cluster, col_cluster, metric, method, row_linkage, col_linkage, tree_kws)
   1067                 self.data2d, metric=metric, method=method, label=False, axis=0,
   1068                 ax=self.ax_row_dendrogram, rotate=True, linkage=row_linkage,
-> 1069                 tree_kws=tree_kws
   1070             )
   1071         else:

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/_decorators.py in inner_f(*args, **kwargs)
     44             )
     45         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
---> 46         return f(**kwargs)
     47     return inner_f
     48

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in dendrogram(data, linkage, axis, label, metric, method, rotate, tree_kws, ax)
    774     plotter = _DendrogramPlotter(data, linkage=linkage, axis=axis,
    775                                  metric=metric, method=method,
--> 776                                  label=label, rotate=rotate)
    777     if ax is None:
    778         ax = plt.gca()

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in __init__(self, data, linkage, metric, method, axis, label, rotate)
    582
    583         if linkage is None:
--> 584             self.linkage = self.calculated_linkage
    585         else:
    586             self.linkage = linkage

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in calculated_linkage(self)
    649                 warnings.warn(msg)
    650
--> 651         return self._calculate_linkage_scipy()
    652
    653     def calculate_dendrogram(self):

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/seaborn/matrix.py in _calculate_linkage_scipy(self)
    618     def _calculate_linkage_scipy(self):
    619         linkage = hierarchy.linkage(self.array, method=self.method,
--> 620                                     metric=self.metric)
    621         return linkage
    622

/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/scipy/cluster/hierarchy.py in linkage(y, method, metric, optimal_ordering)
   1063
   1064     if not np.all(np.isfinite(y)):
-> 1065         raise ValueError("The condensed distance matrix must contain only "
   1066                          "finite values.")
   1067

ValueError: The condensed distance matrix must contain only finite values.
ValueError: The condensed distance matrix must contain only finite values.

[Sat Jan 16 16:06:24 2021]
Error in rule analyze:
    jobid: 2
    log: Results/Code.ipynb (check log file(s) for error message)

RuleException:
CalledProcessError in line 82 of /ddn1/vol1/site_scratch/leuven/314/vsc31426/atlas_analyse/atlas_analyze/Snakefile:
Command 'set -euo pipefail;  jupyter-nbconvert --log-level ERROR --execute --output /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/Results/Code.ipynb --to notebook --ExecutePreprocessor.timeout=-1 /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/.snakemake/scripts/tmpbmbi86fl.Analyis_genome_abundances.ipynb' returned non-zero exit status 1.
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2338, in run_wrapper
  File "/ddn1/vol1/site_scratch/leuven/314/vsc31426/atlas_analyse/atlas_analyze/Snakefile", line 82, in __rule_analyze
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 566, in _callback
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/concurrent/futures/thread.py", line 57, in run
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 552, in cached_or_run
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2350, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena/.snakemake/log/2021-01-16T160453.469631.snakemake.log
Traceback (most recent call last):
  File "./analyze.py", line 21, in <module>
    "snakemake "
  File "/vsc-hard-mounts/leuven-data/314/vsc31426/miniconda3/envs/analyze/lib/python3.7/site-packages/snakemake/shell.py", line 213, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail;  snakemake -d /ddn1/vol1/site_scratch/leuven/314/vsc31426/NGS_oct/Narmena -j 1 -s ./Snakefile' returned non-zero exit status 1.
@SilasK
Copy link
Member

SilasK commented Jan 16, 2021

This nbconvert is anoying. I probably won't continue to use it.

If it worked so far you can either look at the Results/Code.ipynb

or directly use the R code from the Tutorial.

@CedricMidoux
Copy link
Contributor

Hello @SilasK and @Sofie8
It finally worked for me.
It was necessary to downgrade numpy to numpy==1.16.5 like here : numpy/numpy#18355
Bye.

@SilasK SilasK mentioned this issue Mar 17, 2021
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@SilasK @Sofie8 @CedricMidoux