Skip to content
This repository has been archived by the owner on Jan 31, 2022. It is now read-only.

Structure of RPM package #68

Closed
jsturdy opened this issue Mar 10, 2018 · 12 comments
Closed

Structure of RPM package #68

jsturdy opened this issue Mar 10, 2018 · 12 comments

Comments

@jsturdy
Copy link
Contributor

jsturdy commented Mar 10, 2018

Information

  • The packages and modules will be packaged and end up in the system PYTHONPATH, e.g., (default python on cc7): /usr/lib/python2.7/site-packages/`
    • I've been toying with adding a global namespace gempython, under which there would be the gemplotting, vftaqc, etc., modules, but this is still to be finalized into a proposal, and then discussed
    • This would necessarily involve a small redesign of the import statements, but I have for the most part decided on a workable solution for people who are doing private development and not using the production software
  • Executable scripts, e.g., amc_info_uhal.py will all go into /opt/cmsgemos/bin, to make it simple to add this to the PATH variable and have the script ready

Current RPM structure (UPDATED 12-03-2018)

gempython_gemplotting     /opt/cmsgemos/bin/anaInfo.py
gempython_gemplotting     /opt/cmsgemos/bin/anaUltraLatency.py
gempython_gemplotting     /opt/cmsgemos/bin/anaUltraScurve.py
gempython_gemplotting     /opt/cmsgemos/bin/anaUltraThreshold.py
gempython_gemplotting     /opt/cmsgemos/bin/anaXDAQLatency.py
gempython_gemplotting     /opt/cmsgemos/bin/ana_scans.py
gempython_gemplotting     /opt/cmsgemos/bin/anaoptions.py
gempython_gemplotting     /opt/cmsgemos/bin/anautilities.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/__init__.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/__init__.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/__init__.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/__init__.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/__init__.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/__init__.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/__init__.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/__init__.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/__init__.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/fitScanData.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/fitScanData.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/fitting/fitScanData.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/__init__.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/__init__.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/__init__.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemPlotter.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemPlotter.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemPlotter.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemSCurveAnaToolkit.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemSCurveAnaToolkit.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemSCurveAnaToolkit.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemTreeDrawWrapper.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemTreeDrawWrapper.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/gemTreeDrawWrapper.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/overlay_fit.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/overlay_fit.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/overlay_fit.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_eff.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_eff.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_eff.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_noise_vs_trim.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_noise_vs_trim.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_noise_vs_trim.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_scurves_by_thresh.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_scurves_by_thresh.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_scurves_by_thresh.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_summary.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_summary.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plot_vfat_summary.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plotoptions.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plotoptions.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/plotoptions.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/scurvePlottingUtitilities.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/scurvePlottingUtitilities.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/scurvePlottingUtitilities.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/summary_plots.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/summary_plots.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/macros/summary_plots.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/PanChannelMaps.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/PanChannelMaps.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/PanChannelMaps.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/__init__.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/__init__.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/__init__.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/buildMapFiles.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/buildMapFiles.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/buildMapFiles.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/chamberInfo.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/chamberInfo.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/chamberInfo.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/channelMaps.py
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/channelMaps.pyc
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/mapping/channelMaps.pyo
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython/gemplotting/requirements.txt
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/PKG-INFO
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/SOURCES.txt
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/dependency_links.txt
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/not-zip-safe
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/requires.txt
gempython_gemplotting     /usr/lib/python2.7/site-packages/gempython_gemplotting-1.0.1-py2.7.egg-info/top_level.txt

Current tarball structure

gempython_gemplotting-1.0.1/
gempython_gemplotting-1.0.1/gempython/
gempython_gemplotting-1.0.1/gempython/gemplotting/
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaInfo.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaUltraLatency.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaUltraScurve.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaUltraThreshold.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaXDAQLatency.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/ana_scans.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anaoptions.py
gempython_gemplotting-1.0.1/gempython/gemplotting/bin/anautilities.py
gempython_gemplotting-1.0.1/gempython/gemplotting/fitting/
gempython_gemplotting-1.0.1/gempython/gemplotting/fitting/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/fitting/fitScanData.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/gemPlotter.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/gemSCurveAnaToolkit.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/gemTreeDrawWrapper.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/overlay_fit.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_eff.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_noise_vs_trim.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_scurves_by_thresh.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_vfat_and_channel_Scurve.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plot_vfat_summary.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/plotoptions.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/scurvePlottingUtitilities.py
gempython_gemplotting-1.0.1/gempython/gemplotting/macros/summary_plots.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/PanChannelMaps.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/buildMapFiles.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/chamberInfo.py
gempython_gemplotting-1.0.1/gempython/gemplotting/mapping/channelMaps.py
gempython_gemplotting-1.0.1/gempython/gemplotting/__init__.py
gempython_gemplotting-1.0.1/gempython/gemplotting/requirements.txt
gempython_gemplotting-1.0.1/gempython/__init__.py
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/PKG-INFO
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/SOURCES.txt
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/dependency_links.txt
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/not-zip-safe
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/requires.txt
gempython_gemplotting-1.0.1/gempython_gemplotting.egg-info/top_level.txt
gempython_gemplotting-1.0.1/README.md
gempython_gemplotting-1.0.1/setup.cfg
gempython_gemplotting-1.0.1/setup.py
gempython_gemplotting-1.0.1/PKG-INFO
@bdorney
Copy link
Contributor

bdorney commented Mar 12, 2018

Before we finalize the RPM for cc7 I think it may be a useful idea to fix all ROOT 5.34 vs. 6.X.Y issues; I know @BenjaminRS was working on solving at least one.

The issues I'm referring to are:

#63
#63

@jsturdy
Copy link
Contributor Author

jsturdy commented Mar 12, 2018

What is the usage of these:

ll gem-plotting-tools/macros
total 104
-rwxr-xr-x. 1 sturdy zh  1655 Mar  6 15:54 gemPlotterAllChannels.sh
-rwxr-xr-x. 1 sturdy zh 22749 Mar  6 15:55 gemPlotter.py
-rwxr-xr-x. 1 sturdy zh  6964 Mar  6 15:57 gemSCurveAnaToolkit.py
-rwxr-xr-x. 1 sturdy zh 16859 Mar  6 15:57 gemTreeDrawWrapper.py
-rw-r--r--. 1 sturdy zh     0 Mar  6 15:46 __init__.py
-rw-r--r--. 1 sturdy zh  1156 Mar  7 15:45 overlay_fit.py
-rwxr-xr-x. 1 sturdy zh  8028 Mar  6 15:57 plot_eff.py
-rwxr-xr-x. 1 sturdy zh   390 Mar  6 15:57 plot_noise_vs_trim.py
-rw-r--r--. 1 sturdy zh   651 Mar  6 15:46 plotoptions.py
-rw-r--r--. 1 sturdy zh  1092 Mar  6 15:57 plot_scurves_by_thresh.py
-rwxr-xr-x. 1 sturdy zh   376 Mar  6 15:57 plot_vfat_and_channel_Scurve.py
-rwxr-xr-x. 1 sturdy zh   252 Mar  6 15:57 plot_vfat_summary.py
-rw-r--r--. 1 sturdy zh  5211 Mar  6 15:57 scurvePlottingUtitilities.py
-rwxr-xr-x. 1 sturdy zh  4230 Mar  6 15:57 summary_plots.py

By which I mean

  • Are they scripts that must be in the PATH?
  • Are they imported in other tools?
  • Are they a mishmash of these?

@jsturdy
Copy link
Contributor Author

jsturdy commented Mar 12, 2018

Replying to @bdorney's comment, those bugs are not connected to this development.
Adding this step will allow for an slc6+root-5.34 RPM to be built (which will be functional and is a good thing) as well as a currently broken cc7+default root RPM (which doesn't have to be installed anywhere)

@bdorney
Copy link
Contributor

bdorney commented Mar 13, 2018

Same question here as in vfatqc-python-scripts:

In the RPM structure above how are you handling: $GEM_PLOTTING_PROJECT/mapping/chamberInfo.py?

Here ana_scans.py requires the dictionaries that are defined and filled in this file. Additionally, it is expected that the user is able to change the contents of those dictionaries. While this will most likely not occur often for P5 in the present setup of the software it will most certainly be changing for QC8.

@bdorney
Copy link
Contributor

bdorney commented Mar 13, 2018

What is the usage of these:
...
...
By which I mean
Are they scripts that must be in the PATH?

The following should be in the $PATH:

  • summary_plots.py,
  • plot_vfat_summary.py,
  • plot_vfat_and_channel_Scurve.py,
  • plot_scurves_by_thresh.py (might need to add executable permissions),
  • plot_noise_vs_trim.py,
  • gemTreeDrawWrapper.py,
  • gemPlotter.py,
  • gemPlotterAllChannels.sh,
  • gemSCurveAnaToolkit.py, and
  • plot_eff.py.

Are they imported in other tools?

The following files have pieces that are imported by other tools:

  • plotoptions.py,
  • scurvePlottingUtitilities.py, and
  • gemTreeDrawWrapper.py.

Are they a mishmash of these?

Yes as shown above.

@jsturdy
Copy link
Contributor Author

jsturdy commented Mar 13, 2018

Let me clarify a bit:

  • Which of these will a user call directly?
    • These are the ones that I would like to wind up in /opt/cmsgemos/bin
  • Which of these are called indirectly, i.e., via another script?
    • These ones I would possibly prefer to end up in /usr/lib/python<ver>/site-packages/gempython/<package>/{macros or bin}, which could be appended (or prepended) to the PATH
    • Mostly, I don't want to clutter up /opt/cmsgemos/bin

Probably a request for a future improvement, but to properly package this as a python package, the separation between modules and scripts/executables should be exact, i.e., nothing that gets imported (from) should stay in a script which is also executed.

See response in cms-gem-daq-project/vfatqc-python-scripts#160 for more info on handling the imports

@bdorney
Copy link
Contributor

bdorney commented Mar 14, 2018

Which of these will a user call directly?

The following files:

  • summary_plots.py,
  • plot_vfat_summary.py,
  • plot_vfat_and_channel_Scurve.py,
  • plot_scurves_by_thresh.py (need to add executable permissions, as a mistake it lacks them),
  • plot_noise_vs_trim.py,
  • gemTreeDrawWrapper.py,
  • gemPlotter.py,
  • gemPlotterAllChannels.sh,
  • gemSCurveAnaToolkit.py, and
  • plot_eff.py.

Which of these are called indirectly, i.e., via another script?

  • plotoptions.py,
  • scurvePlottingUtitilities.py, and
  • The function getStringNoSpecials(...) is imported from the file gemTreeDrawWrapper.py.

Probably a request for a future improvement, but to properly package this as a python package, the separation between modules and scripts/executables should be exact, i.e., nothing that gets imported (from) should stay in a script which is also executed.

Based on this what I understand as your request is that the executable scripts only include a:

if __name__ == '__main__':
    # Code

And do not include additional function definitions. For example in gemTreeDrawWrapper.py the follwoing functions are defined:

  • getStringNoSpecials(...), and
  • getPlotFromTree(...)

You would prefer these functions be in a utilities or other file and imported here?

Do I understand the request correctly: Executables include only a main and no additional definitions/variables/functions/etc...

@bdorney
Copy link
Contributor

bdorney commented Mar 14, 2018

See response in cms-gem-daq-project/vfatqc-python-scripts#160 for more info on handling the imports

Okay we'll move that discussion to that issue in vfatqc-python-scripts.

@jsturdy
Copy link
Contributor Author

jsturdy commented Mar 14, 2018

Do I understand the request correctly: Executables include only a main and no additional definitions/variables/functions/etc...

As a bare minimum, no additional function definitions that are used elsewhere, as it complicates the package organization.
If there's this nested mishmash, and only tools in the same directory cross-pollinate, then having a scripts/macros directory for this type of tools makes sense. It has to then be added to the PATH but can be figured out in the packaging step

@bdorney
Copy link
Contributor

bdorney commented Mar 14, 2018

As a bare minimum, no additional function definitions that are used elsewhere, as it complicates the package organization.

Okay. So we will try to resolve this nested mishmash and try to segment things into two subdirectories: utilities and marcos. Things in utilities would be assumed to be imported by other scripts. Things in macros would be assumed to be executable and need to be placed in $PATH. I assume you would like this strategy to cover more than just the current macros subdirectory:

https://github.com/cms-gem-daq-project/gem-plotting-tools/tree/master/macros

e.g. the other scripts like anaUltra*.py should also be placed in macros.

Would you like this addressed as a hotfix to master for a near-term release or to be added to develop for a longer term release?

@jsturdy
Copy link
Contributor Author

jsturdy commented Mar 14, 2018

e.g. the other scripts like anaUltra*.py should also be placed in macros.

Hmm, I guess in this case the distinction I'm trying to raise is quite muddled...
How I initially imagined it, the "tools" themselves will go in /opt/cmsgemos/bin (here I view ana<blah>.py as the tools).
The gemplotting "scripts" (current random assortment of drawing/display scripts) will live in /usr/lib/python<version>/site-packages/gempython/gemplotting/{macros or bin} which would then be probably added to the PATH

Effectively what I want is to separate the "library" from the "things that use the library"

So probably in the short term, I'll put a select set of tools in /opt/cmsgemos/bin and leave the rest inside the package itself

@bdorney
Copy link
Contributor

bdorney commented Aug 14, 2018

I think it's safe to close this issue. Reopen if still relevant.

@bdorney bdorney closed this as completed Aug 14, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants