Standard runs #161

marzece · 2018-05-31T23:02:20Z

Add basic standard run editor. I've tested with a local copy of orca and things seem to behave as expected.

tlatorre-uchicago

I've made a few comments. The big thing that I think is necessary is to add some server side verification of the type of the values entered. I think this shouldn't be too hard to add using wtforms which is what is used for the channel database, pmtic resistors, and the mtca+ crate mapping.

tlatorre-uchicago · 2018-06-01T00:42:48Z

minard/views.py

+        new_values = {k:int(v) if type(v)==float and v.is_integer() else v for k,v in new_values.iteritems()}
+        if not new_values:
+            flash("No new values given. Update not performed", "danger")
+        elif(passw != app.config["SECRET_KEY"]):


believe it or not SECRET_KEY is already taken. It's used by flask to generate some sort of cryptographic tokens for sessions.

I know that's why I used it, I thought that was what was used for the WTFForms validate?

No, the password for the channeldb etc. is a password in the postgres database, so minard doesn't handle checking that at all, it just passes it to the database. The SECRET_KEY is a very long string of random characters.

tlatorre-uchicago · 2018-06-01T02:02:20Z

minard/standard_runs.py

+    doc = dict(orca_db.get(uuid))
+    for k, v in new_values.iteritems():
+        doc[k] = v
+    doc["_id"] = uuid4().hex


I just checked how the standard runs are saved in Orca and they don't explicitly include an id version. @mastbaum or @jcaravaca can you confirm that this is kosher?

As far as I know a document always has an id field, if it doesn't couch creates one for a document. The python couchdb library recommends the client create the hash though b/c if the libraries POST request fails and retries you can end up posting the same document multiple times.

Ah ok that makes sense.

tlatorre-uchicago · 2018-06-01T02:03:02Z

minard/standard_runs.py

+    for k, v in new_values.iteritems():
+        doc[k] = v
+    doc["_id"] = uuid4().hex
+    doc["time_stamp"] = time()


Is this the same format that's used by the other standard runs?

yes, the couch view that orca uses sorts by doc.time_stamp so this is the field that should be updated

ah ok. For some reason I thought the timestamps in couch were stored as text.

tlatorre-uchicago · 2018-06-01T02:03:43Z

minard/standard_runs.py

+    doc["_id"] = uuid4().hex
+    doc["time_stamp"] = time()
+
+    if not doc.has_key("run_version") or not bool(doc["run_version"]):


What is the bool check for? Is that just to see if it's zero?

I think that was left over from debugging, I can remove it

tlatorre-uchicago · 2018-06-01T02:04:12Z

minard/standard_runs.py

+    doc["run_type"] = doc["run_type"].upper()
+    doc["run_version"] = doc["run_version"].upper()
+    try:
+        del doc["_rev"]


Might be good to add a comment explaining this.

tlatorre-uchicago · 2018-06-01T02:10:13Z

minard/views.py

+        info = request.form.get("info")
+        passw = request.form.get("password")
+        new_values = [x[:-8] for x in request.form.keys() if x[-8:] == "_replace"]
+        new_values = {x:request.form[x+"_value"] for x in new_values}


The python running on minard is python 2.6 which unfortunately doesn't support this syntax for creating dictionaries.

tlatorre-uchicago · 2018-06-01T02:21:58Z

minard/views.py

+        new_values = {k:True if v.lower() == 'true' else v for k,v in new_values.iteritems() if v}
+        new_values = {k:False if v.lower() == 'false' else v for k,v in new_values.iteritems()}
+        new_values = {k:float(v) if v.isdigit() else v for k,v in new_values.iteritems()}
+        new_values = {k:int(v) if type(v)==float and v.is_integer() else v for k,v in new_values.iteritems()}


I think it would be good to have some server side verification of all of the values here. Do I understand that currently I could add "foo" as the trigger mask and it would get saved as a string? I'm not sure what Orca does in that case. You can use wtforms to do this validation. It's used by the channel database here:

minard/minard/channeldb.py

Line 7 in 926f1b7

class ChannelStatusForm(Form):

and here: https://github.com/snoplus/minard/blob/master/minard/views.py#L372.

I agree that verification would be good, but I didn't want to get to bogged down figuring out whats valid input for all ~50 standard run settings. I figured making it expert only might be good enough because the most common use case for this (I suspect) will be copy/pasting settings from one run to another. That being said I can take a crack at adding validation this weekend, it may not be as time consuming as I expect.

And I looked into using WTFForms, the issue I ran into with that is (I think) it requires you know the fields you want ahead of time, but since couch is so amorphous the fields can change pretty easily. I could just implement a set of forms for the current standard run settings, but that will then mean we have to update this webpage whenever we add/remove some standard run setting. Maybe there's some way to dynamically create forms, if so that'd be the best of both worlds.

tlatorre-uchicago · 2018-06-01T02:26:06Z

minard/standard_runs.py

+    # standard run keys are [doc-version, run name, run version, timestamp]
+    # I want to groupby by run name then within that group all run versions sorted
+    # by timestamp
+    rows = sorted(sr_view.rows, key=lambda x: x.key[1] + x.key[2])


I think here it would probably be best only to show the latest version (or even better have a drop down menu on the standard runs page which allows you to select which standard run version). The reason is that there are a lot of older standard runs from previous versions that don't show up in Orca but do show up on the page and I think that might be confusing.

Ah yeah, I had that at first but I removed it so I could test on old unused runs. I'll add it back in.

marzece · 2018-06-07T19:16:52Z

I think I've addressed all comments now. Let me know if there's anything further. Also this update will require two new settings for the config file. One for the couchdb detector expert username and another for the couchdb run version (which should be 2)

tlatorre-uchicago · 2018-06-08T16:49:07Z

Ok, can you email me the details of the couchdb detector expert name?

One weird issue that I ran into when testing it out. I opened up the "SUPERNOVA - NO_N100" standard run and I tried entering "foo" into the CAEN_coincidenceLevel field and then submitting and I got this error:

MTC_PulserRate: Number must be between 0.04 and 390000

but the form has the pulser rate set to 10.00. In addition, the CAEN coincidenceLevel field now shows "None" as the current value instead of 0.

marzece · 2018-06-10T19:38:13Z

There was some issues creating the new couch user. Should probably work that out before merging. I'm not sure about the MTC_PulserRate/CAEN_Coincedence thing. I'll look into it a bit.

tannerbk · 2018-06-14T04:16:37Z

The TUBII settings are wrong again for most run types after some cable swaps and it would be nice to make the CAEN channel offsets (y-axis) the same for all run-types. Once this is merged I'd suggest we make those changes right away.

marzece · 2018-12-17T22:27:21Z

Andy and I tested this out a bit today and worked out (somewhat) the couch permissions stuff. After some testing this seems to be working well enough, so I think it can be merged now. I mentioned this in an earlier comment but this change requires two new fields in the settings file. Those are as follows
ORCA_STANDARD_RUN_VERSION = 2
COUCH_DETECTOR_EXPERT_NAME = ""
the expert username I'll send offline.

tlatorre-uchicago · 2019-01-17T19:41:23Z

Has the issue I mentioned earlier been fixed? Would also be nice if you could squash all the commits into one.

marzece added 18 commits May 26, 2018 14:30

WIP get started on standard run interface

a5d7bf7

get overview page for standard runs working okay

e7d20eb

add detail view for standard run

53bd774

implement updating standard runs

a9ac6aa

allow for trailing slash in some urls

51f12c4

redo form layout and add meta-info inputs

a885cbc

change comment

ab6bf43

add back button if standard run detail view

b56a205

minor changes

017dc07

add link for standard runs to nav bar

4696ffa

add template for SR overview (which I forgot to add to git)

b61d71c

Change page title

4c48d46

create new documents rather then new versions of existing ones

4e0e5d4

update timestamp for new documents

7d5d4e8

redirect to new standard run upon creation of one

236555e

Add error checking for couchdb interactions

25a0fe1

add some basic validation

b7e01d7

coerce types for new values

90fc799

tlatorre-uchicago reviewed Jun 1, 2018

View reviewed changes

marzece added 4 commits June 1, 2018 13:15

Only display standard runs matching orca's standard run version

d290e21

remove dictionary list comprehesion for python 2.6

b8522de

move standard run password validation to couch DB

f8eacba

reimplement UI using wtforms and add validators

79e41e4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standard runs #161

Standard runs #161

marzece commented May 31, 2018

tlatorre-uchicago left a comment

tlatorre-uchicago Jun 1, 2018

marzece Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

marzece Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

marzece Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

marzece Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

marzece Jun 1, 2018

tlatorre-uchicago Jun 1, 2018

marzece Jun 1, 2018

marzece commented Jun 7, 2018

tlatorre-uchicago commented Jun 8, 2018

marzece commented Jun 10, 2018

tannerbk commented Jun 14, 2018

marzece commented Dec 17, 2018

tlatorre-uchicago commented Jan 17, 2019

Standard runs #161

Are you sure you want to change the base?

Standard runs #161

Conversation

marzece commented May 31, 2018

tlatorre-uchicago left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marzece commented Jun 7, 2018

tlatorre-uchicago commented Jun 8, 2018

marzece commented Jun 10, 2018

tannerbk commented Jun 14, 2018

marzece commented Dec 17, 2018

tlatorre-uchicago commented Jan 17, 2019