Can't submit PR for new experiment

[earcanal]

I'd like to submit a PR for a new experiment. After merging with upstream, I get the following error when I try to push my fork:

Counting objects: 4111, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3573/3573), done.
remote: fatal: pack exceeds maximum allowed size
fatal: The remote end hung up unexpectedlyB | 7.18 MiB/s   
error: pack-objects died of signal 13
error: failed to push some refs to '[email protected]:earcanal/expfactory-experiments.git'

The error message truncates it, but the push is in the GiB range (2.8G .git/objects/pack ) i.e. above the 100M maximum. I realise this is more of a git(hub) question, but I'd be grateful for any pointers on how to submit the PR, as I haven't managed to resolve the large commit issue after reading this, and this SO post.

[vsoch]

hey @earcanal do you want to give a go at contributing your experiment to the Version 2.0 library? After you check our the docs, the contributing instructions are here:

https://expfactory.github.io/expfactory/contribute

You can be the first tester of my tiny-dinosaur-arms-written-docs!

[earcanal]

Haven't quite ported an experiment over to the new world order, but here's my feedback so far on these docs:

1. python -m http.server 9999 assumes Python 3. On Ubuntu I had to run python3 -m http.server 9999.
2. I'm finding it hard to follow Add the Experiment instructions because "the library repository" points to dead link https://expfactory.github.io/library.
3. I need a template index.html to get a working experiment! Took me a while to work out that I could borrow one from test-task.
4. Corollary of previous point is that you need to include any libraries your experiment needs. This confused me a bit as that wasn't needed previously. I think this was some magic performed by expfactory that I only partially understand. Might be worth stating this explicitly by saying that you need to include in your repo any libs referenced by index.html.

[vsoch]

Thanks! Here are some quick responses:

1. Using python as a module we can assume python 3 given that the user is running from inside the container. You are correct, previous versions were python -m SimpleHttpServer 9999
2. Can you show me where this link is?
3. Hmm I think you are misunderstanding. You shouldn't need to touch code/ edit html or even know about it, you just need to familiarize with Singularity and run the image.
4. Yes, truly reproducible means everything is required!

If you like, if you point me to your repository I can update the contribute docs and help with the work to port your experiment. I think it would be better for your learning to try it first, but if the docs aren't good enough then I need to fix them. Let me know your preference!

[vsoch]

oh and I understand what you mean by borrowing from test -task - yes Experiment Factory v2.0 is very different in that you aren't constrained to the framework etc. Your index.html can be any web-friendly thing. In the case of the experiments, I am just using the old jspsych bits to get a starting set.

[earcanal]

Yeah, sorry, I think I need a few more clues. Here's the repo: https://github.com/earcanal/breath-counting-task.

[vsoch]

awesome! No worries this is all very new. I'm going to go for a quick run, but will be back in a bit to work on this. Very excited to try out your experiment! If there are important things I should know in advance, please post them here to save me some wandering-in-confusion time.

[earcanal]

I've run it with 40+ participants, so hopefully instructions are good. You can keep the keyboard on the desk if it's more comfortable than in your lap!

[vsoch]

I couldn't last for 15 minutes of breathing in and out - but holy cow just a few minutes I felt so relaxed! I'm finishing up the repo now, will send you some links to updated docs after!

[vsoch]

Tada! That was super easy! I'm going to give a go at building a container for it now - the experiment is added to the library here:

https://expfactory.github.io/experiments/

Are there other experiments you want to see ported over? To build some kind of battery / mindfullness container? Oh, and I also updated the contribute docs a lot so hopefully it's a lot more clear. It's a slightly different use case to be porting an "old expfactory" experiment to a new one, but I hope I did a good job explaining the details via a general contribution.

https://expfactory.github.io/expfactory/contribute

I'm going to give a go at the container, and will ping you here when I have something to test.

[vsoch]

okey doke! I built you a container if you want to test it out:

https://www.singularity-hub.org/collections/215

I'm working on a Dockerfile (likely not done tonight but in the next few days) for those that have... Windows scream)

[earcanal]

Fantastic. So to make changes, I would now fork https://github.com/expfactory-experiments/breath-counting-task and submit PRs (and I can delete the breath counting repo you started from)?

• s/mindfullness/mindfulness/. I'm guessing I'm unable to fix this in the GitHub topics as I can't touch the repo directly?

I get no data when the breath counting task ends :( Web server reports:

127.0.0.1 - - [09/Nov/2017 09:21:49] "GET /%3Cdiv%20class=centerbox%3E%3Cp%20class%20=%20center-block-text%3EPlace%20your%20attention%20on%20the%20breath.%3C/p%3E%3Cp%20class%20=%20center-block-text%3EPress%20the%20down%20arrow%20after%20your%20first%20exhale%20as%20you%20count%20%22one%22,%20then%20close%20your%20eyes%20and%20continue%20counting%20breaths%20and%20pressing%20the%20correct%20arrow%20key.%3C/p%3E%3C/div%3E HTTP/1.1" 404 -
127.0.0.1 - - [09/Nov/2017 09:44:48] code 501, message Unsupported method ('POST')
127.0.0.1 - - [09/Nov/2017 09:44:48] "POST /save HTTP/1.1" 501 -

More on docs:

• https://expfactory.github.io/expfactory/contribute
  • The dead link I mentioned is "library repository" under Add the Experiment. I found it hard to locate the repository when reading this instructions, is it https://github.com/expfactory/experiments?
  • Explain what CSRF is (I had to look it up, assuming it's https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF))
  • s/remot/remote/

What I'd like to do next is replicate my current study, which raises some questions:

• How do I run surveys now?
• How do I chain experiments and surveys? I think I need a little more narrative on /save and /next when running locally.
• Having seen breath counting ported, I should be able to have a go at porting my Attention Network Task (ANT). This is a modified version of the one from TEF, with no questions at the end asking the participant what they were doing during the task. I have a third variant with no practice block (for a different study where they do the task twice, so know what they're doing the second time). Should these be separate experiments, or should I refactor the ANT to make the questions, and practice block optional?

[vsoch]

It's the middle of the night so I need to go back to sleep, but I have answers for all these things! Here are some quick:

web server 501

We would actually expect 501 if you are running without the endpoint, and on error it (should) just download data to your local machine (akin to test task in the preview). I likely need to test your experiment more thoroughly - could you tell me how to change it so it only takes maybe 10 seconds to do? (aka as short as the test task).

I will make the changes to the contribute page!

How do I run surveys now?

I'm going to just port them all, I actually was working on the first just now :) If you tell me the ones you want, I'll do them first.

How do I chain experiments and surveys?

The container will do it for you, when you select more than one in the portal (the container I built for you only has one, so there is no chaining, but with >1 you would select them at the portal, and then they would be chained.

Custom ANT

Let me think about this. I think what we would want is a set of variables the user can choose when installing the experiment, and they determine how it is built (with or without instruction). To start, go ahead and give a go at starting to port the task. I'll make a repo under expfactory-experiments for you to PR to (and also give you permission to breath counting task so you can change topics, etc.)

More on this tomorrow! TLDR:

• how to run short dummy version of breath counting
• first shot at porting the ANT task

Night!

[vsoch]

And just so I can reproduce your error - are you running your own web server, or the container instance?

[vsoch]

okayt I think I reproduced with test task - you must be running this outside of container, here is what I was able to do:

127.0.0.1 - - [09/Nov/2017 04:48:15] code 404, message File not found
127.0.0.1 - - [09/Nov/2017 04:48:15] "GET /%3Cdiv%20class%20=%20%22shapebox%22%3E%3Cdiv%20id%20=%20%22cross%22%3E%3C/div%3E%3C/div%3E HTTP/1.1" 404 -
127.0.0.1 - - [09/Nov/2017 04:48:34] code 501, message Unsupported method ('POST')
127.0.0.1 - - [09/Nov/2017 04:48:34] "POST /save HTTP/1.1" 501 -

I'll work on a fix for this!

[vsoch]

okay I did a potential fix, and updated the preview https://expfactory-experiments.github.io/breath-counting-task/ The image isn't updated yet (it needs to be rebuild) but let me know if that one works. I'm going to finish up the first survey test next.

[vsoch]

hey @earcanal ! I made you a special testing container, here are complete instructions!

https://expfactory.github.io/expfactory/contribute#test-the-experiment

and you don't need to build it yourself, you can pull from Singularity Hub (instructions in the link above) and the repository is here.

https://www.singularity-hub.org/containers/776

I'm really excited about this - because the test container has the basic expfactory installed in it, so I can actually (in addition to testing) use it as it's own little package manager. So here is pulling the image

singularity pull shub://expfactory/expfactory:test

and then seeing what experiments are available:

$ singularity run expfactory.test
Expfactory Version: 3.0
Experiments
1  adaptive-n-back	https://www.github.com/expfactory-experiments/adaptive-n-back
2  breath-counting-task	https://www.github.com/expfactory-experiments/breath-counting-task
3  test-task	https://www.github.com/expfactory-experiments/test-task
4  tower-of-london	https://www.github.com/expfactory-experiments/tower-of-london

and what I'm going to add is a "build" command, so you would be able to do like:

singularity run expfactory.test build tower-of-london test-task

and then it will generate the container recipe for you!

First I'm going to start working on more porting of the surveys soon - let me know if there are a specific set you are using. Taking a quick break now, back later!

[earcanal]

Good news is the preview with your fix worked for me, in that it downloaded a CSV file. If I understand this correctly, this is amazing as anyone with a browser can run the task (i.e. no local expfactory python plumbing required). In terms of my local use case (I forgot where I wrote this), I can just open experiments (and soon surveys) in browser tabs and have the participant move through them in sequence.

In more confused news, to test my experiment I absolutely was running it outside of a container, for the following reasons:

1. I didn't realise I needed a container
2. I don't understand containers
3. I don't have singularity installed
4. I don't understand singularity
5. I have to upgrade Ubuntu to >16.04 in order to install singularity from a package repo
6. I'm scared

:)

I think we're talking past each other, and I'm struggling to understand the docs, due to my lack of knowledge. Would it be possible to chat/hack online somewhere so I can ask dumb questions? This would probably be a quicker way for me to get up to speed, and for you to get an idea about the kind of overview docs people are likely to need to engage with the overall deployment patterns.

[earcanal]

As you managed to fix file creation, I guess you worked out how to shorten the breath counting task. For completeness:

var taskTime = 900 // 15 minutes in seconds

Surveys I'm currently using:

• State Mindfulness Survey
• Five Facet Mindfulness Survey

[vsoch]

hey @earcanal ! I've totally refactored to use Docker instead, because it's a little more far alone to use with services (and we won't run into some of the issues above). I'm also hoping that the docs are much more clear (and shorter) and thus easier to follow. Specifically, when you said you were scared and didn't understand singularity, this was a huge red flag that prompted this work! So here is a suggested gameplan for you to follow:

• try the getting started with Docker. It's weird at first and then at some point you'll have an epiphany that it makes all of your "dependency woes" go away!
• then when you are ready, try the experiment factory quickstart. It's hugely simplified, and has a lot more functionality for development than the previous.

This is all beta, so please report any and all issues. For example, the experiments (that I've generated) are both saving and saving a file to the local machine, and likely I'm just doing the post incorrectly.

I need a break from working on this, but my next todo will be to fix the bug mentioned above, and I can add those surveys for you. You have a lot of issues open, so I think it might be easiest to close ones related to singularity, and re-open a fresh issue to discuss this updated implementation.

[vsoch]

oup, hold up one second, there is an error rendering the static files (my browser was caching them)

[vsoch]

ok fixed! I think I fixed the download issue too :) It will need just a few minutes to push to docker hub.

[earcanal]

Hi @vsoch. Thanks! I'm also about to go dark on this for a while as I need to develop an new task and run a couple of studies. However, here's some feedback:

I'd got somewhere with Docker previously, so I had a running start this time. That meant that I got breath-counting-task following Quick Start, and read Detailed Start while I was waiting for it to complete :) So I have a fuzzy grasp of the architecture/pattern, and with more than 30 minutes of hacking that should become much clearer.

The questions which immediately come to mind fall under Custom Configuration:

• Experiment output is now JSON rather than CSV. What's left to do to finish the (summary!) use case run study, where output is
  • experiment and survey CSVs containing all participants with a participant column, or
  • analogous database endpoint (worth adding SQLite to the list)?

At this hand-wavy stage, I see myself with R analysis code which can consume data from either CSV(s) or a database in a kind of CI fashion. This is part of a kind of "round-trip" open science replication pattern I have, where I can just point someone (myself initially!) at a repo with the experiment and analysis such that another lab could run the study without having to do any dev.

With a little time I may be able to contribute some code to complete this use case. I'll be back...

[vsoch]

Sounds like better results than after the first time!

Experiment output is now JSON rather than CSV. What's left to do to finish the (summary!) use case run study, where output is experiment and survey CSVs containing all participants with a participant column, or analogous database endpoint (worth adding SQLite to the list)?

(To answer your other question) the number in the data folder corresponds with an anonymous participant id, so 00000 was the first, 00001 would be the second, etc. I think this was something you asked for a while back? It would be up to the experimenter to have a secure file to do the matching.

I'm finishing up a container that will produce the surveys you linked, and I''ll also start to think about customization for the output type and database, It should be fairly easy to do as long as I have a consistent strategy. The "hand to another lab, plug,and go" is one the reasons I like these stupidly simple, flat format "databases," because you can just hand the whole thing over / plug nicely into a program called R.

No worries about "going to dark," I'll update you when I have the survey generation container up to speed!

[vsoch]

hey @earcanal ! Just a quick update - I finished up the surveys you needed, they are added to the table:

https://expfactory.github.io/experiments/

Good luck with your work! Chat more when you get back!