Skip to content

error messages 8377 findings recommendations

Jump to bottom Edit New page
Bernard Tyers edited this page Jul 15, 2020 · 1 revision

Findings and recommendations

==NB: Preliminary findings and recommendations.==

Details of testing

  • research took place remotely between 5th-9th June 2020
  • scheduled 17 sessions, 10 took place (good take-up rate)

Method

  • scenario based usability testing (scenario details and Usability testing guide)

  • probed to understand participants' exposure to dependency conflicts

  • setup virtual machine with pip and error message patch

  • sessions were not recorded - written notes were taken with consent

Usability testing objectives

Testing focused on:

  • ResolutionImpossible error message content

  • Possible solution documentation on how to solve it (iteration 1)

Participants

Good variation in professions and level of Python experience. As a sample:

  • a number of professional Python developers (Python user 8+ years)
  • Data scientist (Python user +20 years, package maintainer, Django core developer)
  • Nuclear scientist (Python user +10 years, Gentoo developer)
  • Product engineer (Python user +10 years, maths and physics background)

...continued

  • Devops engineer (Python user 3+ years)
  • Recently graduated engineer (Maintainer of astrophysics Python packages)
  • IT support technician for military organisation (Python user +3 years)

Findings

Error message

  • high-level findings
  • detailed findings

Documentation

  • high-level findings
  • detailed findings

Error message

High-level findings

  1. Users did not understand the meaning of ~=
  2. The possible solution documentation link was lost in the error message
  3. Most users had no understanding of Error: no distribution found message
  4. Format of constraints was difficult to read
  5. Some experienced users asked for concrete actions to be displayed in the error message
  6. Use of red text made error message reduced readability

Detailed findings

==1. Users did not understand the meaning of ~===

  • 15 of 17 did not understand this
    • This clouds understanding of the issue meaning they could not identify the right troubleshooting path
    • This was across all-levels of Python experience and profession

Quotes

I don't know what ~= means. Is it aprox. equal to 2.3.1? My explanation of what has gone wrong will be clouded by that.

- Participant 10 (Product design engineer)

~= around 2.3.1 or similar, 2.3.X, or maybe older. its confusing.

- Participant 11 (Devops engineer)

I'm thinking it could mean "not equal", but that doesn't make sense. [...] I don't remember what ~= means. - Participant 16 (Software engineer with comp. sci PhD)

..continued

==2. The possible solution documentation link was lost in the error message==

  • Due to current error message layout (link runs on 2 lines) 4 users missed the text

..continued

==3. 14+ users had no understanding of Error: no distribution found message==

  • when asked they did not understand or made no mention of the message
  • 3 professional Python developers said there were 2 error messages being displayed (the conflict & distribution not found)

..continued

==4. Format of constraints was difficult to read==

  • 1 participant suggested reording constraints, lowest to highest

reading left-to-right the constraints should be lowest to highest

- participant 12

  • 1 participant mentioned constraints - <3.0.0.,>=2.4.2 were hard to read

could be confusing - syntax of ward not clear its <3.0.0.,>=2.4.2. I had to read it a few times, I didn't see the , between them. There is no reason you couldn't have used the same syntax (for py2neo constraints as) for ward?

- participant 11

..continued

==5. Some experienced users asked for concrete actions to be displayed in the error message==

  • 3 participants asked why error cause 2 solutions couldn't be provided in the error message

..continued

==6. Use of red text made error message difficult to read==

  • 6 participants commented on the use of red making the error message difficult to read

I've noticed that on windows red text on black is not too helpful. I have to use the --no-colour flag.

- participant 10

for gods sake dont use red. You din't know what the background is so dont' use red.

- participant 12

Documentation

High-level findings

  • Most users found documentation useful
  • Over-emphasis on documentation containing "the issue I have right now"
    • Users were looking for consistency between their situation and the documentation
    • We modified the docs (iteration 2) to be generic and it still worked

..continued

  • Providing the operators in the documentation was helpful
    • some felt they were "in the way" of the solution
  • Experienced Python users saw recommendation for forking package (solution 2) seen as:
    • unnecessary by experienced Python users
    • unhelpful and possibly dangerous for unexperienced users

..continued

  • The error causes were verbose but helpful
  • Experienced participants thought link for Dependency Hell was useful (and funny)
  • Force install information seen as confusing
    • Why can't I do it?
    • This might be something I wanted to try
    • If I can't do it, why tell me about it?

Recommendations

  • Error message: recommendations
  • Documentation: recommendations

Error message: recommendations

  1. Include small number of concise suggestions on how to solve this in the error message
    • e.g. 1. loosen your requirements, 2. remove your rquirements and allow pip to figure it out, and
    • provide link to documentation for further reading

Why? This will help experienced users straight away, and will support less-experienced users by providing more context in documentation.

..continued

  1. Include ResolutionImpossible (or another label) in the error message

Why? This will support ALL users with a concise label they can use for searching for more information.

..continued

  1. There is still a strong case for explaining operators in plain language, especially ~=

:::info Why? :::

Participants of ALL levels of experience did not understand them. This meant they could not call on their experience to identify the right solution/way to solve the issue.

At minimum there is a strong case for changing how operators are displayed.

..continued

  1. Improve how the dependencies are displayed :::info Why? ::: ToBeAdded - 1. <,<, 2. left to right - lower to higher

..continued

  1. The layout of the error message needs to be improved
    • remove the Error: no distribution found message
      • this can be dealt with as separate GH Issue
    • reorder the content in the message
      • ToBeAdded

Documentation: recommendations

  1. Remove the "disclaimer" paragraph

:::info Why? ::: Users did not find this discialmer reassuring. They are using an "official" tool, which has brought them to an "official" page, they expect these to be recommended.

..continued

  1. Remove the "forking package and loosening dependencies yourself"

:::info Why? ::: Does not help users. Instead we should focus on explaining to users how to request the package maintainer loosen their dependencies.

..continued

  1. Make sure the URL relates to solving dependency conflicts

:::info Why? ::: A number of participants said they ignore links in the CLI, as they are often generic/unhelpful. After visiting the link, one commented:

Just because I'm not a fan of clicking on links in commandline, doesn't mean I can't learn.

Questions? Comments?

[pulled out to etherpad]

Add a custom footer

Home Resolver error messages Resolution Impossible error message

Clone this wiki locally