Smarter thumbor doctor (thumbor#1393)

* Smarter thumbor doctor

Now users can pass their thumbor.conf file in and thumbor-doctor
validates every extensible part of their configuration.

Makefile

@@ -53,7 +53,7 @@ format:
 	@black .
 
 flake:
-	@flake8 --config flake8
+	@flake8 --config .flake8
 
 pylint:
 	@pylint thumbor tests

README.mkd

@@ -93,7 +93,17 @@ If you experience any troubles, try running:
 thumbor-doctor
 ```
 
-If you still need help, please [raise an issue](https://github.com/thumbor/thumbor/issues).
+If you have a `thumbor.conf` file, you can use that to help thumbor-doctor:
+
+```bash
+thumbor-doctor -c thumbor.conf
+```
+
+If you still need help, please [raise an issue](https://github.com/thumbor/thumbor/issues). Remember to send your `thumbor-doctor` output in the issue:
+
+```bash
+thumbor-doctor --nocolor -c thumbor.conf
+```
 
 ## 🎯 Features
 

pylintrc

@@ -1,3 +1,6 @@
+[MASTER]
+ignore=tests/snapshots
+
 [MESSAGES CONTROL]
 
 # Disable the message, report, category or checker with the given id(s). You

setup.py

@@ -41,6 +41,7 @@
     "redis==3.*,>=3.4.0",
     "remotecv>=2.3.0",
     "sentry-sdk==0.*,>=0.14.1",
+    "snapshottest>=0.6.0,<1.0.0",
     "yanc==0.*,>=0.3.3",
 ]
 

tests/invalid-thumbor.conf

@@ -0,0 +1,48 @@
+# -*- coding: utf-8 -*-
+# thumbor imaging service
+# https://github.com/thumbor/thumbor/wiki
+
+# Licensed under the MIT license:
+# http://www.opensource.org/licenses/mit-license
+# Copyright (c) 2011 globo.com [email protected]
+
+LOADER = "thumbor.loaders.http_loaderer"
+
+UPLOAD_ENABLED = True
+UPLOAD_PHOTO_STORAGE = "thumbor.storages.file_storager"
+
+STORAGE = "thumbor.storages.file_storagee"
+
+RESULT_STORAGE = "thumbor.result_storages.file_storagee"
+
+STORES_CRYPTO_KEY_FOR_EACH_IMAGE = True
+
+ENGINE = "thumbor.engines.pillage"
+
+## detectors to use to find Focal Points in the image
+## more about detectors can be found in thumbor's docs
+## at https://github.com/thumbor/thumbor/wiki
+DETECTORS = [
+    "thumbor.detectors.face_detectorer",
+    "thumbor.detectors.other_invalid",
+]
+
+USE_CUSTOM_ERROR_HANDLING = True
+ERROR_HANDLER_MODULE = "thumbor.error_handlers.sentryer"
+
+FILTERS = [
+    "invalid-filter",
+]
+
+OPTIMIZERS = [
+   "thumbor.optimizers.jpegtraner",
+   "thumbor.optimizers.gifver",
+]
+
+from thumbor.handler_lists import BUILTIN_HANDLERS
+
+HANDLER_LISTS = BUILTIN_HANDLERS + [
+  'my.invalid.handler',
+]
+
+ALLOW_UNSAFE_URL = True

tests/snapshots/snap_test_doctor.py

@@ -0,0 +1,171 @@
+# -*- coding: utf-8 -*-
+# snapshottest: v1 - https://goo.gl/zC4yUc
+from __future__ import unicode_literals
+
+from snapshottest import Snapshot
+
+
+snapshots = Snapshot()
+
+snapshots['test_get_doctor_output 1'] = '''Using configuration file found at ./tests/invalid-thumbor.conf
+
+Thumbor doctor will analyze your install and verify if everything is working as expected.
+
+Verifying libraries support...
+
+✅ pycurl is installed correctly.
+✅ cairosvg is installed correctly.
+✅ cv2 is installed correctly.
+
+Verifying thumbor compiled extensions...
+
+✅ _alpha
+✅ _bounding_box
+✅ _brightness
+✅ _colorize
+✅ _composite
+✅ _contrast
+✅ _convolution
+✅ _curve
+✅ _equalize
+✅ _fill
+✅ _nine_patch
+✅ _noise
+✅ _rgb
+✅ _round_corner
+✅ _saturation
+✅ _sharpen
+
+Verifying extensibility modules found in your thumbor.conf...
+
+❎ thumbor.storages.file_storagee - Storage for source images could not be imported.
+❎ thumbor.loaders.http_loaderer - Loader for source images could not be imported.
+❎ thumbor.result_storages.file_storagee - ResultStorage could not be imported.
+❎ thumbor.engines.pillage - Engine for transforming images could not be imported.
+❎ thumbor.storages.file_storager - Uploading to thumbor is enabled and the Upload Storage could not be imported.
+❎ thumbor.detectors.face_detectorer - Detector could not be imported.
+❎ thumbor.detectors.other_invalid - Detector could not be imported.
+❎ invalid-filter - Filter could not be imported.
+❎ thumbor.optimizers.jpegtraner - Optimizer could not be imported.
+❎ thumbor.optimizers.gifver - Optimizer could not be imported.
+❎ thumbor.error_handlers.sentryer - Custom error handling is enabled and the error handler module could not be imported.
+✅ thumbor.handler_lists.healthcheck
+✅ thumbor.handler_lists.upload
+✅ thumbor.handler_lists.blacklist
+❎ my.invalid.handler - Custom http handler could not be imported.
+
+Verifying security...
+
+❎ Using default security key.
+❎ Allowing unsafe URLs.
+
+😞 Oh no! We found some things that could improve... 😞
+
+⚠️Warnings⚠️
+* Security
+    Error Message:
+        Using default security key configuration in thumbor.conf.
+
+    Error Description:
+        You should specify a unique security key for thumbor or use a command line param to specify a security key.
+\tFor more information visit https://thumbor.readthedocs.io/en/latest/running.html
+
+⛔Errors⛔
+* thumbor.storages.file_storagee
+    Error Message:
+        No module named 'thumbor.storages.file_storagee'
+
+    Error Description:
+        Storage for source images could not be imported.
+
+* thumbor.loaders.http_loaderer
+    Error Message:
+        No module named 'thumbor.loaders.http_loaderer'
+
+    Error Description:
+        Loader for source images could not be imported.
+
+* thumbor.result_storages.file_storagee
+    Error Message:
+        No module named 'thumbor.result_storages.file_storagee'
+
+    Error Description:
+        ResultStorage could not be imported.
+
+* thumbor.engines.pillage
+    Error Message:
+        No module named 'thumbor.engines.pillage'
+
+    Error Description:
+        Engine for transforming images could not be imported.
+
+* thumbor.storages.file_storager
+    Error Message:
+        No module named 'thumbor.storages.file_storager'
+
+    Error Description:
+        Uploading to thumbor is enabled and the Upload Storage could not be imported.
+
+* thumbor.detectors.face_detectorer
+    Error Message:
+        No module named 'thumbor.detectors.face_detectorer'
+
+    Error Description:
+        Detector could not be imported.
+
+* thumbor.detectors.other_invalid
+    Error Message:
+        No module named 'thumbor.detectors.other_invalid'
+
+    Error Description:
+        Detector could not be imported.
+
+* invalid-filter
+    Error Message:
+        No module named 'invalid-filter'
+
+    Error Description:
+        Filter could not be imported.
+
+* thumbor.optimizers.jpegtraner
+    Error Message:
+        No module named 'thumbor.optimizers.jpegtraner'
+
+    Error Description:
+        Optimizer could not be imported.
+
+* thumbor.optimizers.gifver
+    Error Message:
+        No module named 'thumbor.optimizers.gifver'
+
+    Error Description:
+        Optimizer could not be imported.
+
+* thumbor.error_handlers.sentryer
+    Error Message:
+        No module named 'thumbor.error_handlers.sentryer'
+
+    Error Description:
+        Custom error handling is enabled and the error handler module could not be imported.
+
+* my.invalid.handler
+    Error Message:
+        No module named 'my'
+
+    Error Description:
+        Custom http handler could not be imported.
+
+* Security
+    Error Message:
+        Unsafe URLs are enabled.
+
+    Error Description:
+        It is STRONGLY recommended that you turn off ALLOW_UNSAFE_URLS flag in production environments as this can lead to DDoS attacks against thumbor.
+\tFor more information visit https://thumbor.readthedocs.io/en/latest/security.html
+
+❓Need Help❓
+
+If you don't know how to fix the above problems, please open an issue with thumbor.
+Don't forget to copy this log and add it to the description.
+Open an issue at https://github.com/thumbor/thumbor/issues/new
+'''

tests/test_doctor.py

@@ -0,0 +1,26 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# thumbor imaging service
+# https://github.com/thumbor/thumbor/wiki
+
+# Licensed under the MIT license:
+# http://www.opensource.org/licenses/mit-license
+# Copyright (c) 2011 globo.com [email protected]
+
+
+from thumbor.doctor import run_doctor
+
+
+def test_get_doctor_output(snapshot, capsys):
+    run_doctor(
+        {
+            "nocolor": True,
+            "config": "./tests/invalid-thumbor.conf",
+        },
+        print_version=False,
+        exit_with_error=False,
+        check_pyexiv=False,
+    )
+    result = capsys.readouterr()
+    snapshot.assert_match(result.out)