Package data while publishing, update requirements and speed-up build…

…ing (#68)

* Package data while publishing, update requirements and speed-up building.

* Replace use of os.path by pathlib

* convert README.md to README.rst

.travis.yml

@@ -8,8 +8,8 @@ before_install:
   - gem install sass --version "=3.5.3"
 
 install:
-  - pip install -r requirements.txt
-  - pip install '.[test,website]'
+  - pip install -r requirements-dev.txt
+  - pip install '.[dev]'
   - whotracksme --help
 
 script:

README.rst

@@ -0,0 +1,132 @@
+
+.. image:: https://raw.githubusercontent.com/cliqz-oss/whotracks.me/master/static/img/who-tracksme-logo.png
+    :width: 200px
+    :alt: whotracks.me
+    :target: https://whotracks.me
+
+Bringing Transparency to online tracking - built by Cliqz and Ghostery.
+
+-----------------------------------------------------------------------
+
+This repository contains:
+
+-  data on trackers and websites as shown on `whotracks.me`_ (WTM)
+-  database mapping tracker domains to companies
+-  code to render the `whotracks.me`_ site
+
+Installation
+============
+
+Python 3.6 is needed to build the site. We recommend creating a
+`virtualenv`_ (or ``pipenv``) to install the dependencies.
+
+Furthermore, you will need to `install sass`_.
+
+From Pypi
+---------
+
+.. code:: sh
+
+    $ pip install whotracksme
+
+From source
+-----------
+
+.. code:: sh
+
+    $ pip install -e .
+
+That’s all you need to get started!
+
+Using the data
+==============
+
+To get started with the data, everything you need can be found in
+``whotracksme.data``:
+
+.. code:: python
+
+    from whotracksme.data.loader import DataSource
+
+    # available entities
+    DataSource().trackers
+    DataSource().companies
+    DataSource().sites
+
+For examples of scripts, have a look in the `contrib`_ folder!
+
+Building the site
+=================
+
+Building the site requires a few extra dependencies, not installed by
+default to not make the installation heavier than it needs to be. You
+will need to install ``whotracksme`` from the repository, because not
+all assets are packaged with ``whotracksme`` released on pypi:
+
+.. code:: sh
+
+    $ pip install -e '.[dev]'
+
+Once this is done, you will have access to a ``whotracksme`` entry point
+that can be used this way:
+
+.. code:: sh
+
+    $ whotracksme website [serve]
+
+The ``serve`` part is optional and can be used while making changes on
+the website.
+
+All generated artifacts can be found in the ``_site/`` folder.
+
+Tests
+-----
+
+To run tests, you will need ``pytest``, or simply install ``whotacksme``
+with the ``dev`` extra:
+
+.. code:: sh
+
+    $ pip install -e '.[dev]'
+    $ pytest
+
+Publishing a new version
+========================
+
+.. code:: sh
+
+    $ pip install twine
+    $ python setup.py sdist bdist_wheel
+    $ twine upload --username cliqz-oss dist/*
+
+Contributing
+============
+
+We are happy to take contributions on: \* Guest articles for our blog in
+the topics of tracking, privacy and security. Feel free to use the data
+in this repository if you need inspiration. \* Feature requests that are
+doable using the WTM database. \* Curating our database of tracker
+profiles. Open an issue if you spot anything odd.
+
+Right to Amend
+==============
+
+Please read our `Guideline for 3rd parties`_ wanting to suggest
+corrections to their data.
+
+License
+=======
+
+The content of this project itself is licensed under the `Creative
+Commons Attribution 4.0 license`_, and the underlying source code used
+to generate and display that content is licensed under the `MIT
+license`_.
+
+.. _whotracks.me: https://whotracks.me
+.. _virtualenv: http://docs.python-guide.org/en/latest/dev/virtualenvs/
+.. _install sass: http://sass-lang.com/install
+.. _contrib: ./contrib
+.. _Guideline for 3rd parties: https://github.com/cliqz-oss/whotracks.me/blob/master/RIGHT_TO_AMEND.md
+.. _Creative Commons Attribution 4.0 license: https://creativecommons.org/licenses/by/4.0/
+.. _MIT license: https://github.com/cliqz-oss/whotracks.me/blob/master/LICENSE.md
+

requirements-dev.txt

@@ -0,0 +1,43 @@
+aiofiles==0.3.2
+aiohttp==3.1.2
+argh==0.26.2
+async-timeout==2.0.1
+attrs==17.4.0
+certifi==2018.1.18
+chardet==3.0.4
+colour==0.1.5
+decorator==4.2.1
+docopt==0.6.2
+httptools==0.0.11
+idna==2.6
+idna-ssl==1.0.1
+ipython-genutils==0.2.0
+Jinja2==2.10
+jsonschema==2.6.0
+jupyter-core==4.4.0
+Markdown==2.6.11
+MarkupSafe==1.0
+more-itertools==4.1.0
+multidict==4.1.0
+nbformat==4.4.0
+numpy==1.14.2
+pandas==0.22.0
+pathtools==0.1.2
+plotly==2.5.1
+pluggy==0.6.0
+py==1.5.3
+pytest==3.5.0
+python-dateutil==2.7.2
+pytz==2018.4
+PyYAML==3.12
+requests==2.18.4
+sanic==0.7.0
+six==1.11.0
+squarify==0.3.0
+traitlets==4.3.2
+ujson==1.35
+urllib3==1.22
+uvloop==0.9.1
+watchdog==0.8.3
+websockets==4.0.1
+yarl==1.1.1

requirements.txt

@@ -1,35 +1,6 @@
-aiohttp==3.0.4
-aiofiles==0.3.2
-argh==0.26.2
-boto3==1.4.8
-certifi==2017.11.5
-chardet==3.0.4
-colour==0.1.5
-decorator==4.1.2
 docopt==0.6.2
-httptools==0.0.9
-idna==2.6
-ipython-genutils==0.2.0
-Jinja2==2.10
-jsonschema==2.6.0
-jupyter-core==4.4.0
-Markdown==2.6.9
-MarkupSafe==1.0
-nbformat==4.4.0
+numpy==1.14.2
 pandas==0.22.0
-pathtools==0.1.2
-plotly==2.2.2
-py==1.5.2
-pytest==3.2.5
-pytz==2017.3
-PyYAML==3.12
-requests==2.18.4
-sanic==0.6.0
+python-dateutil==2.7.2
+pytz==2018.4
 six==1.11.0
-squarify==0.3.0
-traitlets==4.3.2
-ujson==1.35
-urllib3==1.22
-uvloop==0.8.1
-watchdog==0.8.3
-websockets==4.0.1