Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance Results #160

Closed
troygrosfield opened this issue Aug 14, 2014 · 2 comments
Closed

Performance Results #160

troygrosfield opened this issue Aug 14, 2014 · 2 comments

Comments

@troygrosfield
Copy link

This isn't as much of an issue as it is simply to be informative about performance results on form rendering for this repo.

The script goes through and times the rendering of 1000 forms.

The Script

from datetime import datetime

from bootstrap3.templatetags.bootstrap3 import bootstrap_form
from django import forms


class SomeTestForm(forms.Form):
    field_1 = forms.CharField(max_length=20)
    field_2 = forms.CharField(max_length=20)
    field_3 = forms.CharField(max_length=20)
    field_4 = forms.CharField(max_length=20)
    field_5 = forms.CharField(max_length=20)


def create_form():
    initial = {
        'field_1': 'Some random string 1.',
        'field_2': 'Some random string 2.',
        'field_3': 'Some random string 3.',
        'field_4': 'Some random string 4.',
        'field_5': 'Some random string 5.'
    }
    return SomeTestForm(initial=initial)

# Test rendering 1000 forms
forms = [create_form() for f in range(1000)]

# Test django forms
start_time_django_rendering = datetime.utcnow()

for form in forms:
    form.as_table()

end_time_django_rendering = datetime.utcnow()
total_django_rendering_time = (end_time_django_rendering - start_time_django_rendering)

print('Total django rendering time: {0}'.format(total_django_rendering_time))

# Test this repo
start_time_bootstrap = datetime.utcnow()

for form in forms:
    bootstrap_form(form)

end_time_bootstrap = datetime.utcnow()
total_bootstrap_time = end_time_bootstrap - start_time_bootstrap

print('Total bootstrap rendering time: {0}'.format(total_bootstrap_time))

Results
Run 1

Total django rendering time:    0:00:01.321577
Total bootstrap rendering time: 0:00:02.189255

Run 2

Total django rendering time:    0:00:01.343208
Total bootstrap rendering time: 0:00:02.226830

Run 3

Total django rendering time:    0:00:01.323405
Total bootstrap rendering time: 0:00:02.235848

Run 4

Total django rendering time:    0:00:01.324829
Total bootstrap rendering time: 0:00:02.209562

Run 5

Total django rendering time:    0:00:01.329026
Total bootstrap rendering time: 0:00:02.216501

I ran the same tests against the django-bootstrap-form repo and they didn't do as well:

@troygrosfield
Copy link
Author

@dyve, feel free to close this once you've seen it. This repo does a pretty good job with rendering performance!

@dyve
Copy link
Member

dyve commented Aug 19, 2014

That's great @troygrosfield, thanks! If you have any tips for better performance, you are most welcome to share them.

@dyve dyve closed this as completed Aug 19, 2014
Elarnon added a commit to mangaki/mangaki that referenced this issue Mar 30, 2017
Allow users without an existing account to import anonymous ratings upon
account creation.

The backend logic behind this is relatively simple and simply requires
overriding django-allauth's signup view to move the anonymous ratings from the
session to the database, associating them to the freshly created user.

On the frontend side, this adds:
 - On the signup page, if the user has made any anonymous ratings, we add a
   checkbox allowing them to import their ratings (active by default), as well
   as a condensed view of their existing ratings. This leverages the
   works_no_poster.html view and allows the user to change their anonymous
   ratings directly on the signup page if they wish to do so.

 - On the login page, if the user has made any anonymous ratings, we add a
   warning message telling them that they will lose their anonymous ratings
   upon login (even though technically, they lose it upon logging out) and that
   they should create a new account if they want to keep those ratings. In the
   future, this should be replaced with a mechanism to merge existing anonymous
   ratings with the user's existing ratings.

As a byproduct, this patch includes a slight re-design of the login and signup
pages, and changes the library we use for displaying bootstrap forms from
django-bootstrap-form to django-bootstrap3. django-bootstrap3 provides better
hooks to customize forms and made said re-design much easier than it would have
been with django-bootstrap-form (in addition, it is
[faster](zostera/django-bootstrap3#160))
Elarnon added a commit to mangaki/mangaki that referenced this issue Apr 7, 2017
Allow users without an existing account to import anonymous ratings upon
account creation.

The backend logic behind this is relatively simple and simply requires
overriding django-allauth's signup view to move the anonymous ratings from the
session to the database, associating them to the freshly created user.

On the frontend side, this adds:
 - On the signup page, if the user has made any anonymous ratings, we add a
   checkbox allowing them to import their ratings (active by default), as well
   as a condensed view of their existing ratings. This leverages the
   works_no_poster.html view and allows the user to change their anonymous
   ratings directly on the signup page if they wish to do so.

 - On the login page, if the user has made any anonymous ratings, we add a
   warning message telling them that they will lose their anonymous ratings
   upon login (even though technically, they lose it upon logging out) and that
   they should create a new account if they want to keep those ratings. In the
   future, this should be replaced with a mechanism to merge existing anonymous
   ratings with the user's existing ratings.

As a byproduct, this patch includes a slight re-design of the login and signup
pages, and changes the library we use for displaying bootstrap forms from
django-bootstrap-form to django-bootstrap3. django-bootstrap3 provides better
hooks to customize forms and made said re-design much easier than it would have
been with django-bootstrap-form (in addition, it is
[faster](zostera/django-bootstrap3#160))
jilljenn pushed a commit to mangaki/mangaki that referenced this issue Jul 2, 2017
* requirements: move pandas, sklearn to production due to DPP recent introduction (#262)

* Link to meta.mangaki.fr deleted (#266)

* Add all recommendation algorithms to production (#265)

* Unify KNN

* Add __init__.py in test folder

* Allow ANY algorithm in production

* Take reviews into account

* Last changes

* Create Dataset class and fit_algo management command (#268)

* Factorize algorithms, create Dataset class and fit_algo management command

* Fix bug

* Remove useless code

* Add migration (#269)

* Add new class RecommendationAlgorithm and refactor algorithms (#270)

* Add new class RecommendationAlgorithm and refactor algorithms

* Fix typo, BTW this PR fixes #267

* Add comment

* Cleanup

* nb_users and nb_works are instance attributes

* Add a template for settings.ini (#272)

* Remove first_name and last_name from Artist (#271)

* Remove first_name and last_name from Artist

* Remove from model as well

* Enable anonymous user to rate works (#277)

This is a first step towards the resolution of #59. It allows
anonymous users to rate works, and keeps the state of the rated works
across consecutive page loads in Django's session.

For now, the usage of this information is limited: although anonymous
user can rate works, they cannot get recommendations from those
ratings, transfer them to a new or existing account, etc. Support for
these will come in subsequent patches.

* Fix path of pickle files (broken so far) (#273)

* Since TensorFlow 1.0, MangakiWALS models cannot be saved anymore

* Fix pickle path everywhere

* Add minimal test for recommendations

* Fix unused parameter

* Enable anonymous users to get recommendations (#278)

This is a second step towards the resolution of #59. It allows
anonymous users to get recommendations (using the KNN algorithm only,
since we do not have trained models for them).

This is implemented by making the various algorithms take directly a
list or (user_id, work_id, choice) triplets instead of building those
from a queryset, as well as some glue to add the relevant triplets
from anonymous users at the interface between the database and the
algorithms themselves.

* Update seed data (unreviewed)

The seed_data.json was not updated after first_name and last_name were
removed from Artist. This fixes that.

* Simplify circle-ci integration (#281)

* Simplify circle config

* Fix matplotlib.pyplot

* Speculative improvements

* relative paths...

* Move coverarc to mangaki

* Try enabling test reports

* Test reports, take 2

* Parallel run is slower

* Remove the Profile.score field (#283)

This provides little information (especially considering how cryptic what it
does is), and the way it is computed gets in the way whenever we want to change
things relative to ratings.

Note that it would be easy to add this again if the needs arises as even though
the value is dynamically updated in various places, it is easy to re-compute it
from the sets of Suggestions and Recommendations the user submitted.

* Completely remove discourse from Mangaki (#282)

We are not using it anymore. Fixes #264.

* Allow import of anonymous ratings on account creation (#279)

Allow users without an existing account to import anonymous ratings upon
account creation.

The backend logic behind this is relatively simple and simply requires
overriding django-allauth's signup view to move the anonymous ratings from the
session to the database, associating them to the freshly created user.

On the frontend side, this adds:
 - On the signup page, if the user has made any anonymous ratings, we add a
   checkbox allowing them to import their ratings (active by default), as well
   as a condensed view of their existing ratings. This leverages the
   works_no_poster.html view and allows the user to change their anonymous
   ratings directly on the signup page if they wish to do so.

 - On the login page, if the user has made any anonymous ratings, we add a
   warning message telling them that they will lose their anonymous ratings
   upon login (even though technically, they lose it upon logging out) and that
   they should create a new account if they want to keep those ratings. In the
   future, this should be replaced with a mechanism to merge existing anonymous
   ratings with the user's existing ratings.

As a byproduct, this patch includes a slight re-design of the login and signup
pages, and changes the library we use for displaying bootstrap forms from
django-bootstrap-form to django-bootstrap3. django-bootstrap3 provides better
hooks to customize forms and made said re-design much easier than it would have
been with django-bootstrap-form (in addition, it is
[faster](zostera/django-bootstrap3#160))

* Enable AniDB testing through mocking using responses library (#285)

* tests: Enable AniDB testing through mocking using responses library

→ Minor fixes in `anidb.py` file (default arguments)
→ Add fixture directory and test fixture directory in settings.
→ Use test fixture directory to feed mocking (fresh fixture from AniDB!)
→ Test `AniDB.search`.
→ Test `AniDB.get_dict`.
→ Add AniDB constants inside the class for convenience purpose.
→ Reformatting and imports optimization.

* Some April cleanup! (#289)

* WorkList: use a property for `is_dpp`

* management command: fix analytics

* knn: fix constructor and reformat according to PEP8.

* chrono: remove unused keyword argument.

* imports: clean unused imports.

* knn: remove the ugly [\n and put a comment on its own line

* knn: moar formatting implementation

* management command: remove analytics

* anonymous-ratings: Review page, clear all ratings, remainder banner (#287)

* anonymous-ratings: Review page, clear all ratings, remainder banner

- Add a remainder banner of anonymous ratings.
- Add a review page by reusing `get_profile` view.
- Add a way to clear all ratings (ugly link)

* chore: reformat for better readability

* views: enforce POST to get CSRF token check and return proper JSON (#293)

* Reformat and remove useless classes in admin (#294)

* admin: reformat code and remove useless classes

* admin: improve code style

* perf/views: select category to prevent performance issue (#292)

* work-list: fetch int_poster also to prevent duplicated queries (#295)

* Update about and events (#286)

* Update about and events

* Add videos

* Fix Soubi and Elarnon

* Fix OpenGraph and Twittercard

* Okay, it was a bad idea

* Remove TensorFlow logging at once

* Let's forget about removing logging

* Factor recommendation algorithms to handle backups (#276)

* Since TensorFlow 1.0, MangakiWALS models cannot be saved anymore

* Fix pickle path everywhere

* Add minimal test for recommendations

* Fix unused parameter

* Add notebook for benchmarks for cold-start

* Fix KNN

* Add LinearRegression for cold-start

* Add MangakiProxyDPP

* Cleanup compare management command

* Edit experiment

* Add computation of delta and update for ALS

* Add BGS from Anava et al.'s WWW 2015 paper

* Fix BGS, withdraw paper from RecSys

* Fix NMF

* Add Autoencoder with TensorFlow

* Now users receive SVD if it exists, otherwise KNN

* Order whole rating list by any algorithm

* Finally manage those damn logs

* Add logging and fix paths

* Fix DPP and clean a lot of code

* Remove useless print

* Oops, forgot one file

* Fix problems linked to merge

* Add an admin interface for merging works (#299)

* Add an admin interface for merging works

* Factor code

* Add minor changes

* Fix bug, and Language to admin

* Take Raito changes into account

* Fix bugs and display number of ratings of each work

* [Hotfix] MAL imports (#301)

* mal-import: refactor the whole code into something more maintainable

- Fix doctor's fields and use MALEntry.
- MAL:
        - Refactor the whole module using a requests.Session and
          instantiating a MALClient which handles the API operations.
        - Rewrite the DB interaction to be more efficient and fix it in
          some edge cases.

* add a default user-agent

* mangaki: fix when MAL import is not available

* mal: add some tests and refactor duplicated / unused code

* mal: rewrite the doctests with None

* mal: add missing fixtures

* Hotfix, KNN did not work

* Improve MAL usage in Mangaki (#302)

* mal-import: refactor the whole code into something more maintainable

- Fix doctor's fields and use MALEntry.
- MAL:
        - Refactor the whole module using a requests.Session and
          instantiating a MALClient which handles the API operations.
        - Rewrite the DB interaction to be more efficient and fix it in
          some edge cases.

* add a default user-agent

* mangaki: fix when MAL import is not available

* mal: add some tests and refactor duplicated / unused code

* mal: rewrite the doctests with None

* mal: add missing fixtures

* MAL: add synonyms and prepare for genres / types

→ Make language a nullable field for WorkTitle.
→ Added the related migration.

* Hotfix profile page with user recommendations

* Hotfix MAL

* [Hotfix] Toggling ratings (+test!) (#306)

* Fix toggle rating and add test

* reducing current_user_set_toggle_rating does too many queries

* Upgrade to Django 1.11

* Revert "Upgrade to Django 1.11"

This reverts commit 4f58bc4.

Meant to ask for review ;-)

* Upgrade to Django 1.11 (#315)

* Add attribute research_ok to Profile (#316)

* Add attribute research_ok to Profile

* Handling tokens

* ADD TESTS §§§

* Add HASH_NACL to Circle settings

* ADD TESTS FOR REAL

* Remove print

* Use built-in Django functions for hashing

* Add token utils

* Salt is not a secret anymore

* Put salt at the correct place

* Fix settings

* Use None instead of 0 for anonymous user identifier.

Previously, we were using 0 for the identifier of an anonymous user,
except that it made us use a `current_user_id` variable for tracking
that and it was easy to just use (erroneously) `request.user.id`
instead.

Since there actually are no requirements on `current_user_id` being a
numeric value, this patch makes it so that we use `None` instead, since
that is the value of `user.id` for an anonymous user, and removes
`current_user_id` altogether.

Fixes #317.

* [Hotfix] Handle recommended ghost works (#323)

* Non-anonymous users don't have anonymous ratings (#324)

While we keep them until the session ends for technical reasons, they
shouldn't be shown.

* Ajout du meta description (#325)

* Add link in profile for sorting any wishlist (#326)

* Fix #322, one could not see their own profile if private

* Add link in profile for sorting any wishlist

* Send templated mail with tokens (#327)

* Fix CSS for footer navbar

* Fix error handling in the research view

* Send mails with tokens

* I mean everyone

* Add jinja2 to production requirements

* Add os.access

* List users in tokens mgmt command

* Only require tensorflow when needed (#329)

TensorFlow is only required by the WALS algorithm, and loading the
library has a prohibitive overhead on lower-powered devices. Let's only
load it when the WALS algorithm is needed, which for now is only when
running the algorithm, which shouldn't be done on the webserver anyways.

* Clarify questions for usage of ratings in Kyoto challenge (#330)

* Kill Doctor management command (#333)

* Better Ansible setup (#321)

This gives a relift to the Ansible setup, allowing several things that
were not included in the previous setup:

 - Possibility to provision a development machine (e.g. Vagrant box)
 - Possibility to easily dump & restore a database
 - Multi-site deployment (deployments are identified by name; except for
   some global nginx and postgresql configuration that shouldn't change
   anyways, care has been taken to ensure all other settings are
   properly isolated so that several deployments with different
   `mangaki_name` values can co-exist on the same machine)
 - git-free deployment; Mangaki can now be built as a PIP package that
   is copied to the remote machine for installation

This deployment is implemented as a single playbook that was made as
declarative as possible; tags and (Ansible's) environment variables
should be used to run the few actions that should be run (collectstatic,
dumping or loading a database, etc.)

* Remove unused test.css file. (#335)

Also, why did we have executable CSS files? WTF.

* Make French great again (space before '!' become  ) (#332)

* add   at good places and make french great again

* typography: add a missing  

* Improve drastically MAL import performance (#311)

* models(work,fts): add full text search fields for Work / WorkTitle

* mal: optimise the hell out of the MAL import procedure

- Use full text search to match all synonyms / titles to prevent
  duplicates.
- Get all existing ratings and then, deduce which ratings to add (not
  fully working, some ratings still appears even though they're already
  in DB!).
- Import your MAL at the speed of the light without any guarantee on
  future violations of theoretical physics.

- Possible improvements: batch full text search queries and deduce a
  subset of works to get from MAL (also figure out a good batch_size).

* migrations: add GIN indexes / triggers / ts vectors fields on Work / WorkTitle

* Address PR comments and correct a test using search

* mal: add some tests using hypothesis

Fix also a bug with empty synonyms fields.

* mal(tests): Use subtests for status code testing

* mal(utils): Explicit the class behind the namedtuple

* migrations: add merge migrations between research OK and MAL data

* Address PR comments

* House-keeping of gitignore and dependencies (#341)

* gitignore: ignore build files when packaging Mangaki

* djdt: bump to 1.8

* Add a data migration to fix MAL external posters (#347)

Move to new MAL's CDN.

* Reduce the amount of queries for ratings selection (#348)

* Load all ratings only at fitting time (recommendations) (#340)

* recommendations: load all ratings if only and only we will fit the algorithm (SVD, ALS, WALS, …)

* Proper handling of algo_name

* Add WorkCluster class for merging works (dedupe, suggestions, or merge from admin) (#307)

* Add WorkCluster to admin

* Add migration

* Allow call to merge from WorkCluster admin

* Add test for merge action

* Add all_objects manager and format_html

* Add migration file for manager

* Butchering migrations

* Make a better test for merging, in order to reduce the number of queries

* Cover all merge functions with tests, reduce the number of queries which was tremendous

* Resolve multiplication of migration leaves

* Cut line

* Use an array rather than string for source_domains on mangaki_dev group vars (#352)

* Kill /users route (#354)

* Kill /users route

* Remove user_list template

* Use timezone-aware now rather than naive datetime (#355)

* [Hotfix] Make posters refreshable once again (fix get_potential_poster) (#357)

* Add get_entry_from_work and fix get_potential_posters for admin

* Address PR comments

* Thanks Django ; I thought I didn't know how to compute in Boole's algebra.

* Remove moar useless comments

* Re-design the work cards (#331)

* Add ribbons displaying card category (#362)

Fixes #141

* Fill Language model and use it with AniDB and MAL (#338)

We add a data migration to support many AniDB and MAL languages, fixing the previous inability of administrators to edit languages of a WorkTitle.

Then, we make MAL import and AniDB API aware of these models so that they can attach more data to our works when operating (e.g. importing, adding new works in the DB).

Finally, we add a new administrator action to refresh WorkTitle of a work linked to AniDB, also its `ext_synopsis` which will be useful for l10n of Mangaki I suppose.

* Add migration for ext_lang / type of WorkTitle (#366)

Missing migration from last PR.

* Rewrite profile view (#364)

This new rewrite gives a huge boost of performance for profiles with a lot of works (1k+).

Also, it improves the privacy of profiles by not displaying information about the count of animes rated, and so on.

Finally, it is paginated and a bit more mobile-friendly that the previous version.

* Integrate Sentry (#350)

Now Mangaki supports Sentry integration, especially for beta.mangaki.fr.

It'll make it easier to track unexpected exceptions on the back-end.

* Add raven>=6.1.0 in setup.py (unreviewed)

* Move to INFO and add console to the root handler (#368)

Fixes Sentry reporting of views exception.

* Hotfix: do not show recommendations tab on anonymous profiles (#369)

* Add i18n in English and Japanese (#356)

Initial i18n of "about us" pages.

* Do not show alternative titles on work detail page (#371)

* Add AniDB to settings template (#372)

* Release the 0.2 (#361)

* s/jp/ja is the good way (#380)

* Forgotten s/jp/ja in base.html template (unreviewed)
collinsmuriuki6kttx added a commit to collinsmuriuki6kttx/itgowo1 that referenced this issue Feb 28, 2022
Allow users without an existing account to import anonymous ratings upon
account creation.

The backend logic behind this is relatively simple and simply requires
overriding django-allauth's signup view to move the anonymous ratings from the
session to the database, associating them to the freshly created user.

On the frontend side, this adds:
 - On the signup page, if the user has made any anonymous ratings, we add a
   checkbox allowing them to import their ratings (active by default), as well
   as a condensed view of their existing ratings. This leverages the
   works_no_poster.html view and allows the user to change their anonymous
   ratings directly on the signup page if they wish to do so.

 - On the login page, if the user has made any anonymous ratings, we add a
   warning message telling them that they will lose their anonymous ratings
   upon login (even though technically, they lose it upon logging out) and that
   they should create a new account if they want to keep those ratings. In the
   future, this should be replaced with a mechanism to merge existing anonymous
   ratings with the user's existing ratings.

As a byproduct, this patch includes a slight re-design of the login and signup
pages, and changes the library we use for displaying bootstrap forms from
django-bootstrap-form to django-bootstrap3. django-bootstrap3 provides better
hooks to customize forms and made said re-design much easier than it would have
been with django-bootstrap-form (in addition, it is
[faster](zostera/django-bootstrap3#160))
collinsmuriuki6kttx added a commit to collinsmuriuki6kttx/itgowo1 that referenced this issue Feb 28, 2022
* requirements: move pandas, sklearn to production due to DPP recent introduction (#262)

* Link to meta.mangaki.fr deleted (#266)

* Add all recommendation algorithms to production (#265)

* Unify KNN

* Add __init__.py in test folder

* Allow ANY algorithm in production

* Take reviews into account

* Last changes

* Create Dataset class and fit_algo management command (#268)

* Factorize algorithms, create Dataset class and fit_algo management command

* Fix bug

* Remove useless code

* Add migration (#269)

* Add new class RecommendationAlgorithm and refactor algorithms (#270)

* Add new class RecommendationAlgorithm and refactor algorithms

* Fix typo, BTW this PR fixes #267

* Add comment

* Cleanup

* nb_users and nb_works are instance attributes

* Add a template for settings.ini (#272)

* Remove first_name and last_name from Artist (#271)

* Remove first_name and last_name from Artist

* Remove from model as well

* Enable anonymous user to rate works (#277)

This is a first step towards the resolution of #59. It allows
anonymous users to rate works, and keeps the state of the rated works
across consecutive page loads in Django's session.

For now, the usage of this information is limited: although anonymous
user can rate works, they cannot get recommendations from those
ratings, transfer them to a new or existing account, etc. Support for
these will come in subsequent patches.

* Fix path of pickle files (broken so far) (#273)

* Since TensorFlow 1.0, MangakiWALS models cannot be saved anymore

* Fix pickle path everywhere

* Add minimal test for recommendations

* Fix unused parameter

* Enable anonymous users to get recommendations (#278)

This is a second step towards the resolution of #59. It allows
anonymous users to get recommendations (using the KNN algorithm only,
since we do not have trained models for them).

This is implemented by making the various algorithms take directly a
list or (user_id, work_id, choice) triplets instead of building those
from a queryset, as well as some glue to add the relevant triplets
from anonymous users at the interface between the database and the
algorithms themselves.

* Update seed data (unreviewed)

The seed_data.json was not updated after first_name and last_name were
removed from Artist. This fixes that.

* Simplify circle-ci integration (#281)

* Simplify circle config

* Fix matplotlib.pyplot

* Speculative improvements

* relative paths...

* Move coverarc to mangaki

* Try enabling test reports

* Test reports, take 2

* Parallel run is slower

* Remove the Profile.score field (#283)

This provides little information (especially considering how cryptic what it
does is), and the way it is computed gets in the way whenever we want to change
things relative to ratings.

Note that it would be easy to add this again if the needs arises as even though
the value is dynamically updated in various places, it is easy to re-compute it
from the sets of Suggestions and Recommendations the user submitted.

* Completely remove discourse from Mangaki (#282)

We are not using it anymore. Fixes #264.

* Allow import of anonymous ratings on account creation (#279)

Allow users without an existing account to import anonymous ratings upon
account creation.

The backend logic behind this is relatively simple and simply requires
overriding django-allauth's signup view to move the anonymous ratings from the
session to the database, associating them to the freshly created user.

On the frontend side, this adds:
 - On the signup page, if the user has made any anonymous ratings, we add a
   checkbox allowing them to import their ratings (active by default), as well
   as a condensed view of their existing ratings. This leverages the
   works_no_poster.html view and allows the user to change their anonymous
   ratings directly on the signup page if they wish to do so.

 - On the login page, if the user has made any anonymous ratings, we add a
   warning message telling them that they will lose their anonymous ratings
   upon login (even though technically, they lose it upon logging out) and that
   they should create a new account if they want to keep those ratings. In the
   future, this should be replaced with a mechanism to merge existing anonymous
   ratings with the user's existing ratings.

As a byproduct, this patch includes a slight re-design of the login and signup
pages, and changes the library we use for displaying bootstrap forms from
django-bootstrap-form to django-bootstrap3. django-bootstrap3 provides better
hooks to customize forms and made said re-design much easier than it would have
been with django-bootstrap-form (in addition, it is
[faster](zostera/django-bootstrap3#160))

* Enable AniDB testing through mocking using responses library (#285)

* tests: Enable AniDB testing through mocking using responses library

→ Minor fixes in `anidb.py` file (default arguments)
→ Add fixture directory and test fixture directory in settings.
→ Use test fixture directory to feed mocking (fresh fixture from AniDB!)
→ Test `AniDB.search`.
→ Test `AniDB.get_dict`.
→ Add AniDB constants inside the class for convenience purpose.
→ Reformatting and imports optimization.

* Some April cleanup! (#289)

* WorkList: use a property for `is_dpp`

* management command: fix analytics

* knn: fix constructor and reformat according to PEP8.

* chrono: remove unused keyword argument.

* imports: clean unused imports.

* knn: remove the ugly [\n and put a comment on its own line

* knn: moar formatting implementation

* management command: remove analytics

* anonymous-ratings: Review page, clear all ratings, remainder banner (#287)

* anonymous-ratings: Review page, clear all ratings, remainder banner

- Add a remainder banner of anonymous ratings.
- Add a review page by reusing `get_profile` view.
- Add a way to clear all ratings (ugly link)

* chore: reformat for better readability

* views: enforce POST to get CSRF token check and return proper JSON (#293)

* Reformat and remove useless classes in admin (#294)

* admin: reformat code and remove useless classes

* admin: improve code style

* perf/views: select category to prevent performance issue (#292)

* work-list: fetch int_poster also to prevent duplicated queries (#295)

* Update about and events (#286)

* Update about and events

* Add videos

* Fix Soubi and Elarnon

* Fix OpenGraph and Twittercard

* Okay, it was a bad idea

* Remove TensorFlow logging at once

* Let's forget about removing logging

* Factor recommendation algorithms to handle backups (#276)

* Since TensorFlow 1.0, MangakiWALS models cannot be saved anymore

* Fix pickle path everywhere

* Add minimal test for recommendations

* Fix unused parameter

* Add notebook for benchmarks for cold-start

* Fix KNN

* Add LinearRegression for cold-start

* Add MangakiProxyDPP

* Cleanup compare management command

* Edit experiment

* Add computation of delta and update for ALS

* Add BGS from Anava et al.'s WWW 2015 paper

* Fix BGS, withdraw paper from RecSys

* Fix NMF

* Add Autoencoder with TensorFlow

* Now users receive SVD if it exists, otherwise KNN

* Order whole rating list by any algorithm

* Finally manage those damn logs

* Add logging and fix paths

* Fix DPP and clean a lot of code

* Remove useless print

* Oops, forgot one file

* Fix problems linked to merge

* Add an admin interface for merging works (#299)

* Add an admin interface for merging works

* Factor code

* Add minor changes

* Fix bug, and Language to admin

* Take Raito changes into account

* Fix bugs and display number of ratings of each work

* [Hotfix] MAL imports (#301)

* mal-import: refactor the whole code into something more maintainable

- Fix doctor's fields and use MALEntry.
- MAL:
        - Refactor the whole module using a requests.Session and
          instantiating a MALClient which handles the API operations.
        - Rewrite the DB interaction to be more efficient and fix it in
          some edge cases.

* add a default user-agent

* mangaki: fix when MAL import is not available

* mal: add some tests and refactor duplicated / unused code

* mal: rewrite the doctests with None

* mal: add missing fixtures

* Hotfix, KNN did not work

* Improve MAL usage in Mangaki (#302)

* mal-import: refactor the whole code into something more maintainable

- Fix doctor's fields and use MALEntry.
- MAL:
        - Refactor the whole module using a requests.Session and
          instantiating a MALClient which handles the API operations.
        - Rewrite the DB interaction to be more efficient and fix it in
          some edge cases.

* add a default user-agent

* mangaki: fix when MAL import is not available

* mal: add some tests and refactor duplicated / unused code

* mal: rewrite the doctests with None

* mal: add missing fixtures

* MAL: add synonyms and prepare for genres / types

→ Make language a nullable field for WorkTitle.
→ Added the related migration.

* Hotfix profile page with user recommendations

* Hotfix MAL

* [Hotfix] Toggling ratings (+test!) (#306)

* Fix toggle rating and add test

* reducing current_user_set_toggle_rating does too many queries

* Upgrade to Django 1.11

* Revert "Upgrade to Django 1.11"

This reverts commit 4f58bc470afb395c22197ac2aecaff1b042a0b75.

Meant to ask for review ;-)

* Upgrade to Django 1.11 (#315)

* Add attribute research_ok to Profile (#316)

* Add attribute research_ok to Profile

* Handling tokens

* ADD TESTS §§§

* Add HASH_NACL to Circle settings

* ADD TESTS FOR REAL

* Remove print

* Use built-in Django functions for hashing

* Add token utils

* Salt is not a secret anymore

* Put salt at the correct place

* Fix settings

* Use None instead of 0 for anonymous user identifier.

Previously, we were using 0 for the identifier of an anonymous user,
except that it made us use a `current_user_id` variable for tracking
that and it was easy to just use (erroneously) `request.user.id`
instead.

Since there actually are no requirements on `current_user_id` being a
numeric value, this patch makes it so that we use `None` instead, since
that is the value of `user.id` for an anonymous user, and removes
`current_user_id` altogether.

Fixes #317.

* [Hotfix] Handle recommended ghost works (#323)

* Non-anonymous users don't have anonymous ratings (#324)

While we keep them until the session ends for technical reasons, they
shouldn't be shown.

* Ajout du meta description (#325)

* Add link in profile for sorting any wishlist (#326)

* Fix #322, one could not see their own profile if private

* Add link in profile for sorting any wishlist

* Send templated mail with tokens (#327)

* Fix CSS for footer navbar

* Fix error handling in the research view

* Send mails with tokens

* I mean everyone

* Add jinja2 to production requirements

* Add os.access

* List users in tokens mgmt command

* Only require tensorflow when needed (#329)

TensorFlow is only required by the WALS algorithm, and loading the
library has a prohibitive overhead on lower-powered devices. Let's only
load it when the WALS algorithm is needed, which for now is only when
running the algorithm, which shouldn't be done on the webserver anyways.

* Clarify questions for usage of ratings in Kyoto challenge (#330)

* Kill Doctor management command (#333)

* Better Ansible setup (#321)

This gives a relift to the Ansible setup, allowing several things that
were not included in the previous setup:

 - Possibility to provision a development machine (e.g. Vagrant box)
 - Possibility to easily dump & restore a database
 - Multi-site deployment (deployments are identified by name; except for
   some global nginx and postgresql configuration that shouldn't change
   anyways, care has been taken to ensure all other settings are
   properly isolated so that several deployments with different
   `mangaki_name` values can co-exist on the same machine)
 - git-free deployment; Mangaki can now be built as a PIP package that
   is copied to the remote machine for installation

This deployment is implemented as a single playbook that was made as
declarative as possible; tags and (Ansible's) environment variables
should be used to run the few actions that should be run (collectstatic,
dumping or loading a database, etc.)

* Remove unused test.css file. (#335)

Also, why did we have executable CSS files? WTF.

* Make French great again (space before '!' become  ) (#332)

* add   at good places and make french great again

* typography: add a missing  

* Improve drastically MAL import performance (#311)

* models(work,fts): add full text search fields for Work / WorkTitle

* mal: optimise the hell out of the MAL import procedure

- Use full text search to match all synonyms / titles to prevent
  duplicates.
- Get all existing ratings and then, deduce which ratings to add (not
  fully working, some ratings still appears even though they're already
  in DB!).
- Import your MAL at the speed of the light without any guarantee on
  future violations of theoretical physics.

- Possible improvements: batch full text search queries and deduce a
  subset of works to get from MAL (also figure out a good batch_size).

* migrations: add GIN indexes / triggers / ts vectors fields on Work / WorkTitle

* Address PR comments and correct a test using search

* mal: add some tests using hypothesis

Fix also a bug with empty synonyms fields.

* mal(tests): Use subtests for status code testing

* mal(utils): Explicit the class behind the namedtuple

* migrations: add merge migrations between research OK and MAL data

* Address PR comments

* House-keeping of gitignore and dependencies (#341)

* gitignore: ignore build files when packaging Mangaki

* djdt: bump to 1.8

* Add a data migration to fix MAL external posters (#347)

Move to new MAL's CDN.

* Reduce the amount of queries for ratings selection (#348)

* Load all ratings only at fitting time (recommendations) (#340)

* recommendations: load all ratings if only and only we will fit the algorithm (SVD, ALS, WALS, …)

* Proper handling of algo_name

* Add WorkCluster class for merging works (dedupe, suggestions, or merge from admin) (#307)

* Add WorkCluster to admin

* Add migration

* Allow call to merge from WorkCluster admin

* Add test for merge action

* Add all_objects manager and format_html

* Add migration file for manager

* Butchering migrations

* Make a better test for merging, in order to reduce the number of queries

* Cover all merge functions with tests, reduce the number of queries which was tremendous

* Resolve multiplication of migration leaves

* Cut line

* Use an array rather than string for source_domains on mangaki_dev group vars (#352)

* Kill /users route (#354)

* Kill /users route

* Remove user_list template

* Use timezone-aware now rather than naive datetime (#355)

* [Hotfix] Make posters refreshable once again (fix get_potential_poster) (#357)

* Add get_entry_from_work and fix get_potential_posters for admin

* Address PR comments

* Thanks Django ; I thought I didn't know how to compute in Boole's algebra.

* Remove moar useless comments

* Re-design the work cards (#331)

* Add ribbons displaying card category (#362)

Fixes #141

* Fill Language model and use it with AniDB and MAL (#338)

We add a data migration to support many AniDB and MAL languages, fixing the previous inability of administrators to edit languages of a WorkTitle.

Then, we make MAL import and AniDB API aware of these models so that they can attach more data to our works when operating (e.g. importing, adding new works in the DB).

Finally, we add a new administrator action to refresh WorkTitle of a work linked to AniDB, also its `ext_synopsis` which will be useful for l10n of Mangaki I suppose.

* Add migration for ext_lang / type of WorkTitle (#366)

Missing migration from last PR.

* Rewrite profile view (#364)

This new rewrite gives a huge boost of performance for profiles with a lot of works (1k+).

Also, it improves the privacy of profiles by not displaying information about the count of animes rated, and so on.

Finally, it is paginated and a bit more mobile-friendly that the previous version.

* Integrate Sentry (#350)

Now Mangaki supports Sentry integration, especially for beta.mangaki.fr.

It'll make it easier to track unexpected exceptions on the back-end.

* Add raven>=6.1.0 in setup.py (unreviewed)

* Move to INFO and add console to the root handler (#368)

Fixes Sentry reporting of views exception.

* Hotfix: do not show recommendations tab on anonymous profiles (#369)

* Add i18n in English and Japanese (#356)

Initial i18n of "about us" pages.

* Do not show alternative titles on work detail page (#371)

* Add AniDB to settings template (#372)

* Release the 0.2 (#361)

* s/jp/ja is the good way (#380)

* Forgotten s/jp/ja in base.html template (unreviewed)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants