Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Put the merged dashboard work live #170

Draft
wants to merge 233 commits into
base: live
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
233 commits
Select commit Hold shift + click to select a range
45cf340
Modify test for incoming commitments
andylolz Mar 5, 2018
4b4582f
Include more types in traceability
andylolz Mar 5, 2018
0a3eb64
Merge branch 'master' into IATI-Dashboard-468-traceability
andylolz Jun 11, 2018
878790b
Linting fixes
andylolz Jul 20, 2018
151de97
Refactor to remove duplication
andylolz Jul 24, 2018
189ffc1
Where set, include condition in codelist path
andylolz Jul 24, 2018
332f478
Merge branch 'master' into 174-codelist-conditions
andylolz Nov 21, 2018
a078c90
Adding linting, preliminary upgrade to Python 3.7
May 7, 2019
ab4e52c
Amending linting errors for CI tests
May 7, 2019
d5b3295
Merge branch 'master' into 174-codelist-conditions
samuele-mattiuzzo May 17, 2019
adfa1a0
workaround map
May 17, 2019
3e399ed
Remove typeerror try catch
May 17, 2019
eb0fb75
Using args in loop tuple mapping
May 21, 2019
b484749
Catch typeerror in aggregate_file
May 21, 2019
f3fceec
Workaround for json dump in aggregate
May 23, 2019
61b40e0
Merge branch 'master' into IATI-Dashboard-468-traceability
andylolz May 28, 2019
ee1794b
Removing deprecated rU from python 3's DictReader
samuele-mattiuzzo Sep 13, 2019
def1ebf
Fixing py3's call to keys() as it now returns an iterable and not list
samuele-mattiuzzo Sep 17, 2019
4a5bd37
Fixing a test that did not make any sense at all
samuele-mattiuzzo Sep 19, 2019
57d1d49
Updated gitignore
samuele-mattiuzzo Sep 19, 2019
bbb081a
Linting
samuele-mattiuzzo Sep 19, 2019
4447b45
Fixing imports
samuele-mattiuzzo Oct 2, 2019
3a7cb2a
Fixing statsrunner's tests
samuele-mattiuzzo Oct 4, 2019
feabf75
Final touches
samuele-mattiuzzo Oct 4, 2019
7959ec4
Merge branch 'master' into python-3-upgrade
samuele-mattiuzzo Oct 4, 2019
bc0ba82
Linting
samuele-mattiuzzo Oct 4, 2019
52b04dc
Linting is the bane of every developer
samuele-mattiuzzo Oct 4, 2019
87662e4
Does this need pushing?
samuele-mattiuzzo Oct 7, 2019
f388814
Merge python upgrade
Oct 10, 2019
84bad0a
Sorting NoneType/null values
Oct 15, 2019
ddcdfb3
more NoneType handling
Oct 15, 2019
3fb7f71
Exceptions for NoneType and null values in budgets
Oct 17, 2019
fdb0765
Formatting json dump output
Oct 18, 2019
d95592d
Not sorting null date values
Oct 24, 2019
a8cf9a5
Merge branch 'master' into 174-codelist-conditions
andylolz Oct 28, 2019
831c604
sum_budget_by_type_by_year issues
Nov 21, 2019
b7e50fb
Fixing budget sum
Nov 25, 2019
3f4db38
Skip invalid years in count_budgets
Nov 25, 2019
30eec1b
Fixing module calls in gitaggregates
Nov 25, 2019
d6ae282
Add recursive date dictionary builder
Nov 27, 2019
9a2a70a
Linting
Nov 27, 2019
9245027
pytest.ini file added
samuele-mattiuzzo Nov 28, 2019
d192ce8
imports fixed
samuele-mattiuzzo Nov 28, 2019
8dbeace
updated gitignore
samuele-mattiuzzo Nov 28, 2019
d1c94bb
linting
samuele-mattiuzzo Nov 28, 2019
f757844
solving the issue around context when running tests vs running script…
samuele-mattiuzzo Nov 28, 2019
20debd5
more linting
samuele-mattiuzzo Nov 28, 2019
86b87d6
Merge pull request #161 from IATI/python-3-upgrade
Feb 6, 2020
1b6bf59
Point at the correct branch of IATI-Rulesets
andylolz Mar 22, 2021
bbedb2e
Move tests to github actions
andylolz Mar 22, 2021
f36bd49
Remove .coverage from version control
andylolz Mar 22, 2021
928b08e
Add a default python gitignore
andylolz Mar 22, 2021
e45a169
Skip all the codelist redirects
andylolz Mar 22, 2021
4af2fd9
Try and sort out test dependencies
andylolz Mar 22, 2021
07868da
Fix README badges
andylolz Mar 22, 2021
45450d7
A couple more README fixes
andylolz Mar 22, 2021
7b934f2
Upgrade lxml
andylolz Mar 22, 2021
ab042d2
Remove some linting exclusions
andylolz Mar 22, 2021
fa60b4f
Omit test files from coverage stats
andylolz Mar 22, 2021
6fd6d2d
Remove requires.io badge from README
andylolz Mar 22, 2021
984019c
Remove ToC from README
andylolz Mar 22, 2021
e710b56
Use codeforIATI fork
andylolz Mar 22, 2021
21c6abf
Refactor slightly
andylolz Mar 22, 2021
e184347
Try running on github actions
andylolz Mar 22, 2021
ff35d08
Fix clone command
andylolz Mar 22, 2021
4bd5c60
Download IATI data
andylolz Mar 22, 2021
fe62d63
Make wget quiet
andylolz Mar 22, 2021
429fd8c
Deploy to github pages
andylolz Mar 22, 2021
b05ee15
Run on cron
andylolz Mar 22, 2021
aa480e4
Commit as codeforIATIbot
andylolz Mar 22, 2021
81cfa99
First go at generating stats
andylolz Mar 22, 2021
be6f852
Don’t run on push
andylolz Mar 22, 2021
890bd44
Move codelist tests to file level
andylolz Mar 23, 2021
b2dd69c
Add CNAME to output
andylolz Mar 23, 2021
c37b774
Partially revert ab4e52ca
andylolz Mar 23, 2021
19310cd
Attempt to fix the dict sorting problem
andylolz Mar 23, 2021
85ed6d7
Generate ckan.json
andylolz Mar 24, 2021
edd0933
Add metadata.json to output
andylolz Mar 24, 2021
40285a3
Fix URL in README
andylolz Mar 24, 2021
c689dac
Oops – leave a copy of ckan.json in helpers
andylolz Mar 24, 2021
632051e
Output to out/current
andylolz Mar 24, 2021
8ecba08
Merge branch 'main' into 174-codelist-conditions
andylolz Mar 25, 2021
a4dfa88
Merge pull request #3 from codeforIATI/174-codelist-conditions
andylolz Mar 25, 2021
e8200fd
Rename license file
andylolz Mar 26, 2021
2ef26f8
Try to get coveralls working again
andylolz Mar 26, 2021
86c839b
Amend copyright notice
andylolz Mar 26, 2021
53b64fd
Store codelist counts against full xpath
andylolz Mar 26, 2021
9ceeb5e
Temporarily switch to andylolz fork for mappings
andylolz Mar 26, 2021
78649ed
Merge pull request #6 from codeforIATI/15-non-default-vocabs
andylolz Mar 26, 2021
4443320
Fix flake8
andylolz Mar 26, 2021
861b6fa
Switch to use JSON mappings
andylolz Mar 28, 2021
47fcf76
Add unofficial mappings
andylolz Mar 28, 2021
05b39a3
Merge pull request #8 from codeforIATI/15-unofficial-vocabs
andylolz Mar 28, 2021
72922df
Add a static directory
andylolz Mar 30, 2021
b461372
Add a custom 404 page
andylolz Mar 30, 2021
dd96355
Change redirect destination for 404 page
andylolz Mar 30, 2021
f68c60b
Revert "Merge pull request #159 from IATI/removing-publisher-stats"
andylolz Apr 3, 2021
ff1c7c0
Linting
andylolz Apr 3, 2021
ef0da30
Relative import
andylolz Apr 3, 2021
a3319e5
Fix some tests
andylolz Apr 3, 2021
9c3062a
Python3 fixes
andylolz Apr 4, 2021
3c66016
Disable various unused stats
andylolz Apr 6, 2021
c08d4db
Fix activities_secondary_reported
andylolz Apr 6, 2021
f2ed2be
Fix decimal imports
andylolz Apr 6, 2021
dd6159d
Merge branch 'main' into 1-add-publishing-statistics
andylolz Apr 6, 2021
4a9c800
Fix datetime imports
andylolz Apr 6, 2021
b4499dd
Merge branch 'main' into 1-add-publishing-statistics
andylolz Apr 6, 2021
2984186
Refactor
andylolz Apr 6, 2021
0b5e5a2
Typos in docstrings
andylolz Apr 6, 2021
508415a
Merge branch 'main' into 1-add-publishing-statistics
andylolz Apr 6, 2021
252dd7a
Point to new repo location
andylolz Apr 7, 2021
c4a486e
Revert "Temporarily switch to andylolz fork for mappings"
andylolz Apr 16, 2021
022c45b
Merge branch 'main' into 1-add-publishing-statistics
Bjwebb Apr 20, 2021
9be481e
Don't invert stats when it would produce a large file
Bjwebb Apr 28, 2021
58b834a
Merge pull request #11 from codeforIATI/1-add-publishing-statistics-e…
andylolz Apr 28, 2021
8100532
Push to IATI-Stats-public instead
andylolz Apr 28, 2021
cd58669
Fix 404 page redirect
andylolz Apr 29, 2021
f628325
Merge pull request #9 from codeforIATI/1-add-publishing-statistics
andylolz Apr 29, 2021
3aa266f
Merge branch 'main' into IATI-Dashboard-468-traceability
andylolz May 3, 2021
6262d39
Workflow tweaks
andylolz May 4, 2021
91cd641
Change github pages location
andylolz May 9, 2021
a383c06
Rename dashboard -> analytics; remove cruft
andylolz May 9, 2021
31ca93c
Merge branch 'main' into IATI-Dashboard-468-traceability
andylolz May 10, 2021
d55446d
Merge pull request #4 from codeforIATI/IATI-Dashboard-468-traceability
andylolz May 10, 2021
750f412
Add activities_with_future_transactions to output
andylolz May 11, 2021
352b885
Update some documentation
andylolz May 11, 2021
8422cd4
Strip activities_with_future_transactions back
andylolz May 12, 2021
d3c60f1
Modify gitaggregate to use git history
andylolz May 12, 2021
5e81be5
Refactor a little bit, for clarity
andylolz May 13, 2021
44c8fb7
Try to fix attribute count for empty attributes
andylolz May 13, 2021
ad8cd99
Merge pull request #13 from codeforIATI/45-blank-attributes
andylolz May 14, 2021
56d3b72
Trim the list of whitelisted stats files
andylolz May 14, 2021
903fefd
Delete data and metadata after loop
andylolz May 15, 2021
ecea139
Trigger a build on the analytics repo
andylolz May 15, 2021
85230f0
Output codelist values
andylolz May 17, 2021
d2b46fb
Run earlier
andylolz May 19, 2021
781d8ab
Add license URL lookup to output
andylolz Jul 20, 2021
c2c6543
Also fetch license titles
andylolz Jul 20, 2021
c356973
Generate activity dates only for humanitarian activities
markbrough Sep 28, 2021
5c1ebb4
Output whether an activity has humanitarian scope with/without the hu…
markbrough Sep 28, 2021
80b871b
Output whether an activity has humanitarian clusters with/without hum…
markbrough Sep 28, 2021
c2d1560
Output whether an activity has humanitarian GLIDE codes with/without …
markbrough Sep 28, 2021
e73f033
Output whether an activity has HRP codes with/without the humanitaria…
markbrough Sep 28, 2021
e983e1e
Merge pull request #14 from codeforIATI/humanitarian-additions
andylolz Sep 28, 2021
a253189
Update link to IATI data dump
andylolz Oct 6, 2021
3cd6386
Use codeforIATI codelists for v2 codelists
andylolz Oct 22, 2021
996ef37
Disable jekyll
andylolz Oct 24, 2021
c468e4f
Fetch v1 codelists from the codeforIATI API
andylolz Nov 3, 2021
e60e1f9
Fix output location
andylolz Nov 3, 2021
b43c40a
Empty commit to trigger tests
andylolz Nov 3, 2021
804046b
Merge pull request #15 from codeforIATI/c4i-v1-codelists
andylolz Nov 3, 2021
1009981
Use C4I fork of IATI-Codelists
andylolz Nov 3, 2021
200d9d7
Increase file size limit to 60MB
andylolz Nov 12, 2021
bd9eada
Merge pull request #18 from codeforIATI/17-file-size
andylolz Nov 12, 2021
6ea000e
Download IATI data from new location
andylolz Nov 16, 2021
a16bb09
Deal with directory structure of iati-data zip
andylolz Nov 17, 2021
6ae799b
Update humanitarian codes - fixes #23
markbrough Jul 26, 2022
2b9b894
Update 5-digit sector codes in tests
markbrough Jul 26, 2022
a092d14
Fix non-humanitarian codes rather than calculating them
markbrough Jul 26, 2022
d32487f
Merge pull request #24 from codeforIATI/update-humanitarian-codes
markbrough Jul 26, 2022
e82cdb6
Don’t build on push
andylolz Sep 20, 2022
669c886
Merge pull request #26 from codeforIATI/25-dont-build-on-push
andylolz Sep 20, 2022
f7e6799
Rename workflow
andylolz Sep 20, 2022
ed894d0
Pushing to dev should build a dev site
andylolz Sep 20, 2022
f7a8c70
Add missing 404.html
andylolz Sep 21, 2022
d4a451d
No need to actually copy this into place
andylolz Sep 21, 2022
2bf9e80
Dev branch should checkout dev output repo
andylolz Sep 21, 2022
5cb3887
Fix broken link in README
andylolz Sep 23, 2022
4ff2829
Run 2 workers for loop in parallel
Bjwebb Sep 20, 2022
f4e3307
Look up the data dump url on codeforiati.org
Bjwebb Sep 23, 2022
336e64b
Merge pull request #28 from codeforIATI/loop-2-workers-and-look-up-da…
Bjwebb Sep 26, 2022
8ce10e5
Calculate publishers spending that is traceable
Bjwebb Sep 26, 2022
cf1c33e
Support 1.0x transaction types in traceability calculations
Bjwebb Sep 27, 2022
a86ab4e
Remove broken exclusion of activities from traceable denominator
Bjwebb Sep 27, 2022
35e9aa9
Remove old exchange rates
andylolz Sep 27, 2022
6474e62
gitignore exchange rates CSV
andylolz Sep 27, 2022
382d305
Fetch C4I exchange rates
andylolz Sep 27, 2022
807dcd7
Reshape exchange rate data
andylolz Sep 27, 2022
d3399ee
Lazy attempt at fixing tests
andylolz Sep 27, 2022
3ba17a7
Merge pull request #30 from codeforIATI/29-imf-exchangerates
andylolz Sep 28, 2022
6067b39
Copy traceability stats to files in publisher directories
Oct 27, 2022
a69d1ea
Remove old exchange rates
andylolz Sep 27, 2022
8b06f5f
gitignore exchange rates CSV
andylolz Sep 27, 2022
c80ebf6
Fetch C4I exchange rates
andylolz Sep 27, 2022
2d31071
Reshape exchange rate data
andylolz Sep 27, 2022
2821590
Lazy attempt at fixing tests
andylolz Sep 27, 2022
c9cd21c
Add downloading OrganisationRegistrationAgency and CRSChannelCode cod…
Oct 20, 2022
7ee7be8
Add activity organisation identifier statistics
Oct 20, 2022
d275977
Add tests for organisation identifier statistics
Oct 20, 2022
3fd882b
Merge pull request #32 from codeforIATI/per_publisher_traceability_files
radix0000 Oct 28, 2022
e89f2ef
Merge pull request #33 from codeforIATI/additional_organisation_ideni…
radix0000 Oct 28, 2022
1790e03
Merge pull request #34 from codeforIATI/dev
radix0000 Nov 3, 2022
10abb7d
Use custom image for github runner to make extra space for build
Nov 4, 2022
77d8ad4
Merge pull request #35 from codeforIATI/extra_space_on_github_runner
radix0000 Nov 4, 2022
08627b0
Add workflow_dispatch event to build
Nov 4, 2022
d483000
Merge pull request #36 from codeforIATI/extra_space_on_github_runner
radix0000 Nov 4, 2022
05fc228
Add a comment as a reminder
andylolz Nov 4, 2022
c2eee4d
End to end tests (just of traceability so far)
Bjwebb Nov 14, 2022
dc14978
Delete a method that we no longer need
Bjwebb Nov 14, 2022
a326f70
Check refs to a publisher's own activities aren't counted
Bjwebb Nov 14, 2022
509f2b1
Test end to end traceability with 1.05 data
Bjwebb Nov 14, 2022
d0ce57a
Update end to end traceability tests to use new currency conversion
Bjwebb Nov 15, 2022
a3eefd5
Merge pull request #37 from codeforIATI/end-to-end-traceability-tests
Bjwebb Feb 14, 2023
0b2ae67
Merge branch 'live' into merge-codeforiati-and-publishingstats
Bjwebb Jul 25, 2024
03b8092
switching php to compressed instead of compress
May 28, 2019
94385f1
Catching more exceptions
May 30, 2019
ea441c5
Create setup script and update README
Jun 20, 2019
6b668f4
Link publishers to dashboard pages, update methodology for year calcu…
Jun 26, 2019
b8cdb63
typo fix and adding methodology change to budget-years
Jun 27, 2019
8b36373
Making sure forwardlooking activities have started to be considered c…
Jul 23, 2019
372c155
Include validation of bnp and add flag to comprehensiveness
Aug 16, 2019
4482065
Pulling Latest Git data to raw repo is now done outside this repository
jarofgreen Aug 4, 2023
182fefb
Changes from live server that were not checked into git
jarofgreen Aug 4, 2023
afc3a72
Bug fix: Check for None in more places
jarofgreen Mar 1, 2024
7825910
wget should retry refused connections
jarofgreen Mar 14, 2024
a83d507
Merge branch 'publishingstats-cherrypicked-filtered' into merge-codef…
Bjwebb Aug 9, 2024
37b2dc5
github actions: Don't calculate the stats on GitHub Actions
Bjwebb Aug 29, 2024
d35af7e
requirements: Switch to pip-tools and update all requirements
Bjwebb Aug 29, 2024
5a61c84
gitaggregate: Revert to IATI/IATI-Dashboard live code
Bjwebb Aug 30, 2024
405da96
gitaggregate: Update IATI-Dashboard code from last commit to Python 3
Bjwebb Sep 1, 2024
51b7c39
git.sh: Add add some missing steps from Code for IATI fork
Bjwebb Aug 30, 2024
50ace80
github actions: Use Python 3.12
Bjwebb Aug 31, 2024
0f0abee
requirements: Install black, isort, flake8-pyproject
Bjwebb Sep 19, 2024
e073579
project.toml: Configure linting with pyproject.toml
Bjwebb Sep 19, 2024
3854134
lint: Run black
Sep 19, 2024
fddff4b
lint: Run isort
Sep 19, 2024
9ff5528
lint: flake8 fixes
Bjwebb Sep 19, 2024
9a2b27c
github actions: Add black and flake8 to the CI tests
Bjwebb Sep 19, 2024
c6133db
Temporarily xfail code for IATI tests broken by IATI-Publishing-Stats
Bjwebb Oct 1, 2024
e65cfa4
helpers/ckan: Revert to use ckan directory from IATI-Registry-Refresher
Bjwebb Nov 5, 2024
7f4f35f
requirements: Update all requirements
Bjwebb Nov 25, 2024
e28105a
git.sh: Stop producing gitaggregate json files with commit hashes in
Bjwebb Dec 6, 2024
2348ffa
codelists: Use official IATI codelists, instead of Code for IATI
Bjwebb Dec 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Don't invert stats when it would produce a large file
  • Loading branch information
Bjwebb committed Apr 28, 2021
commit 9be481e354b7189df7fe0c1f1da11f4c7d80758c
2 changes: 2 additions & 0 deletions statsrunner/invert.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ def invert_dir(basedirname, out_filename, output_dir):
for f in files:
with open(os.path.join(dirname, f)) as fp:
stats_name = f[:-5]
if stats_name in ["iati_identifiers", "by_hierarchy", "bottom_hierarchy"]:
continue
stats_values = json.load(fp)
if type(stats_values) == dict:
if stats_name not in out:
Expand Down