Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor of Codebase #12

Open
wants to merge 76 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
056edbe
update git ignore to ignore environ and profile but track Rbuildignore
Mar 21, 2024
b6c7c50
ignore files and folders flaged by check function
Mar 21, 2024
71cfd70
set up internal data in package and document it. remove geojson
Mar 21, 2024
f242e46
lazy data code added by usethis use_data
Mar 21, 2024
9073cc0
will be used in fucntions for reading tiles
Mar 21, 2024
1cd9bc3
change read from github to natively use internal data
Mar 21, 2024
49339c4
rename bm tiles sf into black marble tiles sf native data
Mar 21, 2024
6e9df51
add new dependencies and suggests
Mar 21, 2024
c769a63
migration of functions developed in shiny black marble repo
Mar 21, 2024
da079b3
add tests via testtaht
Mar 21, 2024
33e08e5
polishing the logic of the utils fucntions and adding clarity where n…
Mar 21, 2024
455f0ea
make intersects function and format some code
Mar 23, 2024
b55ab37
leave only main functions here
Mar 23, 2024
000ad54
pending testing for using fuctions to bm_extract and bm_raster
Mar 23, 2024
cf75004
re documenting the man
Mar 23, 2024
78b1824
miseed :: notation
Mar 23, 2024
e9969be
still fixing namespace errors
Mar 23, 2024
b92f32b
before styelr
Mar 26, 2024
b4b8269
AFter STYLEr
Mar 26, 2024
d163306
looks like bm raster works
Mar 26, 2024
4432db3
check point vnp46a2 works but the others dont
Mar 26, 2024
93b94d7
issue in expand.grid to expand_grid from tidyr
Mar 26, 2024
c5ca75b
MISSING tidyr dependency to make bm raster work for vnp46a2
Mar 26, 2024
b7ec880
was renamed
Mar 26, 2024
65c95d9
rename variable param to bm variable
Mar 31, 2024
2aa8c2a
continue renaming ambbiguos param to black marble vairbel
Mar 31, 2024
cf4a281
made logic explicit to review that out what incorreclty used in the m…
Mar 31, 2024
7300d3e
make quiet tests
Mar 31, 2024
1ce9cac
identify slow operation in create black marble dataset. also fixed ad…
Mar 31, 2024
ef26409
Tests Pass for bm raster!
Mar 31, 2024
4ea1185
styler style pkg changes accepted
Mar 31, 2024
97e2461
plan next steps
Mar 31, 2024
7562a84
add notice when product id is not in 4 types
Mar 31, 2024
6d74356
add VNP46A1 test
Mar 31, 2024
f54f126
VNP46A1 test fails
Mar 31, 2024
6426f49
employ mutate
Mar 31, 2024
1e03c13
documentng benchmark por this case in duckdb
Mar 31, 2024
8efd8d6
add logic for allowing VNP46A1
Apr 1, 2024
0f30325
date was being converted to number causing vnp46a3 to fail
Apr 2, 2024
f6ac517
test bra!
Apr 2, 2024
5be60c8
renamed
Apr 2, 2024
711b053
test pass! bm_raster done
Apr 2, 2024
ff452da
move towards quarto docs
Apr 5, 2024
256b2a0
coalesce functiosn
Apr 5, 2024
c8fdf5c
simplify logic
Apr 5, 2024
9075545
demo webr test
Apr 5, 2024
13a2951
gh pages deploy
Apr 5, 2024
f20ecc5
tidy up bm extract logic to correctly use bm raster in it and pass do…
Apr 5, 2024
a878780
pause Error occurred during download: HTTP 504 Gateway Timeout. added…
Apr 5, 2024
e79140d
bm extract wokrs but returns list isntead of unidifed datafrmae and c…
Apr 5, 2024
3636b8a
ISSUE was that no data was available for vnp46a3 so we need to safegu…
Apr 5, 2024
c80c155
typo
Apr 5, 2024
bc64d4c
update title
Apr 11, 2024
38cc4ce
updatre read me
Apr 11, 2024
68dc1bc
add tst to multiple dats on bm extact
Apr 17, 2024
6478d7e
add tests
Apr 17, 2024
d7255f7
add key tests for later release
Apr 17, 2024
133121b
Implement add_n_pixels function
Apr 17, 2024
860933a
Logic to crrectly download files
Apr 17, 2024
bfa04a6
separete tests for easy of testing
Apr 17, 2024
9378a9b
date_i to ensure proper iteration and saving to files
Apr 18, 2024
439031c
tests pass
Apr 18, 2024
300973b
deplopy gh
Apr 18, 2024
08ce5aa
change test expectation to correctlyc ount files in test folder
Apr 18, 2024
2409c12
correct typo. MAIN TESTS COMPLETED
Apr 18, 2024
8d615d3
all test pass file test_local()
Apr 18, 2024
96245fe
update roxygen
Apr 26, 2024
fa38086
update prnt to cli and add cusotm funciton for quiet to be easy passe…
Apr 26, 2024
7313c7b
update cli to custom inform
Apr 26, 2024
9188122
fixing stuff to get check() passes
Apr 26, 2024
478ca4a
dont run examples
Apr 26, 2024
d231060
run tests
Apr 26, 2024
47d1d36
add cli dependcy for main funcitons and fs suggest for tests
Apr 26, 2024
e2937e8
jiji return to original form
Apr 26, 2024
e67b619
update to R 4.4.0
May 4, 2024
30a391c
test R cmd on github actions and secrets
May 4, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .Rbuildignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
^renv$
^renv\.lock$
^.*\.Rproj$
^\.Rproj\.user$
^\\.Renviron$
^\\.Rprofile$
^docs$
^\.github$
^readme_figures$
^_pkgdown$
^pkgdown$

4 changes: 3 additions & 1 deletion .github/workflows/R-CMD-check.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,11 @@ jobs:

env:
GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
BEARER_NASA_TOKEN: ${{ secrets.BEARER_NASA_TOKEN }}
R_KEEP_PKG_SOURCE: yes

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- uses: r-lib/actions/setup-pandoc@v2

Expand All @@ -47,3 +48,4 @@ jobs:
- uses: r-lib/actions/check-r-package@v2
with:
upload-snapshots: true
build_args: 'c("--no-manual","--compact-vignettes=gs+qpdf")'
7 changes: 5 additions & 2 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
*.DS_Store
*.Rbuildignore
#*.Rbuildignore
*.Rhistory
.Rhistory
docs
.Rprofile
.Renviron

blackmarbler-vignette.html



inst/doc
.Rproj.user

21 changes: 15 additions & 6 deletions DESCRIPTION
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Description: Geographically referenced data and statistics of nighttime lights f
License: MIT + file LICENSE
Encoding: UTF-8
Roxygen: list(markdown = TRUE)
RoxygenNote: 7.2.1
RoxygenNote: 7.3.1
URL: https://worldbank.github.io/blackmarbler/
BugReports: https://github.com/worldbank/blackmarbler/issues
Imports:
Expand All @@ -20,12 +20,21 @@ Imports:
purrr,
lubridate,
tidyr,
raster,
sf,
curl,
exactextractr,
stringr,
httr
httr2,
terra,
cli
Suggests:
geodata,
ggplot2,
knitr
geodata,
ggplot2,
knitr,
leaflet,
testthat (>= 3.0.0),
fs
Depends:
R (>= 2.10)
LazyData: true
Config/testthat/edition: 3
36 changes: 24 additions & 12 deletions NAMESPACE
Original file line number Diff line number Diff line change
@@ -1,16 +1,28 @@
# Generated by roxygen2: do not edit by hand

export(add_n_pixels)
export(apply_scaling_factor_to_viirs_data)
export(bind_extracted_data)
export(bm_extract)
export(bm_raster)
import(dplyr)
import(exactextractr)
import(hdf5r)
import(httr)
import(lubridate)
import(purrr, except = c(flatten_df, values))
import(raster, except = c(union, select, intersect, origin, tail, head, values))

import(readr)
import(sf)
import(stringr)
import(tidyr, except = c(extract))
export(clean_raster_data)
export(convert_h5_to_raster)
export(count_n_obs)
export(create_black_marble_dataset_df)
export(create_raster_from_data_metadata)
export(custom_inform)
export(define_blackmarble_variable)
export(define_raster_name)
export(download_and_convert_raster)
export(download_h5_viirs_sat_image)
export(extract_and_process)
export(extract_bounding_box)
export(extract_daily_data)
export(extract_data_and_metadata_from_hdf5)
export(extract_monthly_data)
export(intersect_bm_tiles)
export(julian_to_month)
export(process_tiles)
export(read_black_marble_csv)
export(remove_fill_value_from_satellite_data)
export(retrieve_and_process_nightlight_data)
Loading