Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement n_hours_per_met_file #102

Merged
merged 7 commits into from
Oct 3, 2024
Merged

implement n_hours_per_met_file #102

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
feee7b1
num hours per met file
jmineau Sep 17, 2024
944e688
only call find_met_files twice if met_subgrid is enabled
jmineau Sep 18, 2024
e93e780
implement met_hours using n_hours_per_met_file
jmineau Sep 19, 2024
f6128a9
call unique on request and output
jmineau Sep 19, 2024
da53c9d
add n_hours_per_met_file param to stilt_cli
jmineau Sep 24, 2024
a92d9fd
n_hours_per_met_file guidance
jmineau Sep 30, 2024
6ab0091
Merge branch 'main' into foward_met_files
John-C-Lin Oct 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ str(receptors)
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `met_path` | Absolute path to ARL compatible meteorological data files |
| `met_file_format` | String detailing file naming convention for meteorological data files using a mixture of datetime and regex syntax. The formatting string accepts `grep` compatible regular expressions (`.\*.arl`), `strftime` compatible datetime strings (`%Y%m%d%H`) or any combination of the two. Datetime syntax is expanded to all unique combinations required for the receptor and simulation duration and the intersection between the requested files and files available in `met_path` is determined with `grep`, allowing partial matching and compatible regular expressions to be used to identify the relevant data. Matching does not require the full format to be specified - e.g. `\*.arl`, `%Y`, `%Y%m%d`, `%Y%m%d_d0.*.arl` would all match with a file named `20180130_d01.arl`. |
| `n_hours_per_met_file` | Number of hours per meteorological data file. Defaults to 6 |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest adding guidance about how to know what value to set, since ARL met files can be a bit of a black box to users

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't had any experience with ARL met files other than the HRRR archive that Lin Group keeps on CHPC. @tartanrunner25 do you have any suggestions for user guidance on determining the number of hours in an ARL met file?

| `met_subgrid_buffer` | Percent to extend footprint area for meteorological subdomain when using `met_subgrid_enable`. Defaults to 0.1 (10%) |
| `met_subgrid_enable` | Enables extraction of spatial subdomains from files in `met_path` using HYSPLIT's `xtrct_grid` binary prior to executing simulations. If enabled, will create files in `<output_wd>/met/`. This can substantially accelerate simulation speed at the cost of increased disk usage. Defaults to disabled |
| `met_subgrid_levels` | If set, extracts the defined number of vertical levels from the meteorological data files to further accelerate simulations. Defaults to `NA`, which includes all vertical levels available |
Expand Down
2 changes: 2 additions & 0 deletions r/run_stilt.r
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ yres <- xres
# Meteorological data input
met_path <- '<path_to_arl_meteorological_data>'
met_file_format <- '%Y%m%d.%Hz.hrrra'
n_hours_per_met_file <- 6
met_subgrid_buffer <- 0.2
met_subgrid_enable <- F
met_subgrid_levels <- NA
Expand Down Expand Up @@ -239,6 +240,7 @@ stilt_apply(FUN = simulation_step,
met_subgrid_levels = met_subgrid_levels,
mgmin = mgmin,
n_hours = n_hours,
n_hours_per_met_file = n_hours_per_met_file,
n_met_min = n_met_min,
ncycl = ncycl,
ndump = ndump,
Expand Down
55 changes: 37 additions & 18 deletions r/src/find_met_files.r
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,61 @@
#' find_met_files searches for meteorological data files
#' @author Ben Fasoli
#' @author Ben Fasoli & James Mineau
#'
#' Searches for available meteorological files matching the given strftime
#' compatible file naming convention
#'
#' @param t_start time of simulation start
#' @param n_hours number of hours to run each simulation; negative indicates
#' backward in time
#' @param n_hours_per_met_file number of hours of meteorological data in each
#' met file
#' @param met_file_format grep compatible file naming convention to identify
#' meteorological data files necessary for the timing of the simulation
#' indicated by \code{t_start} and \code{n_hours}
#' @param n_hours number of hours to run each simulation; negative indicates
#' backward in time
#' @param met_path directory to find meteorological data
#'
#' @import dplyr
#' @export

find_met_files <- function(t_start, met_file_format, n_hours, met_path) {
find_met_files <- function(t_start, n_hours, n_hours_per_met_file,
met_file_format, met_path) {
require(dplyr)


ts <- as.POSIXct(t_start, tz = 'UTC')
is_backward <- n_hours < 0

# TODO: implement n_hours_per_met_file to better determine file names at
# varying time resolutions
request <- as.POSIXct(t_start, tz='UTC') %>%
c(. + c(1, -1, n_hours, is_backward * (n_hours - 5)) * 3600) %>%
jmineau marked this conversation as resolved.
Show resolved Hide resolved
range() %>%
(function(x) seq(x[1], x[2], by = 'hour')) %>%
strftime(tz = 'UTC', format = met_file_format)

met_bracket <- n_hours_per_met_file - 1 # ts can be in the middle of a met file

# Generate the hours to search for
if (is_backward) {
met_hours <- seq(
ts - as.difftime(abs(n_hours) + met_bracket, units = 'hours'),
ts,
by = 3600
)
} else {
met_hours <- seq(
ts - as.difftime(met_bracket, units = 'hours'),
ts + as.difftime(n_hours, units = 'hours'),
by = 3600
)
}

# Format the request and remove duplicates
request <- met_hours %>%
strftime(tz = 'UTC', format = met_file_format) %>%
unique()

# Find the available files
available <- dir(met_path, full.names = T, recursive = T)
available <- available[!grepl('.lock', available)]


# Find the files that match the request
idx <- do.call(c, lapply(request, function(pattern) {
grep(pattern = pattern, x = available)
}))

if (any(idx < 1))
return()
unique(available[idx])

unique(available[idx]) # Available files that match the request
}
29 changes: 16 additions & 13 deletions r/src/simulation_step.r
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ simulation_step <- function(before_footprint = list(function() {output}),
mgmin = 10,
mhrs = 9999,
n_hours = -24,
n_hours_per_met_file = 6,
n_met_min = 1,
ncycl = 0,
ndump = 0,
Expand Down Expand Up @@ -257,33 +258,35 @@ simulation_step <- function(before_footprint = list(function() {output}),
link_files(exe, rundir)

# Find necessary met files
met_files <- find_met_files(r_run_time, met_file_format, n_hours, met_path)
met_files <- find_met_files(r_run_time, n_hours, n_hours_per_met_file,
met_file_format, met_path)
if (length(met_files) < n_met_min) {
msg <- paste('Insufficient number of meteorological files found. Check',
'specifications in run_stilt.r')
warning(msg)
cat(msg, '\n', file = file.path(rundir, 'stilt.log'), append = T)
return()
}

if (met_subgrid_enable) {
met_path <- file.path(output_wd, 'met')
calc_met_subgrids(met_files, met_path, exe,
projection, xmn, xmx, ymn, ymx,
levels = met_subgrid_levels,
met_subgrid_buffer = met_subgrid_buffer)

# Find necessary met files for subgrids
met_files <- find_met_files(r_run_time, n_hours, n_hours_per_met_file,
met_file_format, met_path)
if (length(met_files) < n_met_min) {
msg <- paste('Insufficient number of meteorological files found. Check',
'specifications in run_stilt.r')
warning(msg)
cat(msg, '\n', file = file.path(rundir, 'stilt.log'), append = T)
return()
}
}

# Find necessary met files
met_files <- find_met_files(r_run_time, met_file_format, n_hours, met_path)
if (length(met_files) < n_met_min) {
msg <- paste('Insufficient number of meteorological files found. Check',
'specifications in run_stilt.r')
warning(msg)
cat(msg, '\n', file = file.path(rundir, 'stilt.log'), append = T)
return()
}


# Execute particle trajectory simulation, and read results into data frame
output$receptor <- list(run_time = r_run_time,
lati = r_lati,
Expand Down
Loading