Skip to content

Home

Jump to bottom Edit New page
Bakinam Tarik Essawy edited this page Oct 8, 2015 · 35 revisions

Software Details for the VIC/iRODS Data Pre-Processing Pipeline

Bakinam T. Essawy*
Jonathon L. Goodall*
*Department of Civil and Environmental Engineering, University of Virginia, Charlottesville, Virginia

precipitation.r

processncdcPrcp.r

Description: Process precipitation data from study area.

Shell scripts used: run_convert_prcp.scr Argument file: None

Source code: convertPrcp.cpp Source code language: C++ Compiler used: g++ Operating system: Ubuntu Input files: prcpncdc.inf, one or more weather station files Output files: prcp.daily, prcp.inf

readPreprocDly.r

Description: Formatting the combined data

Shell scripts used: inputPrcp.scr Argument files: prcp.daily, prcp.inf, and basin_prcp.fmt

Source code: read_prec_dly.f Source code language: FORTRAN Compiler used: f77 –o Additional information from the FORTRAN code: This program reads the output from the script previous script and formats the daily precipitation so the regrid program can read them. It uses the daily data and the station information files Output file: prcpncdc.daily

  3. Time of adjustment of the combined data • iRODS sub-rule name: precTobAdj • Shell script: ” prcp_tobAdj.scr” • Argument files needed to execute the shell script: i. prec_tob_adj.input

  1. basin_prcp.fmt
  2. prcp.inf
  3. coop_tob.his
  4. basin_prcp_adj.fmt • Script Language used within the Shell script: Fortran • Script used: prec_tob_adj.f • Compiler: f77 –o • Additional information from the Fortran code: This program reads the output from the read_prec_dly.f program and adjusts the daily precipitation to account for the time of observation for each gauge. This prepares a new file in the same format as the .fmt file for input to the regridding program.

  4. Create mask (ascii file) for the study area • iRODS sub-rule name: createMask • Shell script: ” run_convert_tif_ascii.scr” • There is no argument files needed by this rule • Script Language used within the Shell script: Python • Two python Scripts used: i. convert_tif_ascii.py Converting the DEM.tif to mask_125.asc Libraries used: sys, Image, logging • Input file : DEM.tif ii. tiff2ascii.py

  1. Library used : gdal • Input files:
    a. DEM.tif • Output file: mask_125.asc  
  2. Regridding the precipitation data for the basin: a. iRODS sub-rule name: regridPrcp b. Shell script: ” regrd_prcp.scr” c. There is no argument files needed by this rule This script will store the content of the regridPrcp directory to the tmp directory under the name of prcp.log Output files is : adjprcp_grid.grd  
  3. Estimate monthly precipitation from NCDC data

a. iRODS sub-rule name: mkMonthlyPrcp b. Shell script: ” mk_monthly_ir.scr” c. There is no argument files needed by this rule d. Script Language used within the Shell script: C e. Script used: mk_monthly.c f. Compiler: gcc –o g. Additional information from the Fortran code: This program reads the output file from the regridding program (adjprcp_grid.grd). It then generates monthly values for jan,feb,mar,etc that will be used together with PRISM data to scale the timeseries in each gridcell
• Output files is: mk_monthly • Another shell script was used through the ” mk_monthly_ir.scr” this script is called “run.mk_monthly.scr” • Additional information from the script: Script made by Bernt Matheussen 02.08.98 This script is used to run 'mk_monthly'. The output directory for 'jan.monthly, feb.monthly, etc, must be made before program execution. binflag=1 means data is in 2-byte signed binary formatmult is the data multiplier (saved as data*mult) -- only used for binary • Output directory is called : mk_mnth o monthly.jan o monthly.feb o monthly.mar o monthly.apr o monthly.may o monthly.jun o monthly.jul o monthly.aug o monthly.sep o monthly.oct o monthly.nov o monthly.dec o monthly   7. Estimate monthly PRISM data a. iRODS sub-rule name: mkPrism b. Shell script: “get_prism_ir.scr” c. There is no argument files needed by this rule d. Script Language used within the Shell script: C e. Script used: get_prism.c f. Compiler: gcc –o g. Additional information from the script: This program reads the prism data (C.Daly et al 1993) and converts them into monthly gridded data at a chosen resolution. The program needs a maskfile as input together with the prism rawdata. The rawdata used in this program is located at /can-usa-prism/can_us_prism.jan . These files contains prism data for USA. If other prism rawdatafiles are used, then lines 28 to 38 should be changed in this program. If there are gridcells forexample in Canada that the prism data is not covering. The no_data value will be written in the outputfiles corresponding gridcell . The rescaling program will check for this void value at a later stage and the precip value will not be scaled for this gridcell h. Input files:
i. mask_125.asc ii. prism-rawdata/

  1. can_us_prism.jan
  2. can_us_prism.feb
  3. can_us_prism.mar
  4. can_us_prism.apr
  5. can_us_prism.may
  6. can_us_prism.jun
  7. can_us_prism.jul
  8. can_us_prism.aug
  9. can_us_prism.sep
  10. can_us_prism.oct
  11. can_us_prism.nov
  12. can_us_prism.dec

• Output file is a directory that has the monthly prism data: o prism.jan o prism.feb o prism.mar o prism.apr o prism.may o prism.june o prism.jul o prism.aug o prism.sep o prism.oct o prism.nov o prism.dec

  8. Rescaling the precipitation data a. iRODS sub-rule name: mkRescale b. Shell script: ” rescale_ir.scr” c. There is no argument files needed by this rule d. Script Language used within the Shell script: C e. Script used: rescale.c f. Compiler: gcc –o g. Additional information from the script: This program rescales the gridded data from the program regrid with the ratio of precipvalue in cell * (prism.jan/monthly.jan). This is done to achieve monthly precipitation in each grid cell equal to prism monthly meansInputfiles are the gridded data file, monthly values (mk_monthly) and prism monthly values (get_prism). Output is a new rescaled gridded file that can be turned into the final VIC input files. h. Input files:
i. mask_125.asc ii. mk_mnth/monthly. iii. /prism. iv. adjprcp_grid.grd v. Startyear: 1998 vi. Startmonth:1 vii. Endyear:2007 viii. Endmonth:12 i. Output file: adjprcp_rsm.rsc   9. Process maximum temperature data from study area

a. iRODS sub-rule name: processncdcTmax b. Shell script: “run_convert_tmax.scr” c. There is no argument files needed by this rule d. Script Language used within the Shell script: C++ e. Script used: convertTmax.cpp f. Compiler: g++ –o g. Input files: i. stationinfo.txt ii. Climate directory that includes 17 stations for time series from 1998 till 2007:

  1. US1SCCK0008
  2. US1SCCK0009
  3. US1SCCS0001
  4. US1SCUN0001
  5. US1SCUN0002
  6. US1SCYR0004
  7. US1SCYR0010
  8. US1SCYR0018
  9. US1SCYR0020
  10. US1SCYR0021
  11. USC00381633
  12. USC00385232
  13. USC00386293
  14. USC00387722
  15. USC00389218
  16. USC00389625
  17. USR0000SWHI h. Output files: i. Tmax/ tmax.daily  
  18. formatting the combined data • iRODS sub-rule name: readPreprocTmaxDly • Shell script: “inputTmax.scr” • Argument files used i. tmax.daily ii. tmax.inf iii. basin_tmax.fmt

• Script Language used within the Shell script: Fortran • Script used: read_temp_dly.f • Compiler: f77 –o • Additional information from the script: This program reads the output from the script inputPrcp.scr and formats the daily temperature so the regrid program can read them. Only the .fmt (cmb_prcp.fmt) file is output

• Output file: o basin_tmax.fmt

  1. time of adjustment of the combined data • iRODS sub-rule name: tmaxTobAdj • Shell script: “tmax_tobAdj.scr” • Argument files used i. tmax_tob_adj.input • Script Language used within the Shell script: Fortran • Script used: tmax_tob_adj.f • Compiler: g77 -o • Additional information from the script: This program reads the output from the read_prec_dly.f program and adjusts the daily precipitation to account for the time of observation for each gauge. This prepares a new file in the same format as the .fmt file for input to the regridding program.

• Output file: o tmax_tob_adj.input   12. regridding the maximum temperature data for the basin • iRODS sub-rule name: regridTmax • Shell script: “regridTmax.scr” • No argument files for this sub-rule The shell script regrid/Tmax/*o , then the regrdTmax will be stored in the tmp directory under tmax.log.   13. process minimum temperature data from study area

• iRODS sub-rule name: processncdcTmin • Shell script: “run_convert_tmin.scr” • No argument files used for this sub-rule • Script Language used within the Shell script: C++ • Script used: convertTmin.cpp • Compiler: g++ -o • Input files: i. stationinfo.txt ii. tminncdc.inf . iii. climate/ • Output file: o tmin.daily o tmin.inf   14. process minimum temperature data from study area

• iRODS sub-rule name: readPreprocTminDly • Shell script: “inputTmin.scr” • Argument files: i. tmin.daily ii. tmin.inf iii. basin_tmin.fmt • Script Language used within the Shell script: Fortran • Script used: read_temp_dly.f • Compiler: f77 –o • Additional information from the script: This program reads the output from the script inputPrcp_tmax/tmin.scr and formats the daily temperature so the regrid program can read them Only the *.fmt (cmb_prcp.fmt) file is output • Input files: i. tmin.daily ii. tmin.inf

• Output file: o basin_tmin.fmt   15. time of adjustment of the combined data • iRODS sub-rule name: tminTobAdj • Shell script: “tmin_tobAdj.scr” • Argument files: i. tmin_tob_adj.input • Script Language used within the Shell script: Fortran • Script used: read_temp_dly.f • Compiler: f77 –o • Additional information from the script: • This program reads the output from the read_prec_dly.f program and adjusts the daily precipitation to account for the time of observation for each gauge. This prepares a new file in the same format as the .fmt file for input to the regridding program. • Input files: i. tmin.daily ii. tmin.inf

• Output file: o basin_tmin.fmt   16. regridding the minimum temperature data for the basin • iRODS sub-rule name: regridTmin • Shell script: “regrd_tmin.scr” • No argument files for this sub-rule • The shell script regrid/Tmin/*o , then the regrdTmin will be stored in the tmp directory under tmin.log   17. run meteorological data (prcp_grid.rsc, tmax_grid.grd and tmin_grid.grd) to create gridded data for the study area.

o iRODS sub-rule name: vicInputMeteoData o Shell script: “run_vicinput_ir.scr” o No argument files for this sub-rule o Script Language used within the Shell script: C o Script used: vicinput.c o Compiler: gcc -o o Additional information from the script: o This program reads the output from A.H.s regridding program and generates the vicinput files. The input needed is prcp, tmin, tmax,maskfile, outp_directory, binary_flag o Additional script is used to set the input files script: run_vicinput.scr o Input files: • adjprcp_rsm.rsc as prcp • adjtmax_grid.grd as tmax • adjtmin_grid.grd as tmin • mask_125.asc as e_msk

• Outputs: o binflag=0 o vic_err.txt o Dirctory meteoCombined/ • data_34.4995_-81.2093 • data_34.6245_-81.2093 • data_34.6245_-81.3343 • data_34.6245_-81.4593 • data_34.6245_-81.5843 • data_34.6245_-81.7093 • data_34.7495_-81.2093 • data_34.7495_-81.3343 • data_34.7495_-81.4593 • data_34.7495_-81.5843 • data_34.7495_-81.7093 • data_34.8745_-81.0843 • data_34.8745_-81.2093 • data_34.8745_-81.3343 • data_34.8745_-81.4593 • data_34.8745_-81.5843 • data_34.9995_-81.3343 • data_34.9995_-81.4593 • data_34.9995_-81.5843   18. collect wind speed data from NCEP/NCAR • iRODS sub-rule name: getNCARwind • Shell script: “getwind.scr” • No argument files for this sub-rule • This script sets the limits for the lat and long, output directory, and uses the ncks tool fro conversion of the netCDF data to text o Min latitude = 31 o Max latitude = 40 o Min longitude = -90 o Max longitude = -75 o Startyear is 1998 o End year is 2007 o Output directory is : raw_wind

• Input files: • gauss_t62_lat.list • gauss_t62_lon.list

  19. regridding the NCEP/NCAR wind speed data • iRODS sub-rule name: regridNCARwind • Shell script: “run_regrid_wind_ir.scr” • No argument files for this sub-rule • Script Language used within the Shell script: C • Script used: regrid_wind.c • Compiler: gcc -o • Additional information from the script: This program takes the yearly u-wind and v-wind NCEP/NCAR reanalysis data and regrids it from the t62-gaussian grid to a grid corresponding to the input mask file. The required inputs are those listed in the "run_regrid_wind.scr" file, and the files "gauss_t62_lat.list" and "gauss_t62_lon.list" must reside in the same directory as this file. These two files have the global lat/long coordinates of the reanalysis data. The output is one wind timeseries file for each grid cell active in the mask. This program also dumps two other files as output:

  1. wind_latlong.txt which is a list of file names for output
  2. cell_avg.out which is an average for each of 365 days for each cell These two files are used in the fill_data.c program, used to fill the wind data files with average daily wind speeds for each cell, for the years of VIC data preceding the reanalysis data.
    Input files: • raw_wind directory • gauss_t62_lat.list • gauss_t62_lon.list • Outputs: • wind_latlong.txt • cell_avg.out • Another script is used called run_regrid_wind.scr This script set the input to run the first script: • Start year = 1998 • Start month = 1 • End year = 2007 • End month = 12 • Min lat = 31 • Max lat = 40 • Min long = -90 • Max long = -75 • Mask = mask_125.asc • Input directory = raw_wind • Output directory = temp_wind • gauss_t62_lat.list • gauss_t62_lon.list

  20. Combine the wind data with meteorological data • iRODS sub-rule name: combineWind • No argument files for this sub-rule • The script that sets input and output directory is: run_combine_wind.scr • Input directory: meteoCombined/
This is set to be the met_file_path • Input directory: temp_wind/ This is set to be the wind_file_path • latlong99.txt is set to be Latlong • Output directory is forcingData/

• This script combine the wind and the met file path and the latlong and append it to the output directory. • Shell script: “run_combine_wind_ir.scr” • Script Language used within the Shell script: C • Script used: combine_wind.c • Compiler: gcc -o • Additional information from the script: requires input paths and a filename list from vicinput.c   21. Adding preproc and append data into a combined file. o iRODS sub-rule name: latlonSoil o Shell script: “ldas_latlon.scr” o Argument files: • mask_125.asc • latlon.txt o Script Language used within the Shell script: C o Script used: mask2latlon.c o Compiler: gcc –o o The script will output latlon.txt   22. prepare soil data o iRODS sub-rule name: ldasSoil o Shell script: “ldas_soil.scr” o Argument files: • soil_ldas.txt • latlon.txt • ldassoildata.txt o Script Language used within the Shell script: C o Script used: create_LDAS_soil_nearest.c o Compiler: gcc –o o Additional Information from the script: Create a VIC soil file with the LDAS soil hydraulic parameters */ Takes two input files:

  1. LDAS domain soil file
  2. file containing latitude and longitude for each grid cell NOTE: soil files will be in the 53 column format of VIC 4.0 with 3 soil layers • The script will output ldassoildata.txt  
  3. prepare vegetation data o iRODS sub-rule name: ldasVeg o Shell script: “ldas_veg.scr” o Argument files: • ldas_lai.expanded.vegparams • ldassoildata.txt • ldasvegdata.txt o Script Language used within the Shell script: C o Script used: create_LDAS_soil_nearest.c o Compiler: gcc –o o Additional Information from the script: Extracts a vegetation parameter file to match the input soil parameter file. Set up for 1/8 degree LDAS VIC implementation set for 3 root zones and monthly LAI in vegetation parameter file. o Output file: ldasvegdata.txt
Add a custom footer
Add a custom sidebar
Clone this wiki locally