update 1.2.0

Documentation/EGADS Algorithm Handbook - LATEX/cover/alg_cover.tex

@@ -24,7 +24,7 @@
 \setlength{\voffset}{.0cm}
 
 \logoeufar
-\title{EUFAR, Olivier Henry}{}{EGADS Lineage Algorithm Handbook}{Version 1.1.2}
+\title{EUFAR, Olivier Henry}{}{EGADS Lineage Algorithm Handbook}{Version 1.2.0}
 %\author{}{}
 
 

Documentation/changelog.txt

@@ -10,6 +10,26 @@ To fix list:
 ------------
 
 
+December 11 2019, Release version 1.2.0
+=======================================
+EGADS can read and write HDF5 file format, following or not EUFAR convention.
+Group function has been added to the NetCdf and EgadsNetCdf classes. The conversion to NasaAmes from NetCdf using the dedicated function is not compatible with groups.
+
+  FIXED:
+    - during the conversion of a netcdf file to a nasa ames file, the normal comment wasn't created properly. It has been fixed in NetCdf and EgadsNetCdf classes.
+	- corrected few logging messages in the NetCdf and EgadsNetCdf classes.
+	- fixed few typos.
+	- corrected documentation of help part in all different classes.
+
+  ADDED:
+    - Hdf and EgadsHdf classes have been implemented to read/write HDF5 files.
+	- Group has been implemented in NetCdf and EgadsNetCdf classes.
+	- tests has been added for Hdf and EgadsHdf.
+
+  MODIFIED:
+    - the documentation has been updated.
+
+
 September 6 2019, Release version 1.1.2
 =======================================
   MODIFIED:

INSTALL.txt

@@ -21,6 +21,8 @@ To use EGADS Lineage, the following packages must be installed:
 
  4. Python netCDF4 libraries 1.3.0 or newer
 
+ 5. H5py 2.10 or newer
+
  5. Quantities 0.12.1 or newer
 
  6. python_dateutil 2.6.1 or newer.

PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 1.2
 Name: egads-lineage
-Version: 1.1.2
+Version: 1.2.0
 Summary: EGADS: EUFAR General Airborne Data-processing Software
 Home-page: https://github.com/EUFAR/egads/tree/Lineage
 Author: EUFAR, Olivier Henry
@@ -26,3 +26,4 @@ Requires: numpy (>=1.14)
 Requires: netCDF4 (>=1.3.0)
 Requires: python_dateutil (>=2.6.1)
 Requires: quantities (>=0.12.1)
+Requires: h5py (>=2.10.0)

README.md

@@ -1,7 +1,7 @@
 Version:
 -------
 
-EGADS Lineage 1.1.2
+EGADS Lineage 1.2.0
 
 
 Developments:
@@ -38,6 +38,7 @@ Features:
 * Fully documented algorithms
 * I/O routines for common file formats:
   * NetCDF
+  * Hdf
   * NASA Ames
   * CSV
   * ASCII

doc/source/conf.py

@@ -48,9 +48,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '1.1.2'
+version = '1.2.0'
 # The full version, including alpha/beta/rc tags.
-release = '1.1.2'
+release = '1.2.0'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

doc/source/example_files/ncdump_hdf_example_file.txt

@@ -0,0 +1,25 @@
+=> ncdump -h main_hdf_file.hdf5                                               
+   netcdf main_hdf_file {                                                          
+   dimensions:                                                                     
+           time = 5 ;                                                              
+   variables:                                                                      
+           double corrected_sea_level(time) ;                                      
+                   string corrected_sea_level:name = "corr sea level" ;            
+                   corrected_sea_level:scale_factor = 1. ;                         
+                   corrected_sea_level:_FillValue = -9999 ;                        
+                   string corrected_sea_level:units = "mm" ;                       
+           double sea_level(time) ;                                                
+                   string sea_level:name = "sea level" ;                           
+                   sea_level:scale_factor = 1. ;                                   
+                   sea_level:_FillValue = -9999 ;                                  
+                   string sea_level:units = "mm" ;                                 
+           double time(time) ;                                                     
+                   string time:name = "time" ;                                     
+                   string time:units = "seconds since 19700101T00:00:00" ;         
+
+   // global attributes:                                                           
+                   string :Conventions = "CF-1.0" ;                                
+                   string :history = "the hdf file has been created by EGADS" ;    
+                   string :comments = "no comments on the hdf file" ;              
+                   string :institution = "My institution" ;                        
+   }                                                                               

doc/source/index.rst

@@ -11,6 +11,6 @@ Welcome to EGADS's documentation!
 
    intro
    install
-   tutorial
+   tutorial-global
    alg_development
    egadsapi

doc/source/install.rst

@@ -12,6 +12,7 @@ Use of EGADS requires the following packages:
 * numpy 1.14 or newer. Available at http://numpy.scipy.org/
 * scipy 1.00 or newer. Available at http://www.scipy.org/
 * Python netCDF4 libraries 1.3.0 or newer. Available at https://pypi.python.org/pypi/netCDF4
+* h5py 2.10.0 or newer. Available at https://pypi.org/project/h5py
 * python_dateutil 2.6.1 or newer. Available at https://pypi.python.org/pypi/python-dateutil
 * quantities 0.12.1 or newer. Available at https://pypi.org/project/quantities
 * requests 2.18.4 or newer. Optional, only for update checking. Available at https://pypi.org/project/requests/
@@ -47,6 +48,8 @@ Actually, the number of option is limited and will probably incrase in the futur
 * ``path`` in ``LOG`` section: a string corresponding to an OS path ; it is used to set the directory path where the log file is saved.
 * ``check_update`` in ``OPTIONS`` section: True or False ; it is used to let EGADS check for an update automatically when it is imported.
 
+The file containing all options is now stored in the folder ``.egads_lineage`` in the user $HOME directory.
+
 
 Log
 ***
@@ -85,4 +88,4 @@ If the ``check_update`` option is set on True in the egads.ini file, EGADS will
    >>> egads.set_options(check_update=True)
    >>> exit()
 
-The use of pip is still required to update EGADS package. The module Requests is optional for EGADS but is mandatory to check for an update.
+The module Requests is optional for EGADS but is mandatory to check for an update.

doc/source/tutorial-csv.inc

@@ -0,0 +1,177 @@
+Working with CSV files
+***********************
+
+:class:`egads.input.text_file_io.EgadsCsv` is designed to easily input or output data in CSV format. Data input using :class:`~.EgadsCsv` is separated into a list of arrays, which each column a separate array in the list. 
+
+Opening
+----------
+
+To open a csv file, simply create a :class:`~.EgadsCsv` instance with the parameters *filename*, *perms*, *delimiter* and *quotechar*:
+
+    >>> import egads
+    >>> f = egads.input.EgadsCsv('/pathname/filename.nc', 'r', ',','"')
+
+.. function:: EgadsCsv(filename[, perms='r', delimiter=',', quotechar='"'])
+
+   Open a text file.
+
+   :param filename: path and filename of a text file
+   :type filename: string
+   :param perms: permissions ; optional
+   :type perms: string
+   :param delimiter: a one-character string used to separate fields ; optional
+   :type delimiter: string
+   :param quotechar: a one-character string used to quote fields containing special characters ; optional
+   :type quotechar: string
+   :rtype: csv file
+
+Valid values for permissions are:
+
+* ``r`` -- Read: opens file for reading only. Default value if nothing is provided.
+* ``w`` -- Write: opens file for writing, and overwrites data in file.
+* ``a`` -- Append: opens file for appending data.
+* ``r+`` -- Read and write: opens file for both reading and writing.
+
+
+File Manipulation
+------------------
+
+The following methods are available to control the current position in the file and display more information about the file.
+
+.. function:: f.display_file()
+
+   Prints contents of the file out to a standard output.
+
+.. function:: f.get_position()
+
+   Returns the current position in the file as an integer.
+
+.. function:: f.seek(location[, from_where='b'])
+
+   Seeks to a specified location in the text file.
+
+   :param location: it is an integer specifying how far to seek
+   :type location: int
+   :param from_where: it is an option to specify from where to seek, valid options for *from_where* are ``b`` to seek from beginning of file, ``c`` to seek from current position in file and ``e`` to seek from the end of the file ; optional
+   :type from_where: string
+   :rtype: position in the text file
+
+.. function:: f.reset()
+
+   Resets the position to the beginning of the file.
+
+Reading Data
+------------
+
+Reading data is done using the ``read(lines, format)`` method on a file that has been opened with ``r`` or ``r+`` permissions:
+
+    >>> import egads
+    >>> f = egads.input.EgadsCsv()
+    >>> f.open('mycsvfile.csv','r')
+    >>> single_line_as_list = f.read(1)
+    >>> all_lines_as_list = f.read()
+
+.. function:: f.read([lines=None, format=None])
+
+   Returns a list of items read in from the CSV file.
+
+   :param lines: if it is provided, the function will read in the specified number of lines, otherwise it will read the whole file ; optional
+   :type lines: int
+   :param format: it is an optional list of characters used to decompose the elements read in from the CSV files to their proper types, options are  ; optional
+   :type format: string
+   :rtype: list of items read in from the CSV file
+
+Valid options for *format*:
+
+* ``i`` -- int
+* ``f`` -- float
+* ``l`` -- long
+* ``s`` -- string
+
+Thus to read in the line:
+
+``FGBTM,20050105T143523,1.5,21,25``
+
+the command to input with proper formatting would look like this:
+
+   >>> data = f.read(1, ['s','s','f','f'])
+
+Writing Data
+--------------
+
+To write data to a file, use the ``write(data)`` method on a file that has been opened with ``w``, ``a`` or ``r+`` permissions:
+
+   >>> import egads
+   >>> f = egads.input.EgadsCsv()
+   >>> f.open('mycsvfile.csv','a')
+   >>> titles = ['Aircraft ID','Timestamp','Value1','Value2','Value3']
+   >>> f.write(titles) 
+
+where the ``titles`` parameter is a list of strings. This list will be output to the CSV, with each strings separated by the delimiter specified when the file was opened (default is ``,``).
+
+To write multiple lines out to a file, ``writerows(data)`` is used:
+
+   >>> data = [['FGBTM','20050105T143523',1.5,21,25],['FGBTM','20050105T143524',1.6,20,25.6]]
+   >>> f.writerows(data)
+
+Closing
+---------
+
+To close a file, simply call the ``close()`` method:
+
+   >>> f.close()
+
+Tutorial
+---------
+
+Here is a basic CSV file, created by EGADS:
+
+.. literalinclude:: example_files/main_csv_file.csv
+
+This file has been created with the following commands:
+
+* import EGADS module:
+
+    >>> import egads
+
+* create two main variables, following the official EGADS convention:
+
+    >>> data1 = egads.EgadsData(value=[5.0,2.0,-2.0,0.5,4.0], units='mm', name='sea level', scale_factor=1., add_offset=0., _FillValue=-9999)
+    >>> data2 = egads.EgadsData(value=[1.0,3.0,-1.0,2.5,6.0], units='mm', name='corr sea level', scale_factor=1., add_offset=0., _FillValue=-9999)
+
+* create an independant variable, still by following the official EGADS convention:
+
+    >>> time = egads.EgadsData(value=[1.0,2.0,3.0,4.0,5.0], units='seconds since 19700101T00:00:00', name='time')
+
+* create a new EgadsFile instance:
+
+    >>> f = egads.input.EgadsCsv()
+
+* use the following function to open a new file:
+
+    >>> f.open('main_csv_file.csv','w',',','"')
+
+* prepare the headers if necessary:
+
+    >>> headers = ['time', 'sea level', 'corrected sea level']
+
+* prepare an object to receive all data:
+
+    >>> data = [time.value, data1.value, data2.value]
+
+* write the headers and data into the file
+
+    >>> f.write(headers)
+    >>> f.write(data)
+
+* and do not forget to close the file:
+
+    >>> f.close()
+
+.. raw:: latex
+
+    \newpage
+
+.. raw:: latex
+
+    \newpage