Absorbance_information_and_methods.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Author: [Greg Nordin](http://ece.byu.edu/faculty/nordin), [github](https://github.com/gregnordin)<br>\n",
    "Nov. 12, 2016\n",
    "\n",
    "# Spectrometer Measurement\n",
    "\n",
    "## Purpose\n",
    "\n",
    "Measure absorption spectrum for 3D printer resins.\n",
    "\n",
    "\n",
    "## Experimental Setup\n",
    "\n",
    "We use an Ocean Optics QE65-ABS scientific-grade spectrometer ([download pdf datasheet](http://oceanoptics.com/wp-content/uploads/OEM-Data-Sheet-QE65Pro.pdf)).\n",
    "\n",
    "Need figure with:\n",
    "\n",
    "- Spectrometer\n",
    "- Light source\n",
    "- Sample holder\n",
    "- Set up with sample and no-resin\n",
    "- Set up with sample and resin\n",
    "\n",
    "\n",
    "## Measurement Method\n",
    "\n",
    "1. Determine a spectrometer integration time, $\\tau_{int}$, that uses most of the instrument's dynamic range for a given resin or set of resins. Use this integration time for measurements.\n",
    "1. Measure dark spectrum $N$ times: $d^i$ with $i \\in [1,N]$. This results in $N$ data files.\n",
    "1. Measure transmission spectrum of sample with no resin $N$ times: $u^i$ with $i \\in [1,N]$. This results in $N$ data files.\n",
    "1. Measure transmission spectrum of sample with resin $N$ times: $m^i$ with $i \\in [1,N]$. This results in $N$ data files.\n",
    "\n",
    "Note that $N$ does not have to be the same for the dark, no resin, and resin measurements. However, for convenience we typically choose the same $N$ for each case.\n",
    "\n",
    "\n",
    "## Transmittance and Absorbance\n",
    "\n",
    "The transmittance of a sample, $T$, is the fraction of incident light that is transmitted through a sample, which is a function of wavelength,\n",
    "\n",
    "$$ T(\\lambda) = I_t(\\lambda)/I_i(\\lambda), $$\n",
    "\n",
    "where $I_i$ is the incident irradiance (power per unit area, W/cm$^2$) and $I_t$ is the transmitted irradiance. The transmittance can be written in terms of the absorbance, $A$, as\n",
    "\n",
    "$$ T(\\lambda) = 10^{-A(\\lambda)}. $$\n",
    "\n",
    "We can solve for the absorbance as \n",
    "\n",
    "\\begin{align}\n",
    "A(\\lambda) &= -log_{10}(T(\\lambda) \\\\\n",
    "&= -log_{10} \\left( \\frac{I_t(\\lambda)}{I_i(\\lambda)} \\right)  \\\\\n",
    "\\end{align}\n",
    "\n",
    "\n",
    "## Absorbance Calculation\n",
    "\n",
    "The absorbance $A_j$ at a given wavelength, $\\lambda_j$, is calculated according to\n",
    "\n",
    "\\begin{align}\n",
    "b_j &= \\frac{1}{N}\\sum_{i=1}^N{m^i_{j}} - d_j \\\\\n",
    "T_j &= \\frac{b_j}{u_j} \\\\\n",
    "A_j &= -\\log_{10}(T_j)\n",
    "\\end{align}\n",
    "\n",
    "where \n",
    "\n",
    "\\begin{align}\n",
    "b_j &= \\text{dark-corrected average spectrum measurement at } j^{th} \\text{wavelength} \\\\\n",
    "T_j &= \\text{Transmittance at } j^{th} \\text{wavelength} \\\\\n",
    "A_j &= \\text{Absorbance at } j^{th} \\text{wavelength} \\\\\n",
    "m^i_j &= \\text{measurement at } j^{th} \\text{wavelength in } i^{th} \\text{file} \\\\\n",
    "d_{j} &= \\frac{1}{N}\\sum_{i=1}^{N}{d^i_{j}} = \\text{average dark measurement at } j^{th} \\text{wavelength} \\\\\n",
    "u_{j} &= \\frac{1}{N}\\sum_{i=1}^{N}{u^i_{j}} - d_j = \\text{average dark-corrected measurement at } j^{th} \\text{wavelength prior to filling sample with resin} \\\\\n",
    "\\end{align}\n",
    "\n",
    "Note that it is important to subtract the measured dark signal at each wavelength from the corresponding resin and no-resin measurements. The dark signal originates from an accumulation of electrons in each spectrometer detector pixel during the integration time even when no light is incident on the pixel. As such it represents a spurious signal in the spectrum measurement (in the form of a constant offset for a given integration time), which is easily corrected by simply subtracting the average dark signal from each measurement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Implementation\n",
    "\n",
    "## Directory Structure\n",
    "\n",
    "Here is an example directory structure:\n",
    "\n",
    "    ├── 2016-06_resin_paper\n",
    "    │   ├── 161109_process_data.ipynb\n",
    "    │   ├── absorbance.py\n",
    "    │   ├── processed_data\n",
    "    │   │   ├── dark_2015-06-27.csv\n",
    "    │   │   ├── fslclear_2015-07-30.csv\n",
    "    │   │   ├── irgacure819_2015-06-30.csv\n",
    "    │   │   ├── martius_05_2015-06-30.csv\n",
    "    │   │   ├── plasclear_2015-06-27.csv\n",
    "    │   │   ├── pr48_2015-07-27.csv\n",
    "    │   │   ├── pr48noA_2015-07-30.csv\n",
    "    │   │   ├── pr48noPA_2015-07-30.csv\n",
    "    │   │   ├── sudanI_015_2015-06-27.csv\n",
    "    │   │   ├── sudanI_01_2015-06-27.csv\n",
    "    │   │   ├── sudanI_02_2015-06-27.csv\n",
    "    │   │   └── sudanI_04_2015-06-27.csv\n",
    "    │   └── raw_data\n",
    "    │       ├── dark_2015-06-27\n",
    "    │       ├── fslclear_2015-07-30\n",
    "    │       ├── irgacure819_2015-06-30\n",
    "    │       ├── martius_05_2015-06-30\n",
    "    │       ├── plasclear_2015-06-27\n",
    "    │       ├── pr48_2015-07-27\n",
    "    │       ├── pr48noA_2015-07-30\n",
    "    │       ├── pr48noPA_2015-07-30\n",
    "    │       ├── sudanI_015_2015-06-27\n",
    "    │       ├── sudanI_01_2015-06-27\n",
    "    │       ├── sudanI_02_2015-06-27\n",
    "    │       └── sudanI_04_2015-06-27\n",
    "    ├── Absorbance_information_and_methods.ipynb\n",
    "    └── source_spectra\n",
    "        ├── 161112_compare_asiga_visitech.ipynb\n",
    "        ├── asiga_pico_plus_27\n",
    "        │   ├── 161112_asiga_spectrum.ipynb\n",
    "        │   ├── asiga_spectrum.png\n",
    "        │   ├── ave_raw_data\n",
    "        │   └── processed_data\n",
    "        ├── compare_visitech_asiga_spectra.png\n",
    "        ├── spectrum_helper_calcs.py\n",
    "        └── visitech_luxbeam_LRS_WQ\n",
    "            ├── 161112_visitech_source_spectrum.ipynb\n",
    "            ├── Collect_Spectra.ipynb\n",
    "            ├── processed_data\n",
    "            ├── raw_data\n",
    "            └── visitech_spectrum.png\n",
    "\n",
    "The main directory contains a directory, `source_spectra`, dedicated to emission spectrum measurements for various 3D printers. It also contains a directory for various resin spectrum data sets (`2016-06_resin_paper`, which corresponds to our paper [\"Optical Approach to Resin Formulation for 3D Printed Microfluidics\"](https://www.ncbi.nlm.nih.gov/pubmed/26744624)). New resin spectrum data sets are anticipated to be included in the future.\n",
    "\n",
    "Each data set is organized by date and purpose in its own directory named according to YYYY-MM_NAME. Inside the directory are folders for raw data and processed data calculated from the raw data. Any code and/or Jupyter notebooks used in processing the data are included inside the data set directory. The intention is that processed data can always be recreated from the raw data without worrying about conflicting source code versions (for example, `absorbance.py` in `2016-06_resin_paper`).\n",
    "\n",
    "\n",
    "## Raw Data Files\n",
    "\n",
    "\n",
    "In the `raw_data` directory of `2016-06_resin_paper` spectrometer measurements for individual resins are organized as follows:\n",
    "\n",
    "    └── sudanI_04_2015-06-27\n",
    "        ├── meas_04_1.txt\n",
    "        ├── meas_04_10.txt\n",
    "        ├── meas_04_100.txt\n",
    "        ├── meas_04_11.txt\n",
    "        ├── meas_04_12.txt\n",
    "        ├── <etc.>\n",
    "        ├── std_04_1.txt\n",
    "        ├── std_04_10.txt\n",
    "        ├── std_04_100.txt\n",
    "        ├── std_04_11.txt\n",
    "        ├── std_04_12.txt\n",
    "        ├── <etc.>\n",
    "\n",
    "Data file directory names in `raw_data` have the form, `resin_YYYY-MM-DD`, where `YYYY-MM-DD` is the date the data was taken. In each resin directory, data files end in `.txt`. Files that start with `meas_` are spectrum measurements of a sample with resin while files that start with a different name are spectrum measurements of the sample with no resin.\n",
    "\n",
    "There are generally `N` files that start with `meas_` for a given material, and the same number that do not start with `meas_` for samples prior to filling with resin.  \n",
    "\n",
    "Each data file is generated by a single spectrometer measurement. Files are in the following space-separated-value (SSV) format:\n",
    "\n",
    "    wavelength1 wavelength2 wavelength3 wavelength4 ...\n",
    "    measuremen1 measuremen2 measuremen3 measuremen4 ...\n",
    "\n",
    "Here is an example:\n",
    "\n",
    "    1.907908162221780231e+02 1.915984207987377772e+02 1.924059541587151330e+02 ...\n",
    "    2.400000000000000000e+03 2.399000000000000000e+03 2.395000000000000000e+03 ...\n",
    "\n",
    "Wavelength is in nanometers, and measurements are counts from the spectrometer's 16-bit analog-to-digital converter (range of values: [0, 65535]).\n",
    "\n",
    "\n",
    "## Create CSV files with processed absorbance spectra\n",
    "\n",
    "### Integration Time\n",
    "\n",
    "The integration time for the spectrometer measurements in `2016-06_resin_paper` is 10,000 $\\mu$s.\n",
    "\n",
    "### Dark-Correct Raw Data\n",
    "\n",
    "Each average dark-corrected value, $b_j$, for a material depends on using the correct dark value, $d_j$. All of the material measurements shown in the `2016-06_resin_paper/raw_data` directory above use the same dark data, which is in the processed data file `processed_data/dark_2015-06-27.csv`.\n",
    "\n",
    "### File format\n",
    "\n",
    "For each material, we want to create a new CSV data file that has the following format:\n",
    "\n",
    "    # {date_measured : 2015-06-01, thickness_um : XX, description : \"\", concentration_percent : YY, ... }\n",
    "    wavelength1,value1,stdev1\n",
    "    wavelength2,value2,stdev2\n",
    "    wavelength3,value3,stdev3\n",
    "\n",
    "where the first line is a python dict containing relevant information about the measurement, and each following line contains comma-separated wavelength and absorbance (dark- and blank-corrected) values, and possibly the corresponding standard deviation. \n",
    "\n",
    "### Wavelength range\n",
    "\n",
    "In each spectrum measurement file in `2016-06_resin_paper/raw_data` there are 1044 measurements, each corresponding to a different wavelength in the range [190.7908, 993.5721] nm. The measurements near the shortest and longest wavelengths tend to be the most noisy, so we will throw them away and restrict the range to [231.0837 960.6043] nm, which corresponds to wavelengths[50:998] in the original data files, i.e., 948 data points.\n",
    "\n",
    "### Processing steps\n",
    "\n",
    "1. Create average dark data, $d_j$\n",
    "1. Get names of data sets (i.e., names of folders in `raw_data`)\n",
    "2. Choose data sets for which to create processed absorbance files\n",
    "3. Loop over these data sets. For each:\n",
    "    1. Open & read resin and no resin files\n",
    "    1. Calculate absorbance using the method above\n",
    "    1. Save data as csv file (with sample resin thickness, integration time, and dark data file name)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Acknowledgments\n",
    "\n",
    "Spectrum measurements and resin & sample preparation by Hua Gong, Steven Perry, and Mike Beauchamp."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [Root]",
   "language": "python",
   "name": "Python [Root]"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}