4_mdx2_integration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "09e8bfd5-a08a-4b6a-9480-3707502a902c",
   "metadata": {},
   "source": [
    "# Diffuse scattering integration with *mdx2*\n",
    "\n",
    "In this tutorial, we will:\n",
    "\n",
    "- Integrate diffuse scattering on a fractional Miller index grid\n",
    "- Correct for geometry and background scattering\n",
    "\n",
    "Requirements:\n",
    "- `geometry.nxs` from the *mdx2* geometry tutorial\n",
    "- `data.nxs`, `bkg_data_binned.nxs`, `mask.nxs` from the *mdx2* data tutorial\n",
    "\n",
    "Imports (run this before you do anything else)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bd597362-9451-436e-a268-ac7421674cb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# for loading mdx2 objects in python\n",
    "from mdx2.utils import loadobj\n",
    "\n",
    "# for calculating statistics\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "414edb99-bdd5-4ddd-8bd0-d5ccc510f39d",
   "metadata": {},
   "source": [
    "## 1) Integration\n",
    "\n",
    "### *mdx2.integrate*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "cb464351-e1c4-4694-a3e0-feaf11c6e2ad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: mdx2.integrate [-h] [--mask MASK] [--subdivide N N N]\n",
      "                      [--max_spread DEGREES] [--outfile OUTFILE]\n",
      "                      geom data\n",
      "\n",
      "Integrate counts in an image stack on a Miller index grid\n",
      "\n",
      "positional arguments:\n",
      "  geom                  NeXus file with miller_index\n",
      "  data                  NeXus file with image_series\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  --mask MASK           NeXus file with mask (default: None)\n",
      "  --subdivide N N N     subdivisions of the Miller index grid (default: [1, 1,\n",
      "                        1])\n",
      "  --max_spread DEGREES  maximum angular spread for binning partial\n",
      "                        observations (default: 1.0)\n",
      "  --outfile OUTFILE     name of the output NeXus file (default:\n",
      "                        integrated.nxs)\n"
     ]
    }
   ],
   "source": [
    "!mdx2.integrate --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc0e341c-d596-404e-b01e-1aa009dca083",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "\n",
    "The main thing we need to decide is how to divide up reciprocal space into voxels. For the tutorial, I've chosen to subdivide by 4 in each direction, meaning voxels will be centered at h = 1, 1.25, 1.5, 1.75, etc. \n",
    "\n",
    "This is by no means the optimal choice: the limit depends on the mosaicity, experimental geometry, and most importantly what you hope to learn from the diffuse scattering map. You may want to experiment with other subdivisions. But beware that finer maps require a lot more memory and disk space.\n",
    "\n",
    "Integration is the slowest step in data reduction. Be patient."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fca77f9d-f805-4e47-af29-128764f092a1",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading miller_index from geometry.nxs\n",
      "  importing as MillerIndex from mdx2.geometry\n",
      "Reading image_series from data.nxs\n",
      "  importing as ImageSeries from mdx2.data\n",
      "Looping through chunks\n",
      "  binned chunk 0 from 1891042 to 11799 voxels\n",
      "  binned chunk 1 from 1891704 to 16089 voxels\n",
      "  binned chunk 2 from 1891701 to 18091 voxels\n",
      "  binned chunk 3 from 1891522 to 16321 voxels\n",
      "  binned chunk 4 from 1891069 to 12286 voxels\n",
      "  binned chunk 5 from 1890751 to 12084 voxels\n",
      "  binned chunk 6 from 1891722 to 16991 voxels\n",
      "  binned chunk 7 from 1891538 to 19162 voxels\n",
      "  binned chunk 8 from 1891323 to 17110 voxels\n",
      "  binned chunk 9 from 1891159 to 12590 voxels\n",
      "  binned chunk 10 from 1891123 to 11731 voxels\n",
      "  binned chunk 11 from 1891316 to 16922 voxels\n",
      "  binned chunk 12 from 1891533 to 19300 voxels\n",
      "  binned chunk 13 from 1891388 to 17004 voxels\n",
      "  binned chunk 14 from 1891623 to 12171 voxels\n",
      "  binned chunk 15 from 1891288 to 10482 voxels\n",
      "  binned chunk 16 from 1891766 to 15408 voxels\n",
      "  binned chunk 17 from 1891664 to 17729 voxels\n",
      "  binned chunk 18 from 1891496 to 15438 voxels\n",
      "  binned chunk 19 from 1891546 to 10828 voxels\n",
      "  binned chunk 20 from 1891670 to 8306 voxels\n",
      "  binned chunk 21 from 1891818 to 12166 voxels\n",
      "  binned chunk 22 from 1891454 to 14069 voxels\n",
      "  binned chunk 23 from 1891667 to 12286 voxels\n",
      "  binned chunk 24 from 1891782 to 8499 voxels\n",
      "  binned chunk 25 from 1891289 to 5480 voxels\n",
      "  binned chunk 26 from 1892435 to 7545 voxels\n",
      "  binned chunk 27 from 1788977 to 8316 voxels\n",
      "  binned chunk 28 from 1890515 to 7804 voxels\n",
      "  binned chunk 29 from 1891904 to 5445 voxels\n",
      "  binned chunk 30 from 1891975 to 5975 voxels\n",
      "  binned chunk 31 from 1891729 to 8464 voxels\n",
      "  binned chunk 32 from 1869527 to 10014 voxels\n",
      "  binned chunk 33 from 1891118 to 8926 voxels\n",
      "  binned chunk 34 from 1891948 to 6073 voxels\n",
      "  binned chunk 35 from 1891711 to 8609 voxels\n",
      "  binned chunk 36 from 1891318 to 12960 voxels\n",
      "  binned chunk 37 from 1891680 to 15720 voxels\n",
      "  binned chunk 38 from 1891440 to 13717 voxels\n",
      "  binned chunk 39 from 1891692 to 9134 voxels\n",
      "  binned chunk 40 from 1891726 to 10461 voxels\n",
      "  binned chunk 41 from 1891775 to 15818 voxels\n",
      "  binned chunk 42 from 1891925 to 19156 voxels\n",
      "  binned chunk 43 from 1891834 to 16818 voxels\n",
      "  binned chunk 44 from 1891748 to 11332 voxels\n",
      "  binned chunk 45 from 1892093 to 11480 voxels\n",
      "  binned chunk 46 from 1891753 to 17071 voxels\n",
      "  binned chunk 47 from 1891674 to 20314 voxels\n",
      "  binned chunk 48 from 1891936 to 18005 voxels\n",
      "  binned chunk 49 from 1891672 to 12504 voxels\n",
      "  binned chunk 50 from 1891604 to 11686 voxels\n",
      "  binned chunk 51 from 1891793 to 16901 voxels\n",
      "  binned chunk 52 from 1891718 to 19812 voxels\n",
      "  binned chunk 53 from 1891518 to 17811 voxels\n",
      "  binned chunk 54 from 1891529 to 12723 voxels\n",
      "  binned chunk 55 from 1891425 to 11361 voxels\n",
      "  binned chunk 56 from 1891332 to 15865 voxels\n",
      "  binned chunk 57 from 1891850 to 18219 voxels\n",
      "  binned chunk 58 from 1890958 to 16541 voxels\n",
      "  binned chunk 59 from 1890713 to 12323 voxels\n",
      "  binned chunk 60 from 1891137 to 11795 voxels\n",
      "  binned chunk 61 from 1891687 to 16058 voxels\n",
      "  binned chunk 62 from 1891701 to 18145 voxels\n",
      "  binned chunk 63 from 1891507 to 16361 voxels\n",
      "  binned chunk 64 from 1890891 to 12315 voxels\n",
      "  binned chunk 65 from 1890899 to 12080 voxels\n",
      "  binned chunk 66 from 1891657 to 16966 voxels\n",
      "  binned chunk 67 from 1892043 to 19254 voxels\n",
      "  binned chunk 68 from 1891606 to 17175 voxels\n",
      "  binned chunk 69 from 1891639 to 12629 voxels\n",
      "  binned chunk 70 from 1891139 to 11710 voxels\n",
      "  binned chunk 71 from 1891403 to 16905 voxels\n",
      "  binned chunk 72 from 1891447 to 19239 voxels\n",
      "  binned chunk 73 from 1891629 to 17007 voxels\n",
      "  binned chunk 74 from 1891599 to 12209 voxels\n",
      "  binned chunk 75 from 1891416 to 10493 voxels\n",
      "  binned chunk 76 from 1891664 to 15397 voxels\n",
      "  binned chunk 77 from 1891821 to 17616 voxels\n",
      "  binned chunk 78 from 1891871 to 15400 voxels\n",
      "  binned chunk 79 from 1891789 to 10819 voxels\n",
      "  binned chunk 80 from 1892166 to 8319 voxels\n",
      "  binned chunk 81 from 1891573 to 12150 voxels\n",
      "  binned chunk 82 from 1892135 to 14008 voxels\n",
      "  binned chunk 83 from 1891718 to 12210 voxels\n",
      "  binned chunk 84 from 1891917 to 8469 voxels\n",
      "  binned chunk 85 from 1890457 to 5483 voxels\n",
      "  binned chunk 86 from 1893194 to 7518 voxels\n",
      "  binned chunk 87 from 1790752 to 8265 voxels\n",
      "  binned chunk 88 from 1891259 to 7689 voxels\n",
      "  binned chunk 89 from 1889884 to 5397 voxels\n",
      "  binned chunk 90 from 1891076 to 6001 voxels\n",
      "  binned chunk 91 from 1891838 to 8453 voxels\n",
      "  binned chunk 92 from 1869440 to 9940 voxels\n",
      "  binned chunk 93 from 1891615 to 8871 voxels\n",
      "  binned chunk 94 from 1890986 to 6028 voxels\n",
      "  binned chunk 95 from 1891826 to 8624 voxels\n",
      "  binned chunk 96 from 1892483 to 12884 voxels\n",
      "  binned chunk 97 from 1891690 to 15652 voxels\n",
      "  binned chunk 98 from 1891652 to 13695 voxels\n",
      "  binned chunk 99 from 1891382 to 9121 voxels\n",
      "  binned chunk 100 from 1891725 to 10505 voxels\n",
      "  binned chunk 101 from 1891923 to 15803 voxels\n",
      "  binned chunk 102 from 1891857 to 19110 voxels\n",
      "  binned chunk 103 from 1891629 to 16793 voxels\n",
      "  binned chunk 104 from 1891579 to 11325 voxels\n",
      "  binned chunk 105 from 1891652 to 11511 voxels\n",
      "  binned chunk 106 from 1891920 to 17057 voxels\n",
      "  binned chunk 107 from 1891843 to 20324 voxels\n",
      "  binned chunk 108 from 1891789 to 18034 voxels\n",
      "  binned chunk 109 from 1891772 to 12521 voxels\n",
      "  binned chunk 110 from 1891667 to 11741 voxels\n",
      "  binned chunk 111 from 1891688 to 16937 voxels\n",
      "  binned chunk 112 from 1891725 to 19824 voxels\n",
      "  binned chunk 113 from 1892024 to 17752 voxels\n",
      "  binned chunk 114 from 1891325 to 12767 voxels\n",
      "  binned chunk 115 from 1891242 to 11380 voxels\n",
      "  binned chunk 116 from 1891380 to 15866 voxels\n",
      "  binned chunk 117 from 1891653 to 18283 voxels\n",
      "  binned chunk 118 from 1891143 to 16584 voxels\n",
      "  binned chunk 119 from 1891086 to 12267 voxels\n",
      "  binned chunk 120 from 1890691 to 11772 voxels\n",
      "  binned chunk 121 from 1891618 to 16044 voxels\n",
      "  binned chunk 122 from 1891662 to 18197 voxels\n",
      "  binned chunk 123 from 1891503 to 16400 voxels\n",
      "  binned chunk 124 from 1891093 to 12314 voxels\n",
      "  binned chunk 125 from 1890619 to 12079 voxels\n",
      "  binned chunk 126 from 1891819 to 16954 voxels\n",
      "  binned chunk 127 from 1891801 to 19265 voxels\n",
      "  binned chunk 128 from 1891493 to 17191 voxels\n",
      "  binned chunk 129 from 1891327 to 12633 voxels\n",
      "  binned chunk 130 from 1891049 to 11684 voxels\n",
      "  binned chunk 131 from 1891375 to 16854 voxels\n",
      "  binned chunk 132 from 1892224 to 19126 voxels\n",
      "  binned chunk 133 from 1892125 to 16929 voxels\n",
      "  binned chunk 134 from 1891504 to 12202 voxels\n",
      "  binned chunk 135 from 1891705 to 10451 voxels\n",
      "  binned chunk 136 from 1891731 to 15370 voxels\n",
      "  binned chunk 137 from 1891859 to 17574 voxels\n",
      "  binned chunk 138 from 1891799 to 15344 voxels\n",
      "  binned chunk 139 from 1891446 to 10854 voxels\n",
      "  binned chunk 140 from 1891654 to 8329 voxels\n",
      "  binned chunk 141 from 1891572 to 12110 voxels\n",
      "  binned chunk 142 from 1891721 to 13944 voxels\n",
      "  binned chunk 143 from 1891683 to 12099 voxels\n",
      "  binned chunk 144 from 1891896 to 8476 voxels\n",
      "  binned chunk 145 from 1891448 to 5491 voxels\n",
      "  binned chunk 146 from 1892111 to 7541 voxels\n",
      "  binned chunk 147 from 1789644 to 8195 voxels\n",
      "  binned chunk 148 from 1892299 to 7601 voxels\n",
      "  binned chunk 149 from 1893186 to 5403 voxels\n",
      "  binned chunk 150 from 1891345 to 6040 voxels\n",
      "  binned chunk 151 from 1891551 to 8456 voxels\n",
      "  binned chunk 152 from 1868886 to 9858 voxels\n",
      "  binned chunk 153 from 1891007 to 8792 voxels\n",
      "  binned chunk 154 from 1891810 to 5992 voxels\n",
      "  binned chunk 155 from 1891841 to 8662 voxels\n",
      "  binned chunk 156 from 1892356 to 12856 voxels\n",
      "  binned chunk 157 from 1891749 to 15604 voxels\n",
      "  binned chunk 158 from 1890761 to 13677 voxels\n",
      "  binned chunk 159 from 1892015 to 9104 voxels\n",
      "  binned chunk 160 from 1891609 to 10546 voxels\n",
      "  binned chunk 161 from 1891816 to 15791 voxels\n",
      "  binned chunk 162 from 1891753 to 19084 voxels\n",
      "  binned chunk 163 from 1891845 to 16778 voxels\n",
      "  binned chunk 164 from 1891465 to 11335 voxels\n",
      "  binned chunk 165 from 1891951 to 11546 voxels\n",
      "  binned chunk 166 from 1891846 to 17056 voxels\n",
      "  binned chunk 167 from 1891851 to 20338 voxels\n",
      "  binned chunk 168 from 1891456 to 18004 voxels\n",
      "  binned chunk 169 from 1891695 to 12500 voxels\n",
      "  binned chunk 170 from 1891795 to 11747 voxels\n",
      "  binned chunk 171 from 1891714 to 16923 voxels\n",
      "  binned chunk 172 from 1891798 to 19817 voxels\n",
      "  binned chunk 173 from 1891300 to 17873 voxels\n",
      "  binned chunk 174 from 1891496 to 12743 voxels\n",
      "  binned chunk 175 from 1891234 to 11396 voxels\n",
      "  binned chunk 176 from 1891254 to 15869 voxels\n",
      "  binned chunk 177 from 1891498 to 18234 voxels\n",
      "  binned chunk 178 from 1891276 to 16632 voxels\n",
      "  binned chunk 179 from 1890859 to 12307 voxels\n",
      "Summing partial observations over 180 chunks\n",
      "  binned from 2383965 to 2047443 voxels\n",
      "Saving table of integrated data to integrated.nxs\n",
      "Exporting <class 'mdx2.data.HKLTable'> to nexus object\n",
      "  writing hkl_table to integrated.nxs\n",
      "done!\n"
     ]
    }
   ],
   "source": [
    "!mdx2.integrate geometry.nxs data.nxs --mask mask.nxs --subdivide 4 4 4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5efcb3bd-dd0d-4a95-bc62-8f8ffa123c58",
   "metadata": {},
   "source": [
    "### Integration statistics\n",
    "\n",
    "Integration produces a giant table indexed by fractional h,k,l values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "730228a6-78f0-4bdf-95b1-5db7e182e711",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading hkl_table from integrated.nxs\n",
      "  importing as HKLTable from mdx2.data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "      <th>ix</th>\n",
       "      <th>iy</th>\n",
       "      <th>n</th>\n",
       "      <th>phi</th>\n",
       "      <th>pixels</th>\n",
       "      <th>seconds</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>h</th>\n",
       "      <th>k</th>\n",
       "      <th>l</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">-65.5</th>\n",
       "      <th rowspan=\"3\" valign=\"top\">-16.75</th>\n",
       "      <th>-9.75</th>\n",
       "      <td>2</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160.449997</td>\n",
       "      <td>3</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9.50</th>\n",
       "      <td>12</td>\n",
       "      <td>0.869565</td>\n",
       "      <td>2.304348</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160.250000</td>\n",
       "      <td>23</td>\n",
       "      <td>2.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9.25</th>\n",
       "      <td>15</td>\n",
       "      <td>1.892857</td>\n",
       "      <td>1.928571</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160.078568</td>\n",
       "      <td>28</td>\n",
       "      <td>2.8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">-16.50</th>\n",
       "      <th>-9.50</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>6.166667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160.183334</td>\n",
       "      <td>6</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>-9.25</th>\n",
       "      <td>3</td>\n",
       "      <td>0.285714</td>\n",
       "      <td>6.857143</td>\n",
       "      <td>0.0</td>\n",
       "      <td>160.050003</td>\n",
       "      <td>7</td>\n",
       "      <td>0.7</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    counts        ix        iy    n         phi  pixels  \\\n",
       "h     k      l                                                            \n",
       "-65.5 -16.75 -9.75       2  0.333333  1.666667  0.0  160.449997       3   \n",
       "             -9.50      12  0.869565  2.304348  0.0  160.250000      23   \n",
       "             -9.25      15  1.892857  1.928571  0.0  160.078568      28   \n",
       "      -16.50 -9.50       5  0.000000  6.166667  0.0  160.183334       6   \n",
       "             -9.25       3  0.285714  6.857143  0.0  160.050003       7   \n",
       "\n",
       "                    seconds  \n",
       "h     k      l               \n",
       "-65.5 -16.75 -9.75      0.3  \n",
       "             -9.50      2.3  \n",
       "             -9.25      2.8  \n",
       "      -16.50 -9.50      0.6  \n",
       "             -9.25      0.7  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load the table\n",
    "tab = loadobj('integrated.nxs','hkl_table')\n",
    "\n",
    "# convert to pandas dataframe\n",
    "df = tab.to_frame().set_index(['h','k','l'])\n",
    "\n",
    "# show the first several entries\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "476a9bd5-bb2f-4220-9af0-ef9ca8d21749",
   "metadata": {},
   "source": [
    "It is interesting to plot the number of observations as a function of spindle angle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b02150b7-fbdb-4871-94a4-2a8e7248438a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAD4CAYAAADGmmByAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAaEUlEQVR4nO3df5Bd5X3f8ffHki1+1PwWRJEgK4rAEYz5oQXTcZzUqA5yHFu4gXg9TVAzapRQmsbTZmLJ9TT2H+qAZ2ps6kIiBweBYwtBglBjKy4WtT3pEMnLTyFAw9rIsBFFMhAQxggLf/rHfda5u7q7uldnz9690uc1c+ee+73nOfd5dHf3o+ecc8+VbSIiIg7VW7rdgYiI6G0JkoiIqCRBEhERlSRIIiKikgRJRERUMrPbHZhqp5xyivv6+rrdjYiInvLAAw/80PbsVs8dcUHS19fH4OBgt7sREdFTJP1gvOeyaysiIipJkERERCUJkoiIqCRBEhERlSRIIiKikgRJRERUkiCJiIhKEiQREVFJgiQiIio54j7ZXkXfyq/9bHnndR/oYk8iItozFX+3MiOJiIhKEiQREVFJgiQiIipJkERERCUJkoiIqCRBEhERlSRIIiKikgRJRERUkiCJiIhKagsSSedIerjp9oqkj0k6SdK9kp4q9yc2tVklaUjSDkmXN9UXSdpWnrtRkkp9lqQ7Sn2LpL66xhMREa3VFiS2d9i+wPYFwCLgNeBuYCWw2fYCYHN5jKSFwABwLrAEuEnSjLK5m4EVwIJyW1Lqy4GXbJ8F3ABcX9d4IiKitanatbUY+J7tHwBLgbWlvha4oiwvBdbZ3mf7aWAIuETSHOA42/fbNnDbmDYj27oLWDwyW4mIiKkxVUEyAHy1LJ9m+zmAcn9qqc8Fnm1qM1xqc8vy2PqoNrb3Ay8DJ499cUkrJA1KGtyzZ8+kDCgiIhpqDxJJbwM+BNx5sFVb1DxBfaI2owv2Gtv9tvtnz559kG5EREQnpmJG8n7gQdvPl8fPl91VlPvdpT4MnN7Ubh6wq9TntaiPaiNpJnA88GINY4iIiHFMRZB8lH/arQWwEVhWlpcB9zTVB8qZWPNpHFTfWnZ/7ZV0aTn+cfWYNiPbuhK4rxxHiYiIKVLrF1tJOgZ4H/B7TeXrgPWSlgPPAFcB2N4uaT3wOLAfuNb2m6XNNcCtwNHApnIDuAW4XdIQjZnIQJ3jiYiIA9UaJLZfY8zBb9sv0DiLq9X6q4HVLeqDwHkt6q9TgigiIrojn2yPiIhKEiQREVFJgiQiIipJkERERCUJkoiIqCRBEhERlSRIIiKikgRJRERUkiCJiIhKEiQREVFJgiQiIipJkERERCUJkoiIqCRBEhERlSRIIiKikgRJRERUkiCJiIhKEiQREVFJrUEi6QRJd0l6UtITkv6FpJMk3SvpqXJ/YtP6qyQNSdoh6fKm+iJJ28pzN0pSqc+SdEepb5HUV+d4IiLiQHXPSD4P/K3tdwDnA08AK4HNthcAm8tjJC0EBoBzgSXATZJmlO3cDKwAFpTbklJfDrxk+yzgBuD6mscTERFj1BYkko4Dfhm4BcD2G7b/EVgKrC2rrQWuKMtLgXW299l+GhgCLpE0BzjO9v22Ddw2ps3Itu4CFo/MViIiYmrUOSM5E9gD/IWkhyT9uaRjgdNsPwdQ7k8t688Fnm1qP1xqc8vy2PqoNrb3Ay8DJ4/tiKQVkgYlDe7Zs2eyxhcREdQbJDOBi4CbbV8I/IiyG2scrWYSnqA+UZvRBXuN7X7b/bNnz5641xER0ZE6g2QYGLa9pTy+i0awPF92V1Hudzetf3pT+3nArlKf16I+qo2kmcDxwIuTPpKIiBhXbUFi+/8Bz0o6p5QWA48DG4FlpbYMuKcsbwQGyplY82kcVN9adn/tlXRpOf5x9Zg2I9u6ErivHEeJiIgpMrPm7f8B8JeS3gZ8H/gdGuG1XtJy4BngKgDb2yWtpxE2+4Frbb9ZtnMNcCtwNLCp3KBxIP92SUM0ZiIDNY8nIiLGqDVIbD8M9Ld4avE4668GVreoDwLntai/TgmiiIjojnyyPSIiKkmQREREJQmSiIioJEESERGVJEgiIqKSBElERFSSIImIiEoSJBERUUmCJCIiKkmQREREJQmSiIioJEESERGVJEgiIqKSBElERFSSIImIiEoSJBERUUmCJCIiKkmQREREJQmSiIiopNYgkbRT0jZJD0saLLWTJN0r6alyf2LT+qskDUnaIenypvqisp0hSTdKUqnPknRHqW+R1FfneCIi4kBTMSN5r+0LbPeXxyuBzbYXAJvLYyQtBAaAc4ElwE2SZpQ2NwMrgAXltqTUlwMv2T4LuAG4fgrGExERTbqxa2spsLYsrwWuaKqvs73P9tPAEHCJpDnAcbbvt23gtjFtRrZ1F7B4ZLYSERFTo+4gMfC/JT0gaUWpnWb7OYByf2qpzwWebWo7XGpzy/LY+qg2tvcDLwMnj+2EpBWSBiUN7tmzZ1IGFhERDTNr3v67be+SdCpwr6QnJ1i31UzCE9QnajO6YK8B1gD09/cf8HxERBy6WmcktneV+93A3cAlwPNldxXlfndZfRg4van5PGBXqc9rUR/VRtJM4HjgxTrGEhERrdUWJJKOlfT2kWXgV4HHgI3AsrLaMuCesrwRGChnYs2ncVB9a9n9tVfSpeX4x9Vj2oxs60rgvnIcJSIipkidu7ZOA+4ux75nAl+x/beSvgusl7QceAa4CsD2dknrgceB/cC1tt8s27oGuBU4GthUbgC3ALdLGqIxExmocTwREdFCbUFi+/vA+S3qLwCLx2mzGljdoj4InNei/joliCIiojvyyfaIiKikrSCRdMBsICIiAtqfkfyppK2S/r2kE+rsUERE9Ja2gsT2LwH/hsaptoOSviLpfbX2LCIiekLbx0hsPwV8Evg48CvAjZKelPSv6+pcRERMf+0eI3mnpBuAJ4DLgA/a/sWyfEON/YuIiGmu3dN/vwB8EfiE7R+PFMvlTz5ZS88iIqIntBskvwb8eOQDgpLeAhxl+zXbt9fWu4iImPbaPUbyTRqfKh9xTKlFRMQRrt0gOcr2qyMPyvIx9XQpIiJ6SbtB8iNJF408kLQI+PEE60dExBGi3WMkHwPulDRy+fY5wEdq6VFERPSUtoLE9nclvQM4h8aXST1p+ye19iwiInpCJ1f/vRjoK20ulITt22rpVURE9Iy2gkTS7cA/Bx4GRr4jxECCJCLiCNfujKQfWJhvH4yIiLHaPWvrMeDn6uxIRET0pnZnJKcAj0vaCuwbKdr+UC29ioiIntFukHyqzk5ERETvavf7SL4N7ATeWpa/CzzYTltJMyQ9JOlvyuOTJN0r6alyf2LTuqskDUnaIenypvoiSdvKczdKUqnPknRHqW+R1NfuwCMiYnK0exn53wXuAv6slOYCG9p8jT+kcfn5ESuBzbYXAJvLYyQtBAaAc4ElwE2SZpQ2NwMrgAXltqTUlwMv2T6LxuXsr2+zTxERMUnaPdh+LfBu4BX42ZdcnXqwRpLmAR8A/rypvBRYW5bXAlc01dfZ3mf7aWAIuETSHOA42/eXs8ZuG9NmZFt3AYtHZisRETE12g2SfbbfGHkgaSaNz5EczOeAPwZ+2lQ7zfZzAOV+JJDmAs82rTdcanPL8tj6qDa29wMvAyeP7YSkFZIGJQ3u2bOnjW5HRES72g2Sb0v6BHB0+a72O4H/NVEDSb8O7Lb9QJuv0Wom4QnqE7UZXbDX2O633T979uw2uxMREe1oN0hWAnuAbcDvAV+n8f3tE3k38CFJO4F1wGWSvgw8X3ZXUe53l/WHgdOb2s8DdpX6vBb1UW3KLOl44MU2xxQREZOg3bO2fmr7i7avsn1lWZ5w15btVbbn2e6jcRD9Ptu/BWwElpXVlgH3lOWNwEA5E2s+jYPqW8vur72SLi3HP64e02ZkW1eW18in7yMiplC719p6mta7jM48hNe8DlgvaTnwDHBV2dZ2SeuBx4H9wLUjX+0LXAPcSuNbGjeVG8AtwO2ShmjMRAYOoT8REVFBJ9faGnEUjT/+J7X7Ira/BXyrLL8ALB5nvdXA6hb1QeC8FvXXS18iIqJL2t219ULT7R9sfw64rN6uRUREL2h319ZFTQ/fQmOG8vZaehQRET2l3V1b/71peT+Ny6X85qT3pof0rfzaz5Z3XveBLvYkIqK72v2q3ffW3ZGIiJgczf/RnQrt7tr6TxM9b/uzk9OdiIjoNZ2ctXUxjc9tAHwQ+A6jL2kSERFHoE6+2Ooi23sBJH0KuNP2v6urYxER0RvavUTKGcAbTY/fAPomvTcREdFz2p2R3A5slXQ3jU+4f5jG5dwjIuII1+5ZW6slbQLeU0q/Y/uh+roVERG9ot1dWwDHAK/Y/jwwXC6sGBERR7h2v2r3T4CPA6tK6a3Al+vqVERE9I52ZyQfBj4E/AjA9i5yiZSIiKD9IHmjfM+HASQdW1+XIiKil7QbJOsl/RlwgqTfBb4JfLG+bkVERK846Flb5VsJ7wDeAbwCnAP8V9v31ty3iIjoAQcNEtuWtMH2IiDhERERo7S7a+vvJV1ca08iIqIntRsk76URJt+T9KikbZIenaiBpKMkbZX0iKTtkj5d6idJulfSU+X+xKY2qyQNSdoh6fKm+qLymkOSbiy725A0S9Idpb5FUl/H/wIREVHJhEEi6Yyy+H7gTBpfr/tB4NfL/UT2AZfZPh+4AFgi6VJgJbDZ9gJgc3mMpIXAAHAusAS4SdKMsq2bgRXAgnJbUurLgZdsnwXcAFx/8CFHRMRkOtiMZAOA7R8An7X9g+bbRA3d8Gp5+NZyM7AUWFvqa4EryvJSYJ3tfbafBoaASyTNAY6zfX85Bfm2MW1GtnUXsHhkthIREVPjYEHS/Ef5zE43LmmGpIeB3cC9trcAp9l+DqDcn1pWn8vo7zcZLrW5ZXlsfVQb2/uBl4GTO+1nREQcuoMFicdZbovtN21fAMyjMbs4b4LVW80kPEF9ojajNyytkDQoaXDPnj0H6XVERHTiYEFyvqRXJO0F3lmWX5G0V9Ir7b6I7X8EvkXj2MbzZXcV5X53WW0YOL2p2TxgV6nPa1Ef1UbSTOB44MUWr7/Gdr/t/tmzZ7fb7YiIaMOEQWJ7hu3jbL/d9syyPPL4uInaSpot6YSyfDTwr4AnaXxd77Ky2jLgnrK8ERgoZ2LNp3FQfWvZ/bVX0qXl+MfVY9qMbOtK4L5yHCUiIqZIu19sdSjmAGvLmVdvAdbb/htJ99O45Mpy4BngKgDb2yWtBx4H9gPX2n6zbOsa4FbgaGBTuQHcAtwuaYjGTGSgxvFEREQLtQWJ7UeBC1vUXwAWj9NmNbC6RX0QOOD4iu3XKUEUERHdUeeMJCIipkjfyq917bUTJJOg+Q3ced0HutiTiIip18lX7UZERBwgQRIREZUkSCIiopIESUREVJIgiYiIShIkERFRSYIkIiIqSZBEREQlCZKIiKgkQRIREZXkEimTLJdLiYgjTYIkIqIHdfMijWNl11ZERFSSIImIiEqyaysiYhw55tmeBMkRJr8YETHZsmsrIiIqqW1GIul04Dbg54CfAmtsf17SScAdQB+wE/hN2y+VNquA5cCbwH+0/Y1SXwTcChwNfB34Q9uWNKu8xiLgBeAjtnfWNaaIyTbRmTeZMcZY0+lMrWZ17traD/xn2w9KejvwgKR7gX8LbLZ9naSVwErg45IWAgPAucDPA9+UdLbtN4GbgRXA39MIkiXAJhqh85LtsyQNANcDH6lxTB3JH4mIqGq6hkez2nZt2X7O9oNleS/wBDAXWAqsLautBa4oy0uBdbb32X4aGAIukTQHOM72/bZNYwbS3GZkW3cBiyWprjFFRMSBpuRgu6Q+4EJgC3Ca7eegETaSTi2rzaUx4xgxXGo/Kctj6yNtni3b2i/pZeBk4IdjXn8FjRkNZ5xxxqSNq1f0wv9oIqa7nKgyvtqDRNI/A/4K+JjtVyaYMLR6whPUJ2ozumCvAdYA9Pf3H/B8N+SHMiIOF7UGiaS30giRv7T916X8vKQ5ZTYyB9hd6sPA6U3N5wG7Sn1ei3pzm2FJM4HjgRdrGUxExBTptb0IdZ61JeAW4Anbn216aiOwDLiu3N/TVP+KpM/SONi+ANhq+01JeyVdSmPX2NXA/xizrfuBK4H7ynGUnlVlptLpD1+3ZkVj+9mtGdl0GX8763Wrf0fibLmd96eOf6NeC49mdc5I3g38NrBN0sOl9gkaAbJe0nLgGeAqANvbJa0HHqdxxte15YwtgGv4p9N/N5UbNILqdklDNGYiAzWOpzad/uA2q+OHuHmbndan4rXHazue6fDHsOofiU7/LeoYc5WfwU7fp6pj6bR9lfenyu/O4aK2ILH9d7Q+hgGweJw2q4HVLeqDwHkt6q9TguhwVCVgetV445mscVb53+Z0+YMxle/5VP4MTvXPch2vV/fP73SVS6TEASZrF9l0+eWp8sewyr9FN03WbG46mMx+9sqYe416/JBCx/r7+z04OHhIbfNDGBG9rMouT0kP2O5v9VyutRUREZUkSCIiopIESUREVJIgiYiIShIkERFRSYIkIiIqSZBEREQlCZKIiKgkQRIREZUkSCIiopIESUREVJIgiYiIShIkERFRSYIkIiIqSZBEREQlCZKIiKgkQRIREZXUFiSSviRpt6THmmonSbpX0lPl/sSm51ZJGpK0Q9LlTfVFkraV526UpFKfJemOUt8iqa+usURExPjqnJHcCiwZU1sJbLa9ANhcHiNpITAAnFva3CRpRmlzM7ACWFBuI9tcDrxk+yzgBuD62kYSERHjqi1IbH8HeHFMeSmwtiyvBa5oqq+zvc/208AQcImkOcBxtu9348vlbxvTZmRbdwGLR2YrERExdab6GMlptp8DKPenlvpc4Nmm9YZLbW5ZHlsf1cb2fuBl4ORWLypphaRBSYN79uyZpKFERARMn4PtrWYSnqA+UZsDi/Ya2/22+2fPnn2IXYyIiFamOkieL7urKPe7S30YOL1pvXnArlKf16I+qo2kmcDxHLgrLSIiajbVQbIRWFaWlwH3NNUHyplY82kcVN9adn/tlXRpOf5x9Zg2I9u6ErivHEeJiIgpNLOuDUv6KvAvgVMkDQN/AlwHrJe0HHgGuArA9nZJ64HHgf3AtbbfLJu6hsYZYEcDm8oN4BbgdklDNGYiA3WNJSIixldbkNj+6DhPLR5n/dXA6hb1QeC8FvXXKUEUERHdM10OtkdERI9KkERERCUJkoiIqCRBEhERlSRIIiKikgRJRERUkiCJiIhKEiQREVFJgiQiIipJkERERCUJkoiIqCRBEhERlSRIIiKikgRJRERUkiCJiIhKEiQREVFJgiQiIipJkERERCUJkoiIqKTng0TSEkk7JA1JWtnt/kREHGl6OkgkzQD+J/B+YCHwUUkLu9uriIgjS08HCXAJMGT7+7bfANYBS7vcp4iII8rMbnegornAs02Ph4F3jV1J0gpgRXn4qqQdHb7OKcAPD6mH08/hNBY4vMZzOI0FMp5pR9f/bPFQxvIL4z3R60GiFjUfULDXAGsO+UWkQdv9h9p+OjmcxgKH13gOp7FAxjOdTfZYen3X1jBwetPjecCuLvUlIuKI1OtB8l1ggaT5kt4GDAAbu9yniIgjSk/v2rK9X9J/AL4BzAC+ZHt7DS91yLvFpqHDaSxweI3ncBoLZDzT2aSORfYBhxQiIiLa1uu7tiIiossSJBERUckRHySSviRpt6THxtT/oFx6ZbukzzTVV5XLseyQdPnU93hinYxH0smS/o+kVyV9oTs9Hl+HY3mfpAckbSv3l3Wn1+PrcDyXSHq43B6R9OHu9Hp8nf7ulOfOKD9vfzS1vZ1Yh+9Nn6QfN70/f9qdXo/vEP6uvVPS/aW+TdJRHb2g7SP6BvwycBHwWFPtvcA3gVnl8anlfiHwCDALmA98D5jR7TFUGM+xwC8Bvw98odt9rziWC4GfL8vnAf/Q7f5XHM8xwMyyPAfYPfJ4utw6GU/T838F3An8Ubf7X+G96WtebzreOhzPTOBR4Pzy+ORO/64d8TMS298BXhxTvga4zva+ss7uUl8KrLO9z/bTwBCNy7RMG52Mx/aPbP8d8PrU9rI9HY7lIdsjnyHaDhwladaUdbYNHY7nNdv7yzpH0eKDtt3W4e8Okq4Avk/j/ZlWOh3LdNfheH4VeNT2I6X+gu03O3m9Iz5IxnE28B5JWyR9W9LFpd7qkixzp7x3nRtvPL2onbH8BvDQyC/MNDfueCS9S9J2YBvw+03BMp21HI+kY4GPA5/uau86M9HP2nxJD5X6e7rVwQ6NN56zAUv6hqQHJf1xpxvu6c+R1GgmcCJwKXAxsF7SmbR5SZZpqOV4XOaxPWbCsUg6F7iexv+yesG447G9BThX0i8CayVtsj0tZ49Nxvvd+TRwg+1XpVa/RtPSeGN5DjjD9guSFgEbJJ1r+5Uu9rUd441nJo1d3BcDrwGbJT1ge3O7G86MpLVh4K/LL/NW4Kc0LnLWq5dkGW88vWjcsUiaB9wNXG37e13sYycO+t7YfgL4EY1jP9PdeON5F/AZSTuBjwGfUOPDxNNZy7GUXdsvANh+gMax0rO72M92TfR37du2f2j7NeDrNI6vtC1B0toG4DIASWcDb6NxpcyNwICkWZLmAwuArd3qZAc20Ho8vWgDLcYi6QTga8Aq2/+3a73r3AZaj2e+pJml/gvAOcDOLvWxExtoMR7b77HdZ7sP+Bzw32xPuzMFx9hA6/dmthrfhUT5H/0CGsd+prsNtP478A3gnZKOKT9zvwI83tGWD+WMgMPpBnyVxlT1JzSSeXn5B/4y8BjwIHBZ0/r/hcb/QHYA7+92/ydhPDtpHJR7tay/sNtjOJSxAJ+k8b/2h5tup3az/xXH89s0Dko/XOpXdLv/VX/Wmtp9iul31lYn781vlPfmkVL/YLf7X/W9AX6rjOkx4DOdvl4ukRIREZVk11ZERFSSIImIiEoSJBERUUmCJCIiKkmQREREJQmSiIioJEESERGV/H+q5o6G5sZtMwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "df['phi'].plot.hist(bins=100);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d307895-30ea-42e3-bc94-53f27953ddde",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Question:</b> Why are there more observations at the beginning and end of the scan? (hint: think about partial observations)\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df5232d3-9ae0-46b3-8624-7a6bc14842a0",
   "metadata": {},
   "source": [
    "## 2) Corrections\n",
    "\n",
    "As explained in our 2020 Nature Communications paper , the intensity can be estimated as follows:\n",
    "\n",
    "$I_\\text{meas} = (r_\\text{crystal} - r_\\text{background})/(\\Delta\\Omega E A P)$\n",
    "\n",
    "Where:\n",
    "- $r_\\text{crystal} =$ observed count rate (photons per second)\n",
    "- $r_\\text{background} =$ estimated count rate from background sources\n",
    "- $\\Delta\\Omega =$ solid angle per pixel\n",
    "- $E =$ detection efficiency\n",
    "- $A =$ attenuation factor of air\n",
    "- $P =$ polarization factor\n",
    "\n",
    "*mdx2.correct* computes $I_\\text{meas}$ using this equation.\n",
    "\n",
    "Also, intensity errors are estimated using Poisson statistics:\n",
    "- $r_\\text{crystal} = N s^{-1}$ (counts per second)\n",
    "- $\\Delta r_\\text{crystal} = \\sqrt{N}s^{-1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e9b1642-211a-4aea-aaf9-ea6fc3825ca5",
   "metadata": {},
   "source": [
    "### *mdx2.correct*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3add1ba8-1ae3-4445-996e-f73e07bce08b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: mdx2.correct [-h] [--background BACKGROUND] [--attenuation TF]\n",
      "                    [--efficiency TF] [--polarization TF] [--outfile OUTFILE]\n",
      "                    geom hkl\n",
      "\n",
      "Apply corrections to integrated data\n",
      "\n",
      "positional arguments:\n",
      "  geom                  NeXus file with miller_index\n",
      "  hkl                   NeXus file with hkl_table\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  --background BACKGROUND\n",
      "                        NeXus file with background map (default: None)\n",
      "  --attenuation TF      apply attenuation correction? (default: True)\n",
      "  --efficiency TF       apply efficiency correction? (default: True)\n",
      "  --polarization TF     apply polarization correction? (default: True)\n",
      "  --outfile OUTFILE     name of the output NeXus file (default: corrected.nxs)\n"
     ]
    }
   ],
   "source": [
    "!mdx2.correct --help"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c99861e-9895-40a4-8e39-283821d2bb5d",
   "metadata": {},
   "source": [
    "### Parameters\n",
    "\n",
    "The command line interface allows you to turn off certain corrections, such as background, attenuation, efficiency and polarization.\n",
    "\n",
    "We'll apply all of them for the tutorial dataset.\n",
    "\n",
    "For background count rate we'll use the binned image stack, so that the background subtraction does not add noise to the measurement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "5533654f-fe6e-4a8a-b787-f7b6d4893c79",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading hkl_table from integrated.nxs\n",
      "  importing as HKLTable from mdx2.data\n",
      "Reading corrections from geometry.nxs\n",
      "  importing as Corrections from mdx2.geometry\n",
      "Reading crystal from geometry.nxs\n",
      "  importing as Crystal from mdx2.geometry\n",
      "Reading symmetry from geometry.nxs\n",
      "  importing as Symmetry from mdx2.geometry\n",
      "calculating scattering vector magnitude (s)\n",
      "mapping Miller indices to the asymmetric unit\n",
      "Reading binned_image_series from bkg_data_binned.nxs\n",
      "  importing as GridData from mdx2.geometry\n",
      "subtracting background from count rate\n",
      "correcting solid angle for attenuation\n",
      "correcting solid angle for efficiency\n",
      "correcting solid angle for polarization\n",
      "computing the swept reciprocal space volume fraction (rs_volume)\n",
      "computing intensity and intensity_error\n",
      "Exporting <class 'mdx2.data.HKLTable'> to nexus object\n",
      "  writing hkl_table to corrected.nxs\n",
      "done!\n"
     ]
    }
   ],
   "source": [
    "!mdx2.correct geometry.nxs integrated.nxs --background bkg_data_binned.nxs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38380182-9145-408b-b01f-e9d2b78b01d4",
   "metadata": {},
   "source": [
    "### Intensity statistics\n",
    "\n",
    "As before, we'll load the table as a pandas DataFrame.\n",
    "\n",
    "In addition to **intensity** and **intensity_error**, you'll notice there are some new columns.\n",
    "\n",
    "- **op** = index of the symmetry operator that mapped the observation point to the asymmetric unit\n",
    "- **h**, **k**, **l** = Miller indices mapped to the *asymmetric unit*\n",
    "- **s** = scattering vector magnitude (= 1/resolution)\n",
    "- **rs_volume** = volume of reciprocal space recorded as a fraction of the reciprocal unit cell\n",
    "\n",
    "In our case, a voxel that is \"fully recorded\" during the oscillation would have rs_volume = (1/4^3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9f88a905-2664-4bc8-8f5d-655e55179cad",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading hkl_table from corrected.nxs\n",
      "  importing as HKLTable from mdx2.data\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>intensity</th>\n",
       "      <th>intensity_error</th>\n",
       "      <th>ix</th>\n",
       "      <th>iy</th>\n",
       "      <th>phi</th>\n",
       "      <th>rs_volume</th>\n",
       "      <th>s</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>h</th>\n",
       "      <th>k</th>\n",
       "      <th>l</th>\n",
       "      <th>op</th>\n",
       "      <th>n</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">0.0</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">0.25</th>\n",
       "      <th>2.75</th>\n",
       "      <th>10</th>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>3458472.25</td>\n",
       "      <td>1222.937500</td>\n",
       "      <td>1211.979126</td>\n",
       "      <td>165.493744</td>\n",
       "      <td>0.000405</td>\n",
       "      <td>0.034724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.00</th>\n",
       "      <th>10</th>\n",
       "      <th>0</th>\n",
       "      <td>NaN</td>\n",
       "      <td>4526965.50</td>\n",
       "      <td>1220.806396</td>\n",
       "      <td>1210.483887</td>\n",
       "      <td>165.672577</td>\n",
       "      <td>0.000274</td>\n",
       "      <td>0.037856</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3.25</th>\n",
       "      <th>10</th>\n",
       "      <th>0</th>\n",
       "      <td>38040624.0</td>\n",
       "      <td>19222264.00</td>\n",
       "      <td>1218.000000</td>\n",
       "      <td>1209.000000</td>\n",
       "      <td>165.800003</td>\n",
       "      <td>0.000037</td>\n",
       "      <td>0.040990</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19.75</th>\n",
       "      <th>10</th>\n",
       "      <th>0</th>\n",
       "      <td>55826468.0</td>\n",
       "      <td>25134906.00</td>\n",
       "      <td>958.000000</td>\n",
       "      <td>1023.750000</td>\n",
       "      <td>165.949997</td>\n",
       "      <td>0.000209</td>\n",
       "      <td>0.248377</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20.00</th>\n",
       "      <th>10</th>\n",
       "      <th>0</th>\n",
       "      <td>54929968.0</td>\n",
       "      <td>11178633.00</td>\n",
       "      <td>954.549988</td>\n",
       "      <td>1021.049988</td>\n",
       "      <td>165.904999</td>\n",
       "      <td>0.001056</td>\n",
       "      <td>0.251520</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      intensity  intensity_error           ix           iy  \\\n",
       "h   k    l     op n                                                          \n",
       "0.0 0.25 2.75  10 0         NaN       3458472.25  1222.937500  1211.979126   \n",
       "         3.00  10 0         NaN       4526965.50  1220.806396  1210.483887   \n",
       "         3.25  10 0  38040624.0      19222264.00  1218.000000  1209.000000   \n",
       "         19.75 10 0  55826468.0      25134906.00   958.000000  1023.750000   \n",
       "         20.00 10 0  54929968.0      11178633.00   954.549988  1021.049988   \n",
       "\n",
       "                            phi  rs_volume         s  \n",
       "h   k    l     op n                                   \n",
       "0.0 0.25 2.75  10 0  165.493744   0.000405  0.034724  \n",
       "         3.00  10 0  165.672577   0.000274  0.037856  \n",
       "         3.25  10 0  165.800003   0.000037  0.040990  \n",
       "         19.75 10 0  165.949997   0.000209  0.248377  \n",
       "         20.00 10 0  165.904999   0.001056  0.251520  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# load table as mdx2 object\n",
    "tab = loadobj('corrected.nxs','hkl_table')\n",
    "\n",
    "# convert to pandas DataFrame\n",
    "df = tab.to_frame().set_index(['h','k','l','op','n']).sort_index()\n",
    "\n",
    "# show the first several rows\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d4895509-f55b-4e8c-9bca-82f58eb43ad5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEFCAYAAADew/HIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcCElEQVR4nO3df5DV9X3v8edLoEoSf4CslvKja5Q2VadB3SCNvfcaSYDq5KJ39GadTmQ6tKTG9CZzc3uLTiZYHWZ0pgmt06sNqYzITaLExMhNJGajadM0CqyWiPijbCvRDYyQLFFMIwZ83z++n6PfXc+e/S7s5xx29/WY+c75nvf3+/mczzl8Z998fpzvUURgZmY20o5rdQPMzGxscoIxM7MsnGDMzCwLJxgzM8vCCcbMzLKY2OoGHCumTZsW7e3trW6Gmdmo8vjjj/80ItrqHXOCSdrb2+nu7m51M8zMRhVJPx7smIfIzMwsCycYMzPLwgnGzMyycIIxM7MsnGDMzCwLJxgzM8vCCcbMzLJwgjEzsyycYMzMLAt/k9/e1L7iW4Me23XLZU1siZmNBe7BmJlZFk4wZmaWRbYEI+kESVsk/UjSDkl/meI3SvqJpG1pu7RU5npJPZKek7SoFL9A0vZ07DZJSvHjJd2b4psltZfKLJW0M21Lc71PMzOrL+cczEHgkoh4VdIk4AeSNqVjqyPir8onSzob6ATOAX4D+K6k34qIw8AdwHLgMeBBYDGwCVgG7I+IsyR1ArcCH5E0FVgJdAABPC5pY0Tsz/h+zcysJFsPJgqvpqeT0hYNiiwB7omIgxHxPNADzJM0HTgpIh6NiADuBi4vlVmX9u8DFqTezSKgKyL6UlLpokhKZmbWJFnnYCRNkLQN2EvxB39zOvQJSU9KWitpSorNAF4sFe9NsRlpf2C8X5mIOAS8DJzaoC4zM2uSrAkmIg5HxFxgJkVv5FyK4a4zgbnAHuBz6XTVq6JB/EjLvEnSckndkrr37dvX4J2YmdlwNWUVWUT8HPgHYHFEvJQSzxvAF4F56bReYFap2Exgd4rPrBPvV0bSROBkoK9BXQPbtSYiOiKio62t7i9+mpnZEcq5iqxN0ilpfzLwQeDZNKdScwXwVNrfCHSmlWFnAHOALRGxBzggaX6aX7kGeKBUprZC7ErgkTRP8xCwUNKUNAS3MMXMzKxJcq4imw6skzSBIpFtiIhvSlovaS7FkNUu4GMAEbFD0gbgaeAQcF1aQQZwLXAXMJli9VhtNdqdwHpJPRQ9l85UV5+km4Gt6bybIqIv43s1M7MBsiWYiHgSOK9O/KMNyqwCVtWJdwPn1om/Blw1SF1rgbXDaLKZmY0gf5PfzMyycIIxM7MsnGDMzCwLJxgzM8vCCcbMzLJwgjEzsyycYMzMLAsnGDMzy8IJxszMsnCCMTOzLJxgzMwsCycYMzPLwgnGzMyycIIxM7MsnGDMzCwLJxgzM8vCCcbMzLJwgjEzsyycYMzMLAsnGDMzyyJbgpF0gqQtkn4kaYekv0zxqZK6JO1Mj1NKZa6X1CPpOUmLSvELJG1Px26TpBQ/XtK9Kb5ZUnupzNL0GjslLc31Ps3MrL6cPZiDwCUR8V5gLrBY0nxgBfBwRMwBHk7PkXQ20AmcAywGbpc0IdV1B7AcmJO2xSm+DNgfEWcBq4FbU11TgZXAhcA8YGU5kZmZWX7ZEkwUXk1PJ6UtgCXAuhRfB1ye9pcA90TEwYh4HugB5kmaDpwUEY9GRAB3DyhTq+s+YEHq3SwCuiKiLyL2A128lZTMzKwJss7BSJogaRuwl+IP/mbg9IjYA5AeT0unzwBeLBXvTbEZaX9gvF+ZiDgEvAyc2qCuge1bLqlbUve+ffuO4p2amdlAWRNMRByOiLnATIreyLkNTle9KhrEj7RMuX1rIqIjIjra2toaNM3MzIarKavIIuLnwD9QDFO9lIa9SI9702m9wKxSsZnA7hSfWSfer4ykicDJQF+DuszMrElyriJrk3RK2p8MfBB4FtgI1FZ1LQUeSPsbgc60MuwMisn8LWkY7YCk+Wl+5ZoBZWp1XQk8kuZpHgIWSpqSJvcXppiZmTXJxIx1TwfWpZVgxwEbIuKbkh4FNkhaBrwAXAUQETskbQCeBg4B10XE4VTXtcBdwGRgU9oA7gTWS+qh6Ll0prr6JN0MbE3n3RQRfRnfq5mZDZAtwUTEk8B5deI/AxYMUmYVsKpOvBt42/xNRLxGSlB1jq0F1g6v1WZmNlL8TX4zM8vCCcbMzLJwgjEzsyycYMzMLAsnGDMzyyLnMmU7RrWv+Farm2Bm44B7MGZmloUTjJmZZeEhMqtksGG1Xbdc1uSWmNlo4R6MmZll4QRjZmZZOMGYmVkWTjBmZpaFE4yZmWXhBGNmZlk4wZiZWRZOMGZmloUTjJmZZeEEY2ZmWTjBmJlZFtkSjKRZkr4n6RlJOyR9MsVvlPQTSdvSdmmpzPWSeiQ9J2lRKX6BpO3p2G2SlOLHS7o3xTdLai+VWSppZ9qW5nqfZmZWX86bXR4CPh0RT0g6EXhcUlc6tjoi/qp8sqSzgU7gHOA3gO9K+q2IOAzcASwHHgMeBBYDm4BlwP6IOEtSJ3Ar8BFJU4GVQAcQ6bU3RsT+jO/XzMxKsvVgImJPRDyR9g8AzwAzGhRZAtwTEQcj4nmgB5gnaTpwUkQ8GhEB3A1cXiqzLu3fByxIvZtFQFdE9KWk0kWRlMzMrEmaMgeThq7OAzan0CckPSlpraQpKTYDeLFUrDfFZqT9gfF+ZSLiEPAycGqDuga2a7mkbknd+/btO/I3aGZmb5M9wUh6F/A14FMR8QrFcNeZwFxgD/C52ql1ikeD+JGWeSsQsSYiOiKio62trdHbMDOzYcqaYCRNokguX4qIrwNExEsRcTgi3gC+CMxLp/cCs0rFZwK7U3xmnXi/MpImAicDfQ3qMjOzJsm5ikzAncAzEfH5Unx66bQrgKfS/kagM60MOwOYA2yJiD3AAUnzU53XAA+UytRWiF0JPJLmaR4CFkqakobgFqaYmZk1Sc5VZBcBHwW2S9qWYjcAV0uaSzFktQv4GEBE7JC0AXiaYgXadWkFGcC1wF3AZIrVY5tS/E5gvaQeip5LZ6qrT9LNwNZ03k0R0ZflXZqZWV3ZEkxE/ID6cyEPNiizClhVJ94NnFsn/hpw1SB1rQXWVm2vmZmNLH+T38zMsnCCMTOzLJxgzMwsCycYMzPLwgnGzMyyqJRgJL1tBZeZmVkjVXswfydpi6SPSzolZ4PMzGxsqJRgIuL3gT+kuP1Kt6QvS/pQ1paZmdmoVnkOJiJ2Ap8B/gL4L8Btkp6V9N9yNc7MzEavqnMwvytpNcVvulwCfDgififtr87YPjMzG6Wq3irmbynufHxDRPyyFoyI3ZI+k6VlZmY2qlVNMJcCv6zdfFLSccAJEfEfEbE+W+vMzGzUqjoH812KOxnXvCPFzMzM6qqaYE6IiFdrT9L+O/I0yczMxoKqCeYXks6vPZF0AfDLBuebmdk4V3UO5lPAVyXVfnZ4OvCRLC0yM7MxoVKCiYitkt4D/DbFj4g9GxG/ytoyMzMb1Ybzi5bvA9pTmfMkERF3Z2mVmZmNepUSjKT1wJnANuBwCgfgBGNmZnVVneTvAC6KiI9HxJ+l7X80KiBplqTvSXpG0g5Jn0zxqZK6JO1Mj1NKZa6X1CPpOUmLSvELJG1Px26TpBQ/XtK9Kb5ZUnupzNL0GjslLR3GZ2JmZiOgaoJ5Cvj1YdZ9CPh0uqXMfOA6SWcDK4CHI2IO8HB6TjrWCZwDLAZulzQh1XUHsByYk7bFKb4M2B8RZ1HcsubWVNdUYCVwITAPWFlOZGZmll/VBDMNeFrSQ5I21rZGBSJiT0Q8kfYPUNzHbAawBFiXTlsHXJ72lwD3RMTBiHge6AHmSZoOnBQRj0ZEbViuXKZW133AgtS7WQR0RURfROwHungrKZmZWRNUneS/8WheJA1dnQdsBk6PiD1QJCFJp6XTZgCPlYr1ptiv0v7AeK3Mi6muQ5JeBk4tx+uUKbdrOUXPiNmzZx/5GzQzs7ep+nsw/wjsAial/a3AE1XKSnoX8DXgUxHxSqNT6710g/iRlnkrELEmIjoioqOtra1B08zMbLiq3q7/TyiGoL6QQjOAb1QoN4kiuXwpIr6ewi+lYS/S494U76X4QbOamcDuFJ9ZJ96vjKSJwMlAX4O6zMysSarOwVwHXAS8Am/++NhpjQqkuZA7gWci4vOlQxuB2qqupcADpXhnWhl2BsVk/pY0nHZA0vxU5zUDytTquhJ4JM3TPAQslDQlTe4vTDEzM2uSqnMwByPi9bQ6uNZbeNuQ0wAXAR8FtkvalmI3ALcAGyQtA14ArgKIiB2SNgBPU6xAu6728wDAtcBdFHd03pQ2KBLYekk9FD2XzlRXn6SbKYbyAG6KiL6K79XMzEZA1QTzj5JuACZL+hDwceD/NSoQET+g/lwIwIJByqwCVtWJdwPn1om/RkpQdY6tBdY2aqOZmeVTdYhsBbAP2A58DHgQ8C9ZmpnZoKre7PINip9M/mLe5piZ2VhR9V5kz1N/me+7R7xFZmY2JlSdg+ko7Z9AMe8xdeSbY2ZmY0XVL1r+rLT9JCL+Grgkb9PMzGw0qzpEdn7p6XEUPZoTs7TIzMzGhKpDZJ8r7R+iuG3Mfx/x1piZ2ZhRdRXZB3I3xMzMxpaqQ2T/s9HxAbeCMTMzG9YqsvdR3PsL4MPA9+l/S3wzM7M3VU0w04Dz0w+HIelG4KsR8ce5GmZmZqNb1VvFzAZeLz1/HWgf8daYmdmYUbUHsx7YIul+im/0X0Hx08VmZmZ1VV1FtkrSJuA/pdAfRcS/5GuWmZmNdlWHyADeAbwSEX8D9KYfBTMzM6ur6k8mrwT+Arg+hSYB/zdXo8zMbPSr2oO5AvivwC8AImI3vlWMmZk1UDXBvJ5+6z4AJL0zX5PMzGwsqJpgNkj6AnCKpD8Bvot/fMzMzBoYchWZJAH3Au8BXgF+G/hsRHRlbpuZmY1iQ/Zg0tDYNyKiKyL+PCL+V5XkImmtpL2SnirFbpT0E0nb0nZp6dj1knokPSdpUSl+gaTt6dhtKeEh6XhJ96b4ZkntpTJLJe1M29LqH4eZmY2UqkNkj0l63zDrvgtYXCe+OiLmpu1BAElnA53AOanM7ZImpPPvAJYDc9JWq3MZsD8izgJWA7emuqYCK4ELgXnASklThtl2MzM7SlUTzAcoksy/SXoy9SiebFQgIr4P9FWsfwlwT0QcjIjngR5gnqTpwEkR8WjqSd0NXF4qsy7t3wcsSL2bRUBXRPRFxH6gi/qJzszMMmo4ByNpdkS8APzBCL7mJyRdA3QDn05JYAbwWOmc3hT7VdofGCc9vggQEYckvQycWo7XKdOPpOUUvSNmz559dO9qnGpf8a268V23XNbklpjZsWaoHsw3ACLix8DnI+LH5e0IXu8O4ExgLrCHt34pU3XOjQbxIy3TPxixJiI6IqKjra2tQbPNzGy4hkow5T/W7z7aF4uIlyLicES8QbHMeV461AvMKp06E9id4jPrxPuVkTQROJliSG6wuszMrImGSjAxyP4RSXMqNVcAtRVmG4HOtDLsDIrJ/C0RsQc4IGl+ml+5BnigVKa2QuxK4JE0T/MQsFDSlDS5vzDFzMysiYb6Hsx7Jb1C0ZOZnPZJzyMiThqsoKSvABcD0yT1UqzsuljSXIpktQv4GEVFOyRtAJ4GDgHXRcThVNW1FCvSJgOb0gZwJ7BeUg9Fz6Uz1dUn6WZgazrvpoioutjAzMxGSMMEExETGh0fouzVdcJ3Njh/FbCqTrwbOLdO/DXgqkHqWgusrdxYMzMbccO5Xb+ZmVllTjBmZpaFE4yZmWXhBGNmZlk4wZiZWRZOMGZmloUTjJmZZeEEY2ZmWTjBmJlZFkP+ZLKNXoPdSt/MrBncgzEzsyycYMzMLAsnGDMzy8IJxszMsnCCMTOzLJxgzMwsCycYMzPLwgnGzMyycIIxM7MssiUYSWsl7ZX0VCk2VVKXpJ3pcUrp2PWSeiQ9J2lRKX6BpO3p2G2SlOLHS7o3xTdLai+VWZpeY6ekpbneo5mZDS5nD+YuYPGA2Arg4YiYAzycniPpbKATOCeVuV3ShFTmDmA5MCdttTqXAfsj4ixgNXBrqmsqsBK4EJgHrCwnMjMza45sCSYivg/0DQgvAdal/XXA5aX4PRFxMCKeB3qAeZKmAydFxKMREcDdA8rU6roPWJB6N4uArojoi4j9QBdvT3RmZpZZs+dgTo+IPQDp8bQUnwG8WDqvN8VmpP2B8X5lIuIQ8DJwaoO6zMysiY6VSX7ViUWD+JGW6f+i0nJJ3ZK69+3bV6mhZmZWTbMTzEtp2Iv0uDfFe4FZpfNmArtTfGadeL8ykiYCJ1MMyQ1W19tExJqI6IiIjra2tqN4W2ZmNlCzE8xGoLaqaynwQCnemVaGnUExmb8lDaMdkDQ/za9cM6BMra4rgUfSPM1DwEJJU9Lk/sIUMzOzJsr2g2OSvgJcDEyT1EuxsusWYIOkZcALwFUAEbFD0gbgaeAQcF1EHE5VXUuxIm0ysCltAHcC6yX1UPRcOlNdfZJuBram826KiIGLDczMLLNsCSYirh7k0IJBzl8FrKoT7wbOrRN/jZSg6hxbC6yt3FgzMxtxx8okv5mZjTFOMGZmlkW2ITIb39pXfKtufNctlzW5JWbWKu7BmJlZFk4wZmaWhROMmZll4QRjZmZZOMGYmVkWXkVmY45XsJkdG5xgbNxw4jFrLg+RmZlZFk4wZmaWhROMmZll4TkYG7UGm1Mxs2ODezBmZpaFezA27jXqCXmFmdmRc4KxpvJSYbPxw0NkZmaWhXswdszzZL7Z6OQEY8cEJxGzsaclQ2SSdknaLmmbpO4UmyqpS9LO9DildP71knokPSdpUSl+QaqnR9JtkpTix0u6N8U3S2pv+ps0MxvnWjkH84GImBsRHen5CuDhiJgDPJyeI+lsoBM4B1gM3C5pQipzB7AcmJO2xSm+DNgfEWcBq4Fbm/B+zMys5Fia5F8CrEv764DLS/F7IuJgRDwP9ADzJE0HToqIRyMigLsHlKnVdR+woNa7MTOz5mjVHEwA35EUwBciYg1wekTsAYiIPZJOS+fOAB4rle1NsV+l/YHxWpkXU12HJL0MnAr8tNwIScspekDMnj175N6djRleVm125FqVYC6KiN0piXRJerbBufV6HtEg3qhM/0CR2NYAdHR0vO24mZkduZYMkUXE7vS4F7gfmAe8lIa9SI970+m9wKxS8ZnA7hSfWSfer4ykicDJQF+O92JmZvU1PcFIeqekE2v7wELgKWAjsDSdthR4IO1vBDrTyrAzKCbzt6ThtAOS5qf5lWsGlKnVdSXwSJqnMTOzJmnFENnpwP1pzn0i8OWI+LakrcAGScuAF4CrACJih6QNwNPAIeC6iDic6roWuAuYDGxKG8CdwHpJPRQ9l85mvDEzM3uL/B/7QkdHR3R3d7e6GSPKX15sPk/+23gj6fHS10368Tf5xwAnEjM7Fh1L34MxM7MxxAnGzMyy8BCZ2QjyFzPN3uIejJmZZeEEY2ZmWTjBmJlZFp6DMWuCRkvJPT9jY5V7MGZmloV7MKOIv1BpZqOJezBmZpaFE4yZmWXhIbJjjIfBxh9/OdPGKvdgzMwsCycYMzPLwgnGzMyy8ByM2THKczM22rkHY2ZmWTjBmJlZFh4iMxtlPHRmo8WYTjCSFgN/A0wA/j4ibmlxk97k77vYSHPisWPNmE0wkiYA/wf4ENALbJW0MSKebm3LzJrLd3K2VhmzCQaYB/RExL8DSLoHWAJkSTDukdholPu6dQIb38ZygpkBvFh63gtcWD5B0nJgeXr6qqTnjuL1pgE/PYryubhdw+N2DU/DdunWJrakv1H5ebXQ0bTrNwc7MJYTjOrEot+TiDXAmhF5Mak7IjpGoq6R5HYNj9s1PG7X8Iy3do3lZcq9wKzS85nA7ha1xcxs3BnLCWYrMEfSGZJ+DegENra4TWZm48aYHSKLiEOSPgE8RLFMeW1E7Mj4kiMy1JaB2zU8btfwuF3DM67apYgY+iwzM7NhGstDZGZm1kJOMGZmloUTzBAkLZb0nKQeSSvqHJek29LxJyWdX7Vs5nb9YWrPk5J+KOm9pWO7JG2XtE1Sd5PbdbGkl9Nrb5P02aplM7frz0ttekrSYUlT07Gcn9daSXslPTXI8VZdX0O1q1XX11DtatX1NVS7WnV9zZL0PUnPSNoh6ZN1zsl3jUWEt0E2isUB/wa8G/g14EfA2QPOuRTYRPG9m/nA5qplM7fr/cCUtP8HtXal57uAaS36vC4GvnkkZXO2a8D5HwYeyf15pbr/M3A+8NQgx5t+fVVsV9Ovr4rtavr1VaVdLby+pgPnp/0TgX9t5t8w92Aae/N2MxHxOlC73UzZEuDuKDwGnCJpesWy2doVET+MiP3p6WMU3wPK7Wjec0s/rwGuBr4yQq/dUER8H+hrcEorrq8h29Wi66vK5zWYln5eAzTz+toTEU+k/QPAMxR3OSnLdo05wTRW73YzA/9xBjunStmc7SpbRvE/lJoAviPpcRW3yxkpVdv1e5J+JGmTpHOGWTZnu5D0DmAx8LVSONfnVUUrrq/hatb1VVWzr6/KWnl9SWoHzgM2DziU7Robs9+DGSFD3m6mwTlVyh6pynVL+gDFH4DfL4Uviojdkk4DuiQ9m/4H1ox2PQH8ZkS8KulS4BvAnIplc7ar5sPAP0dE+X+juT6vKlpxfVXW5OurilZcX8PRkutL0rsoktqnIuKVgYfrFBmRa8w9mMaq3G5msHNy3qqmUt2Sfhf4e2BJRPysFo+I3elxL3A/RVe4Ke2KiFci4tW0/yAwSdK0KmVztqukkwHDFxk/rypacX1V0oLra0gtur6Go+nXl6RJFMnlSxHx9Tqn5LvGckwsjZWNoof378AZvDXJdc6Acy6j/wTZlqplM7drNtADvH9A/J3AiaX9HwKLm9iuX+etL/jOA15In11LP6903skU4+jvbMbnVXqNdgaftG769VWxXU2/viq2q+nXV5V2ter6Su/9buCvG5yT7RrzEFkDMcjtZiT9aTr+d8CDFKsweoD/AP6oUdkmtuuzwKnA7ZIADkVxt9TTgftTbCLw5Yj4dhPbdSVwraRDwC+Bziiu5lZ/XgBXAN+JiF+Uimf7vAAkfYVi5dM0Sb3ASmBSqV1Nv74qtqvp11fFdjX9+qrYLmjB9QVcBHwU2C5pW4rdQPEfhOzXmG8VY2ZmWXgOxszMsnCCMTOzLJxgzMwsCycYMzPLwgnGzGycGuomnQPOnZ1unPkv6aaYlw5VxgnGzGz8uovi1jVVfAbYEBHnUXxh9PahCjjBmJmNU1HnJp2SzpT07XRvtH+S9J7a6cBJaf9kKnyr31+0NDOzsjXAn0bETkkXUvRULgFupLgp559R3HXgg0NV5ARjZmbAmzfFfD/w1XR3AYDj0+PVwF0R8TlJvwesl3RuRLwxWH1OMGZmVnMc8POImFvn2DLSfE1EPCrpBGAasLdRZWZmZkRxK//nJV0Fb/6ccu3nsF8AFqT47wAnAPsa1ed7kZmZjVPlm3QCL1HcpPMR4A6Kn1ueBNwTETdJOhv4IvAuign//x0R32lYvxOMmZnl4CEyMzPLwgnGzMyycIIxM7MsnGDMzCwLJxgzM8vCCcbMzLJwgjEzsyz+PxMhk7R5HwS/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot intensity histogram\n",
    "df['intensity'].plot.hist(bins=np.linspace(0,2E8));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3f607461-5aec-45fe-aad7-45c21780e091",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaYAAAD4CAYAAACngkIwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAdnUlEQVR4nO3df5DV9X3v8ecrYBvSRAO6egmLXRJJW3QalA1xJs29SWiAahNMqw2Z3si9l1sSr84kt71zA0mnWB3mSltr6rQxl1SuQBOBmKTSGK9dNT+aGQWWhIiolL2R6goD2yxVkokkmPf94/s5+t317NnDst89n919PWa+c77nfb6fz3mfw3d57/f7/ez3o4jAzMwsF69pdQJmZmZlLkxmZpYVFyYzM8uKC5OZmWXFhcnMzLIytdUJ5OK8886Ljo6OVqdhZjau7Nmz518jom00+3RhSjo6Ouju7m51GmZm44qkfxntPn0qz8zMsuLCZGZmWXFhMjOzrLgwmZlZVlyYzMwsKy5MZmaWFRcmMzPLiguTmZllxYXJzMyy4js/mJkNoWP1fS+vH7rlyhZmMrn4iMnMzLLiwmRmZllxYTIzs6y4MJmZWVYqL0ySpkj6nqSvpeczJHVJOpgep5e2XSOpR9IBSUtK8QWS9qXXbpekFP9FSdtSfKekjlKbFek9DkpaUfXnNDOz0TEWR0wfB54sPV8NPBQRc4GH0nMkzQOWAxcDS4HPSpqS2twBrALmpmVpiq8EjkfERcBtwPrU1wxgLfAOYCGwtlwAzcwsX5UWJkntwJXA35bCy4BNaX0TcFUpvjUiTkbE00APsFDSTODsiHgkIgLYPKhNra97gEXpaGoJ0BUR/RFxHOjilWJmZmYZq/qI6TPA/wR+XopdEBFHANLj+Sk+C3i2tF1vis1K64PjA9pExCngeeDcBn0NIGmVpG5J3X19fSP4eGZmNtoqK0ySfhs4FhF7mm1SJxYN4iNt80ogYkNEdEZEZ1vbqE5Zb2ZmI1TlEdM7gQ9IOgRsBd4r6e+Ao+n0HOnxWNq+F5hdat8OHE7x9jrxAW0kTQXOAfob9GVmZpmrrDBFxJqIaI+IDopBDQ9HxH8EdgC1UXIrgHvT+g5geRppN4dikMOudLrvhKTL0/Wjawe1qfV1dXqPAB4AFkuangY9LE4xMzPLXCvulXcLsF3SSuAZ4BqAiNgvaTvwBHAKuD4iXkptrgPuAqYB96cF4E5gi6QeiiOl5amvfkk3A7vTdjdFRH/VH8zMzM7cmBSmiPgm8M20/kNg0RDbrQPW1Yl3A5fUib9IKmx1XtsIbBxpzmZm1hq+84OZmWXFhcnMzLLiwmRmZllxYTIzs6y4MJmZWVZcmMzMLCsuTGZmlhUXJjMzy4oLk5mZZcWFyczMsuLCZGZmWXFhMjOzrLgwmZlZVlyYzMwsKy5MZmaWFRcmMzPLSmWFSdJrJe2S9H1J+yX9aYrfKOk5SXvTckWpzRpJPZIOSFpSii+QtC+9dnuaYp00Dfu2FN8pqaPUZoWkg2lZgZmZjQtVzmB7EnhvRPxI0lnAdyTVpkS/LSL+oryxpHkUU6NfDLwJeFDSW9P06ncAq4BHga8DSymmV18JHI+IiyQtB9YDH5I0A1gLdAIB7JG0IyKOV/h5zcxsFFR2xBSFH6WnZ6UlGjRZBmyNiJMR8TTQAyyUNBM4OyIeiYgANgNXldpsSuv3AIvS0dQSoCsi+lMx6qIoZmZmlrlKrzFJmiJpL3CMolDsTC/dIOkxSRslTU+xWcCzpea9KTYrrQ+OD2gTEaeA54FzG/Q1OL9Vkroldff19Y38g5qZ2aiptDBFxEsRMR9opzj6uYTitNxbgPnAEeDWtLnqddEgPtI25fw2RERnRHS2tbU1+CRmZjZWxmRUXkT8G/BNYGlEHE0F6+fA54GFabNeYHapWTtwOMXb68QHtJE0FTgH6G/Ql5mZZa7KUXltkt6Y1qcBvwk8la4Z1XwQeDyt7wCWp5F2c4C5wK6IOAKckHR5un50LXBvqU1txN3VwMPpOtQDwGJJ09OpwsUpZmZmmatyVN5MYJOkKRQFcHtEfE3SFknzKU6tHQI+ChAR+yVtB54ATgHXpxF5ANcBdwHTKEbj1Ub33QlskdRDcaS0PPXVL+lmYHfa7qaI6K/ws5qZ2SiprDBFxGPApXXiH2nQZh2wrk68G7ikTvxF4Joh+toIbDyNlM3MLAO+84OZmWXFhcnMzLLiwmRmZllxYTIzs6y4MJmZWVZcmMzMLCsuTGZmlhUXJjMzy4oLk5mZZcWFyczMslLlvfLMzMadjtX3tTqFSc9HTGZmlhUXJjMzy4oLk5mZZcWFyczMsuLCZGZmWalyavXXStol6fuS9kv60xSfIalL0sH0OL3UZo2kHkkHJC0pxRdI2pdeuz1NsU6ahn1biu+U1FFqsyK9x0FJKzAzs3GhyiOmk8B7I+JtwHxgqaTLgdXAQxExF3goPUfSPIqp0S8GlgKfTdOyA9wBrALmpmVpiq8EjkfERcBtwPrU1wxgLfAOYCGwtlwAzcwsX5UVpij8KD09Ky0BLAM2pfgm4Kq0vgzYGhEnI+JpoAdYKGkmcHZEPBIRAWwe1KbW1z3AonQ0tQToioj+iDgOdPFKMTMzs4xVeo1J0hRJe4FjFIViJ3BBRBwBSI/np81nAc+Wmvem2Ky0Pjg+oE1EnAKeB85t0Nfg/FZJ6pbU3dfXdwaf1MzMRkulhSkiXoqI+UA7xdHPJQ02V70uGsRH2qac34aI6IyIzra2tgapmZnZWBmTUXkR8W/ANylOpx1Np+dIj8fSZr3A7FKzduBwirfXiQ9oI2kqcA7Q36AvMzPLXJWj8tokvTGtTwN+E3gK2AHURsmtAO5N6zuA5Wmk3RyKQQ670um+E5IuT9ePrh3UptbX1cDD6TrUA8BiSdPToIfFKWZmZpmr8iauM4FNaWTda4DtEfE1SY8A2yWtBJ4BrgGIiP2StgNPAKeA6yPipdTXdcBdwDTg/rQA3AlskdRDcaS0PPXVL+lmYHfa7qaI6K/ws5qZ2SiprDBFxGPApXXiPwQWDdFmHbCuTrwbeNX1qYh4kVTY6ry2Edh4elmbmVmr+c4PZmaWFRcmMzPLiguTmZllxYXJzMyy4sJkZmZZcWEyM7OsuDCZmVlWXJjMzCwrLkxmZpYVFyYzM8uKC5OZmWWlqcI0zDxKZmZmo6bZI6bPSdol6b/VprIwMzOrQlOFKSJ+A/h9isn3uiV9UdL7Ks3MzMwmpaavMUXEQeCPgU8C/wG4XdJTkn6nquTMzGzyafYa069Lug14Engv8P6I+LW0ftsQbWZL+oakJyXtl/TxFL9R0nOS9qblilKbNZJ6JB2QtKQUXyBpX3rt9jSTLWm2220pvlNSR6nNCkkH07ICMzMbF5qdKPCvgc8Dn4qIn9SCEXFY0h8P0eYU8EcR8V1JbwD2SOpKr90WEX9R3ljSPIoZaC8G3gQ8KOmtaRbbO4BVwKPA14GlFLPYrgSOR8RFkpYD64EPSZoBrAU6gUjvvSMijjf5ec3MrEWaPZV3BfDFWlGS9BpJrwOIiC31GkTEkYj4blo/QXG0NavBeywDtkbEyYh4GugBFkqaCZwdEY9ERACbgatKbTal9XuAReloagnQFRH9qRh1URQzMzPLXLOF6UFgWun561KsKekU26XAzhS6QdJjkjZKmp5is4BnS816U2xWWh8cH9AmIk4BzwPnNujLzMwy12xhem1E/Kj2JK2/rpmGkl4PfBn4RES8QHFa7i3AfOAIcGtt0zrNo0F8pG3Kua2S1C2pu6+vr9HHMDOzMdJsYfqxpMtqTyQtAH7SYPvadmdRFKUvRMRXACLiaES8FBE/p7hutTBt3ksxHL2mHTic4u114gPaSJoKnAP0N+hrgIjYEBGdEdHZ1tY23McxM7Mx0Gxh+gTwJUn/JOmfgG3ADY0apGs9dwJPRsRfluIzS5t9EHg8re8AlqeRdnOAucCuiDgCnJB0eerzWuDeUpvaiLurgYfTdagHgMWSpqdThYtTzMzMMtfUqLyI2C3pV4FfoThN9lRE/GyYZu8EPgLsk7Q3xT4FfFjSfIpTa4eAj6b32C9pO/AExYi+69OIPIDrgLsornPdnxYoCt8WST0UR0rLU1/9km4GdqftboqI/mY+q5mZtVazw8UB3g50pDaXSiIiNg+1cUR8h/rXer7eoM06YF2deDfwqvv1RcSLwDVD9LUR2DjUe5mZWZ6aKkyStlAMWNgL1I5iakO3zczMRk2zR0ydwLx0/cbMzKwyzQ5+eBz4d1UmYmZmBs0fMZ0HPCFpF3CyFoyID1SSlZmZTVrNFqYbq0zCzMysptnh4t+S9MvA3Ih4MN0nb0q1qZmZ2WTU7Ki8P6C4u/cMitF5s4DPAYuqS83MLB8dq+97ef3QLVe2MJOJr9nBD9dT/MHsC/DypIHnV5WUmZlNXs0WppMR8dPak3RfOg8dNzOzUddsYfqWpE8B0yS9D/gS8A/VpWVmZpNVs4VpNdAH7KO4t93XgaFmrjUzMxuxZkfl1aao+Hy16ZiZ2WTX7Ki8p6lzTSki3jzqGZmZ2aR2OvfKq3ktxR29Z4x+OmZmNtk1dY0pIn5YWp6LiM8A7602NTMzm4yaPZV3WenpayiOoN5QSUZmZjapNTsq79bS8r+ABcDvNWogabakb0h6UtJ+SR9P8RmSuiQdTI/TS23WSOqRdEDSklJ8gaR96bXb0xTrpGnYt6X4TkkdpTYr0nsclLQCMzMbF5odlfeeEfR9CvijiPiupDcAeyR1Af8JeCgibpG0mmIo+iclzaOYGv1i4E3Ag5LemqZXv4PilkiPUgxVX0oxvfpK4HhEXCRpObAe+JCkGcBaiiO7SO+9IyKOj+BzmJnZGGr2VN4fNno9Iv6yTuwIcCStn5D0JMU99pYB706bbQK+CXwyxbdGxEngaUk9wEJJh4CzI+KRlMtm4CqKwrSMV+58fg/w1+loagnQFRH9qU0XRTG7u5nPa2ZmrXM6o/LeDuxIz98PfBt4tpnG6RTbpcBO4IJUtIiII5Jq99ybRXFEVNObYj9L64PjtTbPpr5OSXoeOLccr9OmnNcqiiMxLrzwwmY+ipmZVex0Jgq8LCJOAEi6EfhSRPzX4RpKej3wZeATEfFCujxUd9M6sWgQH2mbVwIRG4ANAJ2dnb73n5lZBpod/HAh8NPS858CHcM1knQWRVH6QkR8JYWPSpqZXp8JHEvxXmB2qXk7cDjF2+vEB7RJN5Y9B+hv0JeZmWWu2cK0Bdgl6UZJaylOyW1u1CBd67kTeHLQNagdQG2U3Arg3lJ8eRppNweYC+xKp/1OSLo89XntoDa1vq4GHo6IAB4AFkuankb9LU4xMzPLXLOj8tZJuh94Vwr954j43jDN3gl8BNgnaW+KfQq4BdguaSXwDMVdJIiI/ZK2A09QjOi7Po3IA7gOuAuYRjHo4f4UvxPYkgZK9FOM6iMi+iXdDOxO291UGwhhZmZ5a/YaE8DrgBci4v9IapM0JyKeHmrjiPgO9a/1wBAz30bEOmBdnXg3cEmd+IukwlbntY3AxqHyMzOzPDV1Ki+dvvsksCaFzgL+rqqkzMxs8mr2GtMHgQ8APwaIiMP4lkRmZlaBZgvTT9OgggCQ9EvVpWRmZpNZs4Vpu6T/DbxR0h8AD+JJA83MrALDDn5IQ7S3Ab8KvAD8CvAnEdFVcW5mZjYJDVuYIiIk/X1ELABcjMzMrFLNnsp7VNLbK83EzMyM5v+O6T3Ax9Kdvn9M8fdJERG/XlViZmY2OTUsTJIujIhngN8ao3zMzGySG+6I6e8p7ir+L5K+HBG/OwY5mZnZJDbcNabyLYXeXGUiZmZmMHxhiiHWzczMKjHcqby3SXqB4shpWlqHVwY/nF1pdmZmNuk0LEwRMWWsEjEzM4Pm/47JzMxsTLgwmZlZViorTJI2Sjom6fFS7EZJz0nam5YrSq+tkdQj6YCkJaX4Akn70mu3p3v3kaZg35biOyV1lNqskHQwLbWp183MbByo8ojpLmBpnfhtETE/LV8HkDSPYlr0i1Obz0qqXd+6A1gFzE1Lrc+VwPGIuAi4DVif+poBrAXeASwE1kqaPvofz8zMqlBZYYqIbwP9TW6+DNgaESfTdO09wEJJM4GzI+KRNB/UZuCqUptNaf0eYFE6mloCdEVEf0Qcp7jxbL0CaWZmGWrFNaYbJD2WTvXVjmRmAc+WtulNsVlpfXB8QJuIOAU8D5zboK9XkbRKUrek7r6+vjP7VGZmNirGujDdAbwFmA8cAW5NcdXZNhrER9pmYDBiQ0R0RkRnW1tbg7TNzGysjGlhioijEfFSRPycYgbchemlXmB2adN24HCKt9eJD2gjaSpwDsWpw6H6MjOzcWBMC1O6ZlTzQaA2Ym8HsDyNtJtDMchhV0QcAU5IujxdP7oWuLfUpjbi7mrg4XQd6gFgsaTp6VTh4hQzM7NxoNn5mE6bpLuBdwPnSeqlGCn3bknzKU6tHQI+ChAR+yVtB54ATgHXR8RLqavrKEb4TQPuTwvAncAWST0UR0rLU1/9km4GdqftboqIZgdhmNkk1LH6vlanYCWVFaaI+HCd8J0Ntl8HrKsT7wYuqRN/EbhmiL42AhubTtbMzLLhOz+YmVlWXJjMzCwrLkxmZpYVFyYzM8uKC5OZmWXFhcnMzLLiwmRmZllxYTIzs6y4MJmZWVZcmMzMLCsuTGZmlhUXJjMzy4oLk5mZZcWFyczMsuLCZGZmWXFhMjOzrFRWmCRtlHRM0uOl2AxJXZIOpsfppdfWSOqRdEDSklJ8gaR96bXb0xTrpGnYt6X4TkkdpTYr0nsclFSbft3MzMaBKo+Y7gKWDoqtBh6KiLnAQ+k5kuZRTI1+cWrzWUlTUps7gFXA3LTU+lwJHI+Ii4DbgPWprxkU07i/A1gIrC0XQDMzy1tlhSkivg30DwovAzal9U3AVaX41og4GRFPAz3AQkkzgbMj4pGICGDzoDa1vu4BFqWjqSVAV0T0R8RxoItXF0gzM8vUWF9juiAijgCkx/NTfBbwbGm73hSbldYHxwe0iYhTwPPAuQ36ehVJqyR1S+ru6+s7g49lZmajJZfBD6oTiwbxkbYZGIzYEBGdEdHZ1tbWVKJmZlatsS5MR9PpOdLjsRTvBWaXtmsHDqd4e534gDaSpgLnUJw6HKovMzMbB8a6MO0AaqPkVgD3luLL00i7ORSDHHal030nJF2erh9dO6hNra+rgYfTdagHgMWSpqdBD4tTzMzMxoGpVXUs6W7g3cB5knopRsrdAmyXtBJ4BrgGICL2S9oOPAGcAq6PiJdSV9dRjPCbBtyfFoA7gS2SeiiOlJanvvol3QzsTtvdFBGDB2GYmVmmVBxkWGdnZ3R3d7c6DTNrgY7V94247aFbrhzFTMYfSXsionM0+8xl8IOZmRngwmRmZplxYTIzs6y4MJmZWVZcmMzMLCsuTGZmlhUXJjMzy4oLk5mZZcWFyczMsuLCZGZmWXFhMjOzrFR2E1czs5ydyf3xrFo+YjIzs6y4MJmZWVZcmMzMLCsuTGZmlpWWFCZJhyTtk7RXUneKzZDUJelgepxe2n6NpB5JByQtKcUXpH56JN2epl8nTdG+LcV3SuoY8w9pZmYj0sojpvdExPzSzIergYciYi7wUHqOpHkU06ZfDCwFPitpSmpzB7AKmJuWpSm+EjgeERcBtwHrx+DzmJnZKMjpVN4yYFNa3wRcVYpvjYiTEfE00AMslDQTODsiHolifvjNg9rU+roHWFQ7mjIzs7y1qjAF8I+S9khalWIXRMQRgPR4forPAp4tte1NsVlpfXB8QJuIOAU8D5w7OAlJqyR1S+ru6+sblQ9mZmZnplV/YPvOiDgs6XygS9JTDbatd6QTDeKN2gwMRGwANgB0dna+6nUzMxt7LSlMEXE4PR6T9FVgIXBU0syIOJJO0x1Lm/cCs0vN24HDKd5eJ15u0ytpKnAO0F/V54GBf0V+6JYrq3wrM7MJbcxP5Un6JUlvqK0Di4HHgR3AirTZCuDetL4DWJ5G2s2hGOSwK53uOyHp8nT96NpBbWp9XQ08nK5DmZlZ5lpxxHQB8NU0FmEq8MWI+L+SdgPbJa0EngGuAYiI/ZK2A08Ap4DrI+Kl1Nd1wF3ANOD+tADcCWyR1ENxpLR8LD6YmZmduTEvTBHxA+BtdeI/BBYN0WYdsK5OvBu4pE78RVJhMzOz8cV3FzczOwO+vjz6XJjMbNLwVBfjQ05/YGtmZubCZGZmefGpvAr4nLOZ2cj5iMnMzLLiwmRmZlnxqTwzm9A8Em/8cWGqmK83mZmdHhcmM7NR4l9ER4cL0xjyTmtmNjwXphZxkTKrjq8rjW8uTBlwkTI7cy5GE4cLU2aG+uFywbJWy/EXKBejiUmeP6/Q2dkZ3d3dI26f2w9ILv9x2PjWzH491vtabj9rzZjIP4+S9kRE52j26SOmCSrHH96J/MM5kZzuvtNo+2b+zXPcV621JvQRk6SlwF8BU4C/jYhbhtp2oh0xmVm+JtIvaT5iOg2SpgB/A7wP6AV2S9oREU+0NjMzm+x8LbmxCVuYgIVAT5rKHUlbgWWAC5OZZel0z7xM1EI2kQvTLODZ0vNe4B3lDSStAlalpz+SdOAM3u884F/PoH2Vcs4N8s4v59wg7/xyzg3yzq+p3LR+DDKpr5zfL4925xO5MKlObMAFtYjYAGwYlTeTukf7POtoyTk3yDu/nHODvPPLOTfIO7+cc4Pq85vI0170ArNLz9uBwy3KxczMmjSRC9NuYK6kOZJ+AVgO7GhxTmZmNowJeyovIk5JugF4gGK4+MaI2F/hW47KKcGK5Jwb5J1fzrlB3vnlnBvknV/OuUHF+U3ov2MyM7PxZyKfyjMzs3HIhcnMzLLiwpRIWirpgKQeSavrvC5Jt6fXH5N02XBtJc2Q1CXpYHqcXnptTdr+gKQlLcjtzyU9lbb/qqQ3pniHpJ9I2puWz7Xou7tR0nOlPK7I6LvbVsrrkKS9LfruNko6JunxQW1y2O+Gyi2X/W6o/HLY74bKreX7naTZkr4h6UlJ+yV9vNRmVPY7ACJi0i8UgyP+H/Bm4BeA7wPzBm1zBXA/xd9HXQ7sHK4t8GfA6rS+Glif1uel7X4RmJPaTxnj3BYDU9P6+lJuHcDjGXx3NwL/o877tfy7G9T+VuBPxvq7S6/9e+Cywe/Z6v1umNxavt8Nk19L97tGueWw3wEzgcvS+huAf2YU/7+rLT5iKrx8+6KI+ClQu31R2TJgcxQeBd4oaeYwbZcBm9L6JuCqUnxrRJyMiKeBntTPmOUWEf8YEadS+0cp/s5rJKr67obS8u+uRpKA3wPuHibnoZxJfkTEt4H+Ov22er8bMrdM9rtG391QWv7d1bRyv4uIIxHx3ZTnCeBJirvs1Nqc6X4H+FReTb3bF81qcptGbS+IiCMA6fH803i/qnMr+y8Uvx3VzJH0PUnfkvSuIfIai/xuSKcRNpZOC+T03b0LOBoRB0uxsfruGmn1ftesVu13w2nlfteMLPY7SR3ApcDOFBqN/Q5wYaoZ9vZFDbZppu1I3u90th1xbpI+DZwCvpBCR4ALI+JS4A+BL0o6e4jcqszvDuAtwPyU062n8X5V51bzYQb+1jqW391IjNV3N3wird3vGmn1fteMlu93kl4PfBn4RES8MEy+p/1ZXZgKzdy+aKhtGrU9Wjs8T4/HTuP9qs4NSSuA3wZ+P9LJ4HS4/cO0vofifPBbh8itsvwi4mhEvBQRPwc+zyuH/rl8d1OB3wG21WJj/N010ur9rqEM9rshZbDfNZTDfifpLIqi9IWI+Eppm9HY717+UJN+obgDxg8oLszVLgZePGibKxl4MXDXcG2BP2fgxcA/S+sXM/Bi4A8Y+kJqVbktpZgCpG1QX221XCgujj4HzGjBdzez1P6/U5yjzuK7K31/32rVd1d6vYNXX8Bv6X43TG4t3++Gya+l+12j3HLY79LzzcBn6vR7xvvdy301enEyLRSjUP6Z4reNT6fYx4CPlf5B/ia9vg/obNQ2xc8FHgIOpscZpdc+nbY/APxWC3LroTjvuzctn0vx3wX2px3pu8D7W/TdbUnbPkZxj8OZuXx36bW7an2UYmP93d1NcRrnZxS/la7MaL8bKrdc9ruh8sthv6ubWw77HfAbFKfhHiv9G14xmvtdRPiWRGZmlhdfYzIzs6y4MJmZWVZcmMzMLCsuTGZmlhUXJjMzy4oLk5mZZcWFyczMsvL/AUQcy0t+mIFHAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot fractional reciprocal space volume per observation\n",
    "df['rs_volume'].plot.hist(bins=100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "0b14ac07-154e-4159-9047-683b37b08bf2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.015625"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fractional volume of one voxel\n",
    "1/4**3 # --subdivide 4 4 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "419020e4-372a-4a5e-a764-79ab272b9a18",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEACAYAAACj0I2EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOgklEQVR4nO3cf6zddX3H8efLdhgZCmatzLXFdktRGwdMr8XN/YAxscCyZsu2AGZkBGxIwJksMzTbMrOYJRizbBrBpmGVsETJdE7L6GyWbP6Y2q0Xx6+W1NwVRu8KclGUIGa18N4f57Acj6f3nLbn3tP74flIyL3f7/fTc94ntM9++73fc1JVSJKWvpdNegBJ0ngYdElqhEGXpEYYdElqhEGXpEYYdElqxESDnmRHkieTPDTi+t9Nsj/JviSfWOj5JGkpySTvQ0/yy8CzwJ1V9aYha9cDfwf8alU9neQ1VfXkYswpSUvBRM/Qq+pLwLd79yX5mSSfT3Jvki8neUP30LuBW6vq6e6vNeaS1ONUvIa+HXhPVb0F+CPgtu7+c4Fzk3wlyZ4kmyY2oSSdgpZPeoBeSc4AfgH4VJIXd7+8+3U5sB64CFgNfDnJm6rqO4s8piSdkk6poNP5F8N3quqCAcdmgT1V9QPgkSQH6AR+7yLOJ0mnrFPqkktVPUMn1r8DkI7zu4c/C1zc3b+CziWYg5OYU5JORZO+bfGTwNeA1yeZTXId8C7guiT3A/uAzd3lu4FvJdkP/Cvwvqr61iTmlqRT0URvW5Qkjc8pdclFknTihgZ92Ls5u9e5P5JkJskDSd48/jElScOMcpfLHcBHgTuPcfwyOnebrAcuBD7W/TqvFStW1Nq1a0caUpLUce+99z5VVSsHHRsa9Kr6UpK18yzZTOet+wXsSXJWktdW1ePzPe7atWuZnp4e9vSSpB5J/vtYx8ZxDX0VcKhne7a7T5K0iMYR9AzYN/DWmSRbkkwnmZ6bmxvDU0uSXjSOoM8Ca3q2VwOHBy2squ1VNVVVUytXDrwEJEk6QeMI+k7gmu7dLm8Dvjvs+rkkafyG/lC0+27Oi4AVSWaB9wM/BlBV24BdwOXADPAccO1CDStJOrZR7nK5asjxAm4c20SSpBPiO0UlqREGXZIacap9HvpQa7fes6jP9+gtVyzq80nSifIMXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaMVLQk2xKciDJTJKtA46fmeTuJPcn2Zfk2vGPKkmaz9CgJ1kG3ApcBmwArkqyoW/ZjcD+qjofuAj4yySnjXlWSdI8RjlD3wjMVNXBqjoC3AVs7ltTwCuTBDgD+DZwdKyTSpLmNUrQVwGHerZnu/t6fRR4I3AYeBB4b1W9MJYJJUkjGSXoGbCv+rbfCdwH/BRwAfDRJK/6kQdKtiSZTjI9Nzd3nKNKkuYzStBngTU926vpnIn3uhb4THXMAI8Ab+h/oKraXlVTVTW1cuXKE51ZkjTAKEHfC6xPsq77g84rgZ19ax4DLgFIcjbweuDgOAeVJM1v+bAFVXU0yU3AbmAZsKOq9iW5oXt8G/AB4I4kD9K5RHNzVT21gHNLkvoMDTpAVe0CdvXt29bz/WHg0vGOJkk6Hr5TVJIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREGXZIaYdAlqREjBT3JpiQHkswk2XqMNRcluS/JviRfHO+YkqRhlg9bkGQZcCvwDmAW2JtkZ1Xt71lzFnAbsKmqHkvymgWaV5J0DKOcoW8EZqrqYFUdAe4CNvetuRr4TFU9BlBVT453TEnSMKMEfRVwqGd7truv17nAq5N8Icm9Sa4Z14CSpNEMveQCZMC+GvA4bwEuAV4BfC3Jnqr6xg89ULIF2AJwzjnnHP+0kqRjGuUMfRZY07O9Gjg8YM3nq+p7VfUU8CXg/P4HqqrtVTVVVVMrV6480ZklSQOMEvS9wPok65KcBlwJ7Oxb8zngl5IsT3I6cCHw8HhHlSTNZ+gll6o6muQmYDewDNhRVfuS3NA9vq2qHk7yeeAB4AXg9qp6aCEHlyT9sFGuoVNVu4Bdffu29W1/CPjQ+EaTJB0P3ykqSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUCIMuSY0w6JLUiJGCnmRTkgNJZpJsnWfdW5M8n+S3xzeiJGkUQ4OeZBlwK3AZsAG4KsmGY6z7ILB73ENKkoYb5Qx9IzBTVQer6ghwF7B5wLr3AH8PPDnG+SRJIxol6KuAQz3bs919/y/JKuA3gW3jG02SdDxGCXoG7Ku+7b8Gbq6q5+d9oGRLkukk03NzcyOOKEkaxfIR1swCa3q2VwOH+9ZMAXclAVgBXJ7kaFV9tndRVW0HtgNMTU31/6UgSToJowR9L7A+yTrgf4Argat7F1TVuhe/T3IH8I/9Mddo1m69Z1Gf79FbrljU55O0cIYGvaqOJrmJzt0ry4AdVbUvyQ3d4143l6RTwChn6FTVLmBX376BIa+q3z/5sSRJx8t3ikpSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDXCoEtSIwy6JDVipKAn2ZTkQJKZJFsHHH9Xkge6/301yfnjH1WSNJ+hQU+yDLgVuAzYAFyVZEPfskeAX6mq84APANvHPagkaX6jnKFvBGaq6mBVHQHuAjb3Lqiqr1bV093NPcDq8Y4pSRpmlKCvAg71bM929x3LdcA/DTqQZEuS6STTc3Nzo08pSRpqlKBnwL4auDC5mE7Qbx50vKq2V9VUVU2tXLly9CklSUMtH2HNLLCmZ3s1cLh/UZLzgNuBy6rqW+MZT5I0qlHO0PcC65OsS3IacCWws3dBknOAzwC/V1XfGP+YkqRhhp6hV9XRJDcBu4FlwI6q2pfkhu7xbcCfAT8B3JYE4GhVTS3c2JKkfqNccqGqdgG7+vZt6/n+euD68Y4mSToevlNUkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEQZdkhph0CWpEcsnPYBeWtZuvWdRn+/RW65Y1OeTJskzdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqhEGXpEYYdElqxEhBT7IpyYEkM0m2DjieJB/pHn8gyZvHP6okaT5DP5wryTLgVuAdwCywN8nOqtrfs+wyYH33vwuBj3W/Si8pi/nhY37wmPqNcoa+EZipqoNVdQS4C9jct2YzcGd17AHOSvLaMc8qSZrHKB+fuwo41LM9y4+efQ9aswp4vHdRki3Alu7ms0kOHNe0J2cF8NTx/qJ8cAEmWRi+vgGWyOtr+bXBCb6+JWSxX9/rjnVglKBnwL46gTVU1XZg+wjPOXZJpqtqahLPvRh8fUtXy68NfH2LaZRLLrPAmp7t1cDhE1gjSVpAowR9L7A+ybokpwFXAjv71uwErune7fI24LtV9Xj/A0mSFs7QSy5VdTTJTcBuYBmwo6r2Jbmhe3wbsAu4HJgBngOuXbiRT9hELvUsIl/f0tXyawNf36JJ1Y9c6pYkLUG+U1SSGmHQJakRBl2SGmHQl6gkb0hySZIz+vZvmtRM45JkY5K3dr/fkOQPk1w+6bkWSpI7Jz3DQknyi93/f5dOepZxSHJhkld1v39Fkj9PcneSDyY5c+LzvdR+KJrk2qr6+KTnOBlJ/gC4EXgYuAB4b1V9rnvs61W1ZD8cLcn76Xw20HLgn+m8K/kLwK8Bu6vqLyY33clL0n/Lb4CLgX8BqKrfWPShxijJf1TVxu7376bz+/QfgEuBu6vqlknOd7KS7APO7979t53OXX2fBi7p7v+tic73Egz6Y1V1zqTnOBlJHgR+vqqeTbKWzm+ov62qDyf5z6r6uclOeOK6r+0C4OXAE8DqqnomySuAf6+q8yY538lK8nVgP3A7nXdTB/gknfd3UFVfnNx0J6/391+SvcDlVTWX5MeBPVX1s5Od8OQkebiq3tj9/odOnpLcV1UXTGw4Rnvr/5KT5IFjHQLOXsxZFsiyqnoWoKoeTXIR8Okkr2PwxzAsJUer6nnguST/VVXPAFTV95O8MOHZxmEKeC/wJ8D7quq+JN9f6iHv8bIkr6ZzOTdVNQdQVd9LcnSyo43FQz3/yr8/yVRVTSc5F/jBpIdrMuh0ov1O4Om+/QG+uvjjjN0TSS6oqvsAumfqvw7sAJb0GRBwJMnpVfUc8JYXd3avTy75oFfVC8BfJflU9+s3aevP4ZnAvXT+rFWSn6yqJ7o/61nqJxsA1wMfTvKndD6Q62tJDtH5cMLrJzoZjV5ySfI3wMer6t8GHPtEVV09gbHGJslqOmeyTww49vaq+soExhqLJC+vqv8dsH8F8NqqenACYy2YJFcAb6+qP570LAspyenA2VX1yKRnGYckrwR+ms5fxrNV9c0JjwQ0GnRJeinytkVJaoRBl6RGGHRJaoRBl6RGGHRJasT/ARtWrpeSNEDpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# redundancy of observations\n",
    "c = df.groupby(['h','k','l'])['phi'].count().value_counts().sort_index()\n",
    "c[c>1000].plot.bar();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a7e8226-2d36-4d81-ae54-7258d4ef05b3",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <b>Tip:</b> For more examples of using pandas to compute statistics, see the advanced python notebook (tutorial 6).\n",
    "</div >"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30a363f7-f4a4-428c-9d00-1c7db086f26d",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "In order to make a reciprocal space map, we need to place all of the observations on the same scale and merge them. That will be the topic of the next tutorial in this series.\n",
    "\n",
    "However, so far we've only processed 6 degrees of data (out of 50 total). As you can see from the bar plot above, most observations are unique. So there will not be enough redundant data to refine the scaling model. And the merged dataset will also be incomplete.\n",
    "\n",
    "To proceed, you have 2 options (choose depending on your interest and time available):\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b>Option 1:</b> Re-run the tutorial notebooks 3-4 using the full dataset (refined.expt). This will take approximately 30 minutes to run. You can skip ahead and do tutorial 6 while you wait.\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "    <b>Option 2:</b> Proceed to tutorial 5 using the corrected.nxs that was included in the GitHub release (this was pre-processed with the full 50 degrees of data).\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc8b59a5-b59b-461d-96de-dd714bbb7f13",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}