1_dials_indexing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c03b8759-378b-432e-ac6c-043e782607fe",
   "metadata": {},
   "source": [
    "# Indexing Bragg diffraction data with *DIALS*\n",
    "\n",
    "## Introduction\n",
    "\n",
    "*DIALS* is a program for integrating and scaling Bragg data. It is open source and can be used for both macromolecular and small molecule crystallography. You can read more about it here: https://dials.github.io\n",
    "\n",
    "For this tutorial, we will use *DIALS* for indexing and geometry refinement. Normally this would be followed by integration and scaling, but we will skip those steps to save time.\n",
    "\n",
    "For a more complete *DIALS* tutorial, see [Processing in Detail](https://dials.github.io/documentation/tutorials/processing_in_detail_betalactamase.html).\n",
    "\n",
    "### Tutorial data\n",
    "\n",
    "Download `insulin_2_1.tar` from Zenodo [doi: 10.5281/zenodo.6536805](https://dx.doi.org/10.5281/zenodo.6536805).\n",
    "\n",
    "The tar archive will expand to a directory called `images` with subfolders `insulin_2_1`, `insulin_2_bkg`, and `metrology`.\n",
    "\n",
    "Place the `images` folder in the same directory as this notebook.\n",
    "\n",
    "### Command-line programs in jupyter\n",
    "\n",
    "Normally you would run *DIALS* in the command line. We will run command line programs from within the notebook by prefixing them with an exclamation point.\n",
    "\n",
    "To run the command, click in the cell and press shift-Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b1964e8c-3a50-48f8-8994-e657d290d7e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS 3.dev\n",
      "Python 3.9.12\n",
      "Installed in: /Users/steve/Desktop/erice/lib/python3.9/site-packages/dials\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.version"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8070b6a-c03a-4d91-8986-b9837fab9d1e",
   "metadata": {},
   "source": [
    "## Import the metadata\n",
    "\n",
    "The metadata (such as X-ray energy, oscillation width, exposure time, etc) are stored in the cbf image headers. The *dials.integrate* command reads this in and produces a *.expt* file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b8e65675-cc6a-421b-9b36-590a440d8fd8",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "DIALS 3.dev\n",
      "The following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  experiments = <image files>\n",
      "}\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "  format: <class 'dxtbx.format.FormatCBFMiniPilatusCHESS_6MSN127.FormatCBFMiniPilatusCHESS_6MSN127'>\n",
      "  num images: 500\n",
      "  sequences:\n",
      "    still:    0\n",
      "    sweep:    1\n",
      "  num stills: 0\n",
      "--------------------------------------------------------------------------------\n",
      "Writing experiments to imported.expt\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.import images/insulin_2_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03b23de0-2f81-410e-bdea-d7b3a2968e69",
   "metadata": {},
   "source": [
    "The output file *imported.expt* is just a text file (json format). You can open it by double-clicking in the file browser.\n",
    "\n",
    "*DIALS* also has a method for summarizing expt files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7e27ec7b-d733-4bad-bbd2-78d195e44c4a",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "The following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  experiments = imported.expt\n",
      "}\n",
      "\n",
      "Experiment 0:\n",
      "Experiment identifier: cc2ca819-f182-8ed0-a4f8-6b7b2905b7fa\n",
      "Image template: /Users/steve/Documents/GitHub/ando-lab/erice-2022-data-reduction/images/insulin_2_1/insulin_2_1_####.cbf.bz2\n",
      "Detector:\n",
      "Panel:\n",
      "  name: Panel\n",
      "  type: SENSOR_PAD\n",
      "  identifier: \n",
      "  pixel_size:{0.172,0.172}\n",
      "  image_size: {2463,2527}\n",
      "  trusted_range: {-1,1.0098e+06}\n",
      "  thickness: 1\n",
      "  material: Si\n",
      "  mu: 3.92851\n",
      "  gain: 1\n",
      "  pedestal: 0\n",
      "  fast_axis: {1,0,0}\n",
      "  slow_axis: {0,-1,0}\n",
      "  origin: {-217.491,213.713,-260.05}\n",
      "  distance: 260.05\n",
      "  pixel to millimeter strategy: ParallaxCorrectedPxMmStrategy\n",
      "    mu: 3.92851\n",
      "    t0: 1\n",
      "\n",
      "\n",
      "Max resolution (at corners): 1.155825\n",
      "Max resolution (inscribed):  1.485702\n",
      "\n",
      "Beam:\n",
      "    wavelength: 0.9768\n",
      "    sample to source direction : {0,0,1}\n",
      "    divergence: 0\n",
      "    sigma divergence: 0\n",
      "    polarization normal: {0,1,0}\n",
      "    polarization fraction: 0.999\n",
      "    flux: 0\n",
      "    transmission: 1\n",
      "\n",
      "Beam centre: \n",
      "    mm: (217.49,213.71)\n",
      "    px: (1264.48,1242.52)\n",
      "\n",
      "Scan:\n",
      "    number of images:   500\n",
      "    image range:   {1,500}\n",
      "    oscillation:   {160,0.1}\n",
      "    exposure time: 0.1\n",
      "\n",
      "Goniometer:\n",
      "    Rotation axis:   {-1,0,0}\n",
      "    Fixed rotation:  {1,0,0,0,1,0,0,0,1}\n",
      "    Setting rotation:{1,0,0,0,1,0,0,0,1}\n",
      "\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.show imported.expt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b871a4d1-1029-4636-b6dc-2717ebf6ae63",
   "metadata": {},
   "source": [
    "## DIALS image viewer\n",
    "\n",
    "We'll use the *DIALS* image viewer to inspect the raw data.\n",
    "\n",
    "Launch a new terminal window and type:\n",
    "\n",
    "```bash\n",
    "dials.image_viewer imported.expt\n",
    "```\n",
    "\n",
    "Adjust the brighness slider so that both the Bragg peaks and background scattering are visible.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question 1:</b> Do you see any diffuse scattering?\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question 2:</b> Scroll through some images. Is the crystal mosaicity high or low?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c29deda8-0a2a-49b3-8bb7-cbb86673c6ee",
   "metadata": {},
   "source": [
    "## Masking\n",
    "\n",
    "Some regions of the detector are unreliable. Gaps between panels are automatically masked using *DIALS*. However, we need to make a mask for the beamstop.\n",
    "\n",
    "This can be done manually in the *DIALS* image viewer. But to save time, we'll do it programatically on the command line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "44fa2f65-2425-427e-9ae1-d4282c501565",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The following parameters have been modified:\n",
      "\n",
      "untrusted {\n",
      "  circle = 1264 1242 50\n",
      "}\n",
      "input {\n",
      "  experiments = imported.expt\n",
      "}\n",
      "\n",
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "Writing mask to pixels.mask\n",
      "\u001b[0mThe following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  mask = \"pixels.mask\"\n",
      "  experiments = imported.expt\n",
      "}\n",
      "output {\n",
      "  experiments = \"imported.expt\"\n",
      "}\n",
      "\n",
      "Writing experiments to imported.expt\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.generate_mask imported.expt untrusted.circle=1264,1242,50\n",
    "!dials.apply_mask imported.expt input.mask=pixels.mask output.experiments=imported.expt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3da68739-abd8-4c99-ab1b-1b02475c64dd",
   "metadata": {},
   "source": [
    "Launch the dials image viewer again. Click the \"show mask\" checkbox. You should see the beamstop region masked out."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c334d9f0-6803-44cd-ac44-6d878dc8c57d",
   "metadata": {},
   "source": [
    "## Spot finding\n",
    "\n",
    "Next, *DIALS* will read all of the images and locate Bragg peaks. This can take a while (be patient)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9d9cf4ad-8489-4dac-a0fa-c8f25e8f5f1e",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "DIALS 3.dev\n",
      "The following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  experiments = imported.expt\n",
      "}\n",
      "\n",
      "Setting spotfinder.filter.min_spot_size=3\n",
      "Configuring spot finder from input parameters\n",
      "--------------------------------------------------------------------------------\n",
      "Finding strong spots in imageset 0\n",
      "--------------------------------------------------------------------------------\n",
      "\n",
      "Finding spots in image 1 to 500...\n",
      "Setting nproc=6\n",
      "Setting chunksize=83\n",
      "Extracting strong pixels from images\n",
      " Using multiprocessing with 6 parallel job(s)\n",
      "\n",
      "Found 1600 strong pixels on image 1\n",
      "Found 1569 strong pixels on image 2\n",
      "Found 1304 strong pixels on image 3\n",
      "Found 1484 strong pixels on image 4\n",
      "Found 1568 strong pixels on image 5\n",
      "Found 1653 strong pixels on image 6\n",
      "Found 1766 strong pixels on image 7\n",
      "Found 1719 strong pixels on image 8\n",
      "Found 1966 strong pixels on image 9\n",
      "Found 1949 strong pixels on image 10\n",
      "Found 1729 strong pixels on image 11\n",
      "Found 1793 strong pixels on image 12\n",
      "Found 1471 strong pixels on image 13\n",
      "Found 1612 strong pixels on image 14\n",
      "Found 1589 strong pixels on image 15\n",
      "Found 1721 strong pixels on image 16\n",
      "Found 1745 strong pixels on image 17\n",
      "Found 1789 strong pixels on image 18\n",
      "Found 1835 strong pixels on image 19\n",
      "Found 1697 strong pixels on image 20\n",
      "Found 1549 strong pixels on image 21\n",
      "Found 1504 strong pixels on image 22\n",
      "Found 1590 strong pixels on image 23\n",
      "Found 1726 strong pixels on image 24\n",
      "Found 1801 strong pixels on image 25\n",
      "Found 1732 strong pixels on image 26\n",
      "Found 1633 strong pixels on image 27\n",
      "Found 1683 strong pixels on image 28\n",
      "Found 1593 strong pixels on image 29\n",
      "Found 1337 strong pixels on image 30\n",
      "Found 1604 strong pixels on image 31\n",
      "Found 1671 strong pixels on image 32\n",
      "Found 1423 strong pixels on image 33\n",
      "Found 1461 strong pixels on image 34\n",
      "Found 1684 strong pixels on image 35\n",
      "Found 1629 strong pixels on image 36\n",
      "Found 1767 strong pixels on image 37\n",
      "Found 1720 strong pixels on image 38\n",
      "Found 1546 strong pixels on image 39\n",
      "Found 1497 strong pixels on image 40\n",
      "Found 1745 strong pixels on image 41\n",
      "Found 1647 strong pixels on image 42\n",
      "Found 1652 strong pixels on image 43\n",
      "Found 1470 strong pixels on image 44\n",
      "Found 1741 strong pixels on image 45\n",
      "Found 1611 strong pixels on image 46\n",
      "Found 1651 strong pixels on image 47\n",
      "Found 1712 strong pixels on image 48\n",
      "Found 1736 strong pixels on image 49\n",
      "Found 1606 strong pixels on image 50\n",
      "Found 1786 strong pixels on image 51\n",
      "Found 1678 strong pixels on image 52\n",
      "Found 1700 strong pixels on image 53\n",
      "Found 1684 strong pixels on image 54\n",
      "Found 1534 strong pixels on image 55\n",
      "Found 1724 strong pixels on image 56\n",
      "Found 1666 strong pixels on image 57\n",
      "Found 1595 strong pixels on image 58\n",
      "Found 2040 strong pixels on image 59\n",
      "Found 1841 strong pixels on image 60\n",
      "Found 1698 strong pixels on image 61\n",
      "Found 1620 strong pixels on image 62\n",
      "Found 1743 strong pixels on image 63\n",
      "Found 1571 strong pixels on image 64\n",
      "Found 1863 strong pixels on image 65\n",
      "Found 1518 strong pixels on image 66\n",
      "Found 1492 strong pixels on image 67\n",
      "Found 1469 strong pixels on image 68\n",
      "Found 1695 strong pixels on image 69\n",
      "Found 1570 strong pixels on image 70\n",
      "Found 1776 strong pixels on image 71\n",
      "Found 1833 strong pixels on image 72\n",
      "Found 1666 strong pixels on image 73\n",
      "Found 1743 strong pixels on image 74\n",
      "Found 1611 strong pixels on image 75\n",
      "Found 1253 strong pixels on image 76\n",
      "Found 1463 strong pixels on image 77\n",
      "Found 1371 strong pixels on image 78\n",
      "Found 1590 strong pixels on image 79\n",
      "Found 1594 strong pixels on image 80\n",
      "Found 1508 strong pixels on image 81\n",
      "Found 1444 strong pixels on image 82\n",
      "Found 1502 strong pixels on image 83\n",
      "Found 1480 strong pixels on image 84\n",
      "Found 1479 strong pixels on image 85\n",
      "Found 1516 strong pixels on image 86\n",
      "Found 1222 strong pixels on image 87\n",
      "Found 1490 strong pixels on image 88\n",
      "Found 1432 strong pixels on image 89\n",
      "Found 1412 strong pixels on image 90\n",
      "Found 1442 strong pixels on image 91\n",
      "Found 1467 strong pixels on image 92\n",
      "Found 1357 strong pixels on image 93\n",
      "Found 1378 strong pixels on image 94\n",
      "Found 1446 strong pixels on image 95\n",
      "Found 1541 strong pixels on image 96\n",
      "Found 1456 strong pixels on image 97\n",
      "Found 1336 strong pixels on image 98\n",
      "Found 1372 strong pixels on image 99\n",
      "Found 1369 strong pixels on image 100\n",
      "Found 1512 strong pixels on image 101\n",
      "Found 1529 strong pixels on image 102\n",
      "Found 1665 strong pixels on image 103\n",
      "Found 1617 strong pixels on image 104\n",
      "Found 1649 strong pixels on image 105\n",
      "Found 1519 strong pixels on image 106\n",
      "Found 1521 strong pixels on image 107\n",
      "Found 1415 strong pixels on image 108\n",
      "Found 1445 strong pixels on image 109\n",
      "Found 1609 strong pixels on image 110\n",
      "Found 1290 strong pixels on image 111\n",
      "Found 1374 strong pixels on image 112\n",
      "Found 1460 strong pixels on image 113\n",
      "Found 1593 strong pixels on image 114\n",
      "Found 1539 strong pixels on image 115\n",
      "Found 1437 strong pixels on image 116\n",
      "Found 1415 strong pixels on image 117\n",
      "Found 1311 strong pixels on image 118\n",
      "Found 1388 strong pixels on image 119\n",
      "Found 1437 strong pixels on image 120\n",
      "Found 1281 strong pixels on image 121\n",
      "Found 1303 strong pixels on image 122\n",
      "Found 1422 strong pixels on image 123\n",
      "Found 1645 strong pixels on image 124\n",
      "Found 1427 strong pixels on image 125\n",
      "Found 1330 strong pixels on image 126\n",
      "Found 1233 strong pixels on image 127\n",
      "Found 1290 strong pixels on image 128\n",
      "Found 1572 strong pixels on image 129\n",
      "Found 1337 strong pixels on image 130\n",
      "Found 1484 strong pixels on image 131\n",
      "Found 1462 strong pixels on image 132\n",
      "Found 1577 strong pixels on image 133\n",
      "Found 1592 strong pixels on image 134\n",
      "Found 1274 strong pixels on image 135\n",
      "Found 1279 strong pixels on image 136\n",
      "Found 1415 strong pixels on image 137\n",
      "Found 1426 strong pixels on image 138\n",
      "Found 1357 strong pixels on image 139\n",
      "Found 1459 strong pixels on image 140\n",
      "Found 1528 strong pixels on image 141\n",
      "Found 1439 strong pixels on image 142\n",
      "Found 1256 strong pixels on image 143\n",
      "Found 1331 strong pixels on image 144\n",
      "Found 1148 strong pixels on image 145\n",
      "Found 1192 strong pixels on image 146\n",
      "Found 1255 strong pixels on image 147\n",
      "Found 1234 strong pixels on image 148\n",
      "Found 1301 strong pixels on image 149\n",
      "Found 1599 strong pixels on image 150\n",
      "Found 1373 strong pixels on image 151\n",
      "Found 1303 strong pixels on image 152\n",
      "Found 1284 strong pixels on image 153\n",
      "Found 1366 strong pixels on image 154\n",
      "Found 1369 strong pixels on image 155\n",
      "Found 1407 strong pixels on image 156\n",
      "Found 1456 strong pixels on image 157\n",
      "Found 1524 strong pixels on image 158\n",
      "Found 1455 strong pixels on image 159\n",
      "Found 1574 strong pixels on image 160\n",
      "Found 1265 strong pixels on image 161\n",
      "Found 1287 strong pixels on image 162\n",
      "Found 1235 strong pixels on image 163\n",
      "Found 1363 strong pixels on image 164\n",
      "Found 1314 strong pixels on image 165\n",
      "Found 1267 strong pixels on image 166\n",
      "Found 1319 strong pixels on image 167\n",
      "Found 1282 strong pixels on image 168\n",
      "Found 1234 strong pixels on image 169\n",
      "Found 1366 strong pixels on image 170\n",
      "Found 1326 strong pixels on image 171\n",
      "Found 1234 strong pixels on image 172\n",
      "Found 1359 strong pixels on image 173\n",
      "Found 1443 strong pixels on image 174\n",
      "Found 1645 strong pixels on image 175\n",
      "Found 1368 strong pixels on image 176\n",
      "Found 1260 strong pixels on image 177\n",
      "Found 1063 strong pixels on image 178\n",
      "Found 1298 strong pixels on image 179\n",
      "Found 1380 strong pixels on image 180\n",
      "Found 1319 strong pixels on image 181\n",
      "Found 1333 strong pixels on image 182\n",
      "Found 1224 strong pixels on image 183\n",
      "Found 1385 strong pixels on image 184\n",
      "Found 1162 strong pixels on image 185\n",
      "Found 1205 strong pixels on image 186\n",
      "Found 1480 strong pixels on image 187\n",
      "Found 1517 strong pixels on image 188\n",
      "Found 1294 strong pixels on image 189\n",
      "Found 1214 strong pixels on image 190\n",
      "Found 1237 strong pixels on image 191\n",
      "Found 1217 strong pixels on image 192\n",
      "Found 1224 strong pixels on image 193\n",
      "Found 1282 strong pixels on image 194\n",
      "Found 1341 strong pixels on image 195\n",
      "Found 1178 strong pixels on image 196\n",
      "Found 1247 strong pixels on image 197\n",
      "Found 1320 strong pixels on image 198\n",
      "Found 1247 strong pixels on image 199\n",
      "Found 1212 strong pixels on image 200\n",
      "Found 1172 strong pixels on image 201\n",
      "Found 1301 strong pixels on image 202\n",
      "Found 1227 strong pixels on image 203\n",
      "Found 1265 strong pixels on image 204\n",
      "Found 1110 strong pixels on image 205\n",
      "Found 1222 strong pixels on image 206\n",
      "Found 1414 strong pixels on image 207\n",
      "Found 1288 strong pixels on image 208\n",
      "Found 1332 strong pixels on image 209\n",
      "Found 1316 strong pixels on image 210\n",
      "Found 1356 strong pixels on image 211\n",
      "Found 1522 strong pixels on image 212\n",
      "Found 1345 strong pixels on image 213\n",
      "Found 1276 strong pixels on image 214\n",
      "Found 1549 strong pixels on image 215\n",
      "Found 1425 strong pixels on image 216\n",
      "Found 1434 strong pixels on image 217\n",
      "Found 1392 strong pixels on image 218\n",
      "Found 1242 strong pixels on image 219\n",
      "Found 1286 strong pixels on image 220\n",
      "Found 1317 strong pixels on image 221\n",
      "Found 1304 strong pixels on image 222\n",
      "Found 1256 strong pixels on image 223\n",
      "Found 1275 strong pixels on image 224\n",
      "Found 1121 strong pixels on image 225\n",
      "Found 1144 strong pixels on image 226\n",
      "Found 1203 strong pixels on image 227\n",
      "Found 1196 strong pixels on image 228\n",
      "Found 1224 strong pixels on image 229\n",
      "Found 1066 strong pixels on image 230\n",
      "Found 1148 strong pixels on image 231\n",
      "Found 1092 strong pixels on image 232\n",
      "Found 1028 strong pixels on image 233\n",
      "Found 1099 strong pixels on image 234\n",
      "Found 1265 strong pixels on image 235\n",
      "Found 1270 strong pixels on image 236\n",
      "Found 1279 strong pixels on image 237\n",
      "Found 1247 strong pixels on image 238\n",
      "Found 1205 strong pixels on image 239\n",
      "Found 1135 strong pixels on image 240\n",
      "Found 1134 strong pixels on image 241\n",
      "Found 1119 strong pixels on image 242\n",
      "Found 1170 strong pixels on image 243\n",
      "Found 1131 strong pixels on image 244\n",
      "Found 1225 strong pixels on image 245\n",
      "Found 1136 strong pixels on image 246\n",
      "Found 1253 strong pixels on image 247\n",
      "Found 1168 strong pixels on image 248\n",
      "Found 1302 strong pixels on image 249\n",
      "Found 1260 strong pixels on image 250\n",
      "Found 1325 strong pixels on image 251\n",
      "Found 1342 strong pixels on image 252\n",
      "Found 1171 strong pixels on image 253\n",
      "Found 1265 strong pixels on image 254\n",
      "Found 1197 strong pixels on image 255\n",
      "Found 1224 strong pixels on image 256\n",
      "Found 1102 strong pixels on image 257\n",
      "Found 1258 strong pixels on image 258\n",
      "Found 1165 strong pixels on image 259\n",
      "Found 1154 strong pixels on image 260\n",
      "Found 1089 strong pixels on image 261\n",
      "Found 1162 strong pixels on image 262\n",
      "Found 1308 strong pixels on image 263\n",
      "Found 1371 strong pixels on image 264\n",
      "Found 1290 strong pixels on image 265\n",
      "Found 1268 strong pixels on image 266\n",
      "Found 1293 strong pixels on image 267\n",
      "Found 1272 strong pixels on image 268\n",
      "Found 1353 strong pixels on image 269\n",
      "Found 1293 strong pixels on image 270\n",
      "Found 1202 strong pixels on image 271\n",
      "Found 1410 strong pixels on image 272\n",
      "Found 1333 strong pixels on image 273\n",
      "Found 1286 strong pixels on image 274\n",
      "Found 1343 strong pixels on image 275\n",
      "Found 1165 strong pixels on image 276\n",
      "Found 1142 strong pixels on image 277\n",
      "Found 1158 strong pixels on image 278\n",
      "Found 1190 strong pixels on image 279\n",
      "Found 1199 strong pixels on image 280\n",
      "Found 1345 strong pixels on image 281\n",
      "Found 1260 strong pixels on image 282\n",
      "Found 1196 strong pixels on image 283\n",
      "Found 1127 strong pixels on image 284\n",
      "Found 1155 strong pixels on image 285\n",
      "Found 1156 strong pixels on image 286\n",
      "Found 1036 strong pixels on image 287\n",
      "Found 1100 strong pixels on image 288\n",
      "Found 1186 strong pixels on image 289\n",
      "Found 1204 strong pixels on image 290\n",
      "Found 1329 strong pixels on image 291\n",
      "Found 1322 strong pixels on image 292\n",
      "Found 1195 strong pixels on image 293\n",
      "Found 1180 strong pixels on image 294\n",
      "Found 1250 strong pixels on image 295\n",
      "Found 1103 strong pixels on image 296\n",
      "Found 1056 strong pixels on image 297\n",
      "Found 1207 strong pixels on image 298\n",
      "Found 1219 strong pixels on image 299\n",
      "Found 1232 strong pixels on image 300\n",
      "Found 1146 strong pixels on image 301\n",
      "Found 1175 strong pixels on image 302\n",
      "Found 1273 strong pixels on image 303\n",
      "Found 1210 strong pixels on image 304\n",
      "Found 1198 strong pixels on image 305\n",
      "Found 1057 strong pixels on image 306\n",
      "Found 1089 strong pixels on image 307\n",
      "Found 1088 strong pixels on image 308\n",
      "Found 1060 strong pixels on image 309\n",
      "Found 1232 strong pixels on image 310\n",
      "Found 1098 strong pixels on image 311\n",
      "Found 1146 strong pixels on image 312\n",
      "Found 1107 strong pixels on image 313\n",
      "Found 1045 strong pixels on image 314\n",
      "Found 1129 strong pixels on image 315\n",
      "Found 1112 strong pixels on image 316\n",
      "Found 1026 strong pixels on image 317\n",
      "Found 1263 strong pixels on image 318\n",
      "Found 1164 strong pixels on image 319\n",
      "Found 1289 strong pixels on image 320\n",
      "Found 1295 strong pixels on image 321\n",
      "Found 1144 strong pixels on image 322\n",
      "Found 1111 strong pixels on image 323\n",
      "Found 1032 strong pixels on image 324\n",
      "Found 1173 strong pixels on image 325\n",
      "Found 1168 strong pixels on image 326\n",
      "Found 1091 strong pixels on image 327\n",
      "Found 945 strong pixels on image 328\n",
      "Found 1104 strong pixels on image 329\n",
      "Found 1291 strong pixels on image 330\n",
      "Found 946 strong pixels on image 331\n",
      "Found 1083 strong pixels on image 332\n",
      "Found 1158 strong pixels on image 333\n",
      "Found 1082 strong pixels on image 334\n",
      "Found 1122 strong pixels on image 335\n",
      "Found 1171 strong pixels on image 336\n",
      "Found 1485 strong pixels on image 337\n",
      "Found 1081 strong pixels on image 338\n",
      "Found 944 strong pixels on image 339\n",
      "Found 968 strong pixels on image 340\n",
      "Found 1054 strong pixels on image 341\n",
      "Found 1083 strong pixels on image 342\n",
      "Found 986 strong pixels on image 343\n",
      "Found 891 strong pixels on image 344\n",
      "Found 1124 strong pixels on image 345\n",
      "Found 1039 strong pixels on image 346\n",
      "Found 1155 strong pixels on image 347\n",
      "Found 1067 strong pixels on image 348\n",
      "Found 1185 strong pixels on image 349\n",
      "Found 1121 strong pixels on image 350\n",
      "Found 1153 strong pixels on image 351\n",
      "Found 1221 strong pixels on image 352\n",
      "Found 1112 strong pixels on image 353\n",
      "Found 1099 strong pixels on image 354\n",
      "Found 1052 strong pixels on image 355\n",
      "Found 1325 strong pixels on image 356\n",
      "Found 1059 strong pixels on image 357\n",
      "Found 997 strong pixels on image 358\n",
      "Found 1155 strong pixels on image 359\n",
      "Found 1152 strong pixels on image 360\n",
      "Found 1306 strong pixels on image 361\n",
      "Found 1112 strong pixels on image 362\n",
      "Found 1117 strong pixels on image 363\n",
      "Found 1335 strong pixels on image 364\n",
      "Found 1289 strong pixels on image 365\n",
      "Found 1316 strong pixels on image 366\n",
      "Found 1300 strong pixels on image 367\n",
      "Found 1233 strong pixels on image 368\n",
      "Found 1200 strong pixels on image 369\n",
      "Found 1166 strong pixels on image 370\n",
      "Found 1172 strong pixels on image 371\n",
      "Found 1166 strong pixels on image 372\n",
      "Found 1303 strong pixels on image 373\n",
      "Found 1324 strong pixels on image 374\n",
      "Found 1082 strong pixels on image 375\n",
      "Found 1256 strong pixels on image 376\n",
      "Found 1275 strong pixels on image 377\n",
      "Found 1328 strong pixels on image 378\n",
      "Found 1197 strong pixels on image 379\n",
      "Found 1169 strong pixels on image 380\n",
      "Found 1130 strong pixels on image 381\n",
      "Found 1134 strong pixels on image 382\n",
      "Found 1132 strong pixels on image 383\n",
      "Found 1058 strong pixels on image 384\n",
      "Found 1078 strong pixels on image 385\n",
      "Found 1097 strong pixels on image 386\n",
      "Found 930 strong pixels on image 387\n",
      "Found 986 strong pixels on image 388\n",
      "Found 1036 strong pixels on image 389\n",
      "Found 1122 strong pixels on image 390\n",
      "Found 1197 strong pixels on image 391\n",
      "Found 1203 strong pixels on image 392\n",
      "Found 1407 strong pixels on image 393\n",
      "Found 1237 strong pixels on image 394\n",
      "Found 1123 strong pixels on image 395\n",
      "Found 1277 strong pixels on image 396\n",
      "Found 1139 strong pixels on image 397\n",
      "Found 1182 strong pixels on image 398\n",
      "Found 1116 strong pixels on image 399\n",
      "Found 1265 strong pixels on image 400\n",
      "Found 1095 strong pixels on image 401\n",
      "Found 1024 strong pixels on image 402\n",
      "Found 1073 strong pixels on image 403\n",
      "Found 1149 strong pixels on image 404\n",
      "Found 1197 strong pixels on image 405\n",
      "Found 1203 strong pixels on image 406\n",
      "Found 1089 strong pixels on image 407\n",
      "Found 1157 strong pixels on image 408\n",
      "Found 1164 strong pixels on image 409\n",
      "Found 1165 strong pixels on image 410\n",
      "Found 1115 strong pixels on image 411\n",
      "Found 1043 strong pixels on image 412\n",
      "Found 1189 strong pixels on image 413\n",
      "Found 1345 strong pixels on image 414\n",
      "Found 1176 strong pixels on image 415\n",
      "Found 1174 strong pixels on image 416\n",
      "Found 1184 strong pixels on image 417\n",
      "Found 1167 strong pixels on image 418\n",
      "Found 1205 strong pixels on image 419\n",
      "Found 890 strong pixels on image 420\n",
      "Found 1007 strong pixels on image 421\n",
      "Found 1035 strong pixels on image 422\n",
      "Found 1048 strong pixels on image 423\n",
      "Found 1191 strong pixels on image 424\n",
      "Found 1210 strong pixels on image 425\n",
      "Found 1143 strong pixels on image 426\n",
      "Found 1024 strong pixels on image 427\n",
      "Found 1084 strong pixels on image 428\n",
      "Found 1182 strong pixels on image 429\n",
      "Found 1301 strong pixels on image 430\n",
      "Found 1301 strong pixels on image 431\n",
      "Found 1193 strong pixels on image 432\n",
      "Found 1290 strong pixels on image 433\n",
      "Found 1218 strong pixels on image 434\n",
      "Found 1261 strong pixels on image 435\n",
      "Found 1294 strong pixels on image 436\n",
      "Found 1261 strong pixels on image 437\n",
      "Found 1354 strong pixels on image 438\n",
      "Found 1193 strong pixels on image 439\n",
      "Found 1296 strong pixels on image 440\n",
      "Found 1150 strong pixels on image 441\n",
      "Found 1209 strong pixels on image 442\n",
      "Found 1118 strong pixels on image 443\n",
      "Found 1049 strong pixels on image 444\n",
      "Found 1075 strong pixels on image 445\n",
      "Found 1044 strong pixels on image 446\n",
      "Found 1174 strong pixels on image 447\n",
      "Found 1123 strong pixels on image 448\n",
      "Found 1142 strong pixels on image 449\n",
      "Found 1181 strong pixels on image 450\n",
      "Found 1477 strong pixels on image 451\n",
      "Found 1298 strong pixels on image 452\n",
      "Found 1170 strong pixels on image 453\n",
      "Found 1010 strong pixels on image 454\n",
      "Found 960 strong pixels on image 455\n",
      "Found 1172 strong pixels on image 456\n",
      "Found 1327 strong pixels on image 457\n",
      "Found 1215 strong pixels on image 458\n",
      "Found 1286 strong pixels on image 459\n",
      "Found 1233 strong pixels on image 460\n",
      "Found 1310 strong pixels on image 461\n",
      "Found 1140 strong pixels on image 462\n",
      "Found 1117 strong pixels on image 463\n",
      "Found 1194 strong pixels on image 464\n",
      "Found 1014 strong pixels on image 465\n",
      "Found 1029 strong pixels on image 466\n",
      "Found 1183 strong pixels on image 467\n",
      "Found 1185 strong pixels on image 468\n",
      "Found 1232 strong pixels on image 469\n",
      "Found 1267 strong pixels on image 470\n",
      "Found 1274 strong pixels on image 471\n",
      "Found 1145 strong pixels on image 472\n",
      "Found 1216 strong pixels on image 473\n",
      "Found 1185 strong pixels on image 474\n",
      "Found 1213 strong pixels on image 475\n",
      "Found 1239 strong pixels on image 476\n",
      "Found 1230 strong pixels on image 477\n",
      "Found 1187 strong pixels on image 478\n",
      "Found 1264 strong pixels on image 479\n",
      "Found 1187 strong pixels on image 480\n",
      "Found 1247 strong pixels on image 481\n",
      "Found 1128 strong pixels on image 482\n",
      "Found 1107 strong pixels on image 483\n",
      "Found 1078 strong pixels on image 484\n",
      "Found 1257 strong pixels on image 485\n",
      "Found 1261 strong pixels on image 486\n",
      "Found 1162 strong pixels on image 487\n",
      "Found 1210 strong pixels on image 488\n",
      "Found 1254 strong pixels on image 489\n",
      "Found 1247 strong pixels on image 490\n",
      "Found 1234 strong pixels on image 491\n",
      "Found 1318 strong pixels on image 492\n",
      "Found 1361 strong pixels on image 493\n",
      "Found 1220 strong pixels on image 494\n",
      "Found 1105 strong pixels on image 495\n",
      "Found 1136 strong pixels on image 496\n",
      "Found 1176 strong pixels on image 497\n",
      "Found 1146 strong pixels on image 498\n",
      "Found 1122 strong pixels on image 499\n",
      "Found 1518 strong pixels on image 500\n",
      "\n",
      "Extracted 60928 spots\n",
      "Removed 9801 spots with size < 3 pixels\n",
      "Removed 1 spots with size > 1000 pixels\n",
      "Calculated 51126 spot centroids\n",
      "Calculated 51126 spot intensities\n",
      "Filtered 51111 of 51126 spots by peak-centroid distance\n",
      "\n",
      "Histogram of per-image spot count for imageset 0:\n",
      "51111 spots found on 500 images (max 1117 / bin)\n",
      "*                                                           \n",
      "********* ** *                                              \n",
      "***************** *** ** *      *                          *\n",
      "***************************************************** ******\n",
      "************************************************************\n",
      "************************************************************\n",
      "************************************************************\n",
      "************************************************************\n",
      "************************************************************\n",
      "************************************************************\n",
      "1                         image                          500\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "Saved 51111 reflections to strong.refl\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.find_spots imported.expt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "739ed3ec-e38d-40cc-b57d-705aef429a9e",
   "metadata": {},
   "source": [
    "## Indexing\n",
    "\n",
    "Next we need to assign Miller indices to the spots and to refine the diffraction geometry.\n",
    "\n",
    "We'll skip the usual space group determination steps by telling *DIALS* the space group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "19b0b71a-8e89-4b19-b608-41aa7b023e03",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "DIALS 3.dev\n",
      "The following parameters have been modified:\n",
      "\n",
      "indexing {\n",
      "  known_symmetry {\n",
      "    space_group = \"I 21 3\"\n",
      "  }\n",
      "}\n",
      "input {\n",
      "  experiments = imported.expt\n",
      "  reflections = strong.refl\n",
      "}\n",
      "\n",
      "Found max_cell: 74.2 Angstrom\n",
      "Setting d_min: 1.45\n",
      "FFT gridding: (256,256,256)\n",
      "Number of centroids used: 50261\n",
      "Candidate solutions:\n",
      "+-------------------------------------+----------+----------------+------------+-------------+-------------------+-----------+-----------------+-----------------+\n",
      "| unit_cell                           |   volume |   volume score |   #indexed |   % indexed |   % indexed score |   rmsd_xy |   rmsd_xy score |   overall score |\n",
      "|-------------------------------------+----------+----------------+------------+-------------+-------------------+-----------+-----------------+-----------------|\n",
      "| 68.61 68.61 68.61 109.5 109.5 109.5 |   248622 |           0.02 |      48839 |          97 |              0.04 |      0.07 |            0.06 |            0.13 |\n",
      "| 68.53 68.53 68.53 109.5 109.5 109.5 |   247715 |           0.02 |      50163 |          99 |              0    |      0.07 |            0.11 |            0.13 |\n",
      "| 68.59 68.59 68.59 109.5 109.5 109.5 |   248459 |           0.02 |      50077 |          99 |              0    |      0.07 |            0.12 |            0.15 |\n",
      "| 68.66 68.66 68.66 109.5 109.5 109.5 |   249178 |           0.03 |      48856 |          97 |              0.04 |      0.07 |            0.11 |            0.17 |\n",
      "| 68.53 68.53 68.53 109.5 109.5 109.5 |   247795 |           0.02 |      47005 |          93 |              0.09 |      0.07 |            0.08 |            0.19 |\n",
      "| 68.65 68.65 68.65 109.5 109.5 109.5 |   249035 |           0.03 |      48187 |          95 |              0.06 |      0.07 |            0.11 |            0.2  |\n",
      "| 68.68 68.68 68.68 109.5 109.5 109.5 |   249368 |           0.03 |      47725 |          94 |              0.07 |      0.07 |            0.1  |            0.2  |\n",
      "| 68.58 68.58 68.58 109.5 109.5 109.5 |   248279 |           0.02 |      49206 |          97 |              0.03 |      0.08 |            0.16 |            0.21 |\n",
      "| 68.56 68.56 68.56 109.5 109.5 109.5 |   248097 |           0.02 |      50150 |          99 |              0    |      0.08 |            0.2  |            0.22 |\n",
      "| 68.65 68.65 68.65 109.5 109.5 109.5 |   249096 |           0.03 |      46365 |          92 |              0.11 |      0.07 |            0.1  |            0.23 |\n",
      "| 68.41 68.41 68.41 109.5 109.5 109.5 |   246508 |           0.01 |      46887 |          93 |              0.1  |      0.08 |            0.15 |            0.25 |\n",
      "| 68.39 68.39 68.39 109.5 109.5 109.5 |   246239 |           0.01 |      46524 |          92 |              0.11 |      0.08 |            0.15 |            0.26 |\n",
      "| 68.24 68.24 68.24 109.5 109.5 109.5 |   244605 |           0    |      45415 |          90 |              0.14 |      0.07 |            0.13 |            0.28 |\n",
      "| 68.65 68.65 68.65 109.5 109.5 109.5 |   249014 |           0.03 |      48459 |          96 |              0.05 |      0.08 |            0.2  |            0.28 |\n",
      "| 68.38 68.38 68.38 109.5 109.5 109.5 |   246095 |           0.01 |      44614 |          88 |              0.17 |      0.07 |            0.14 |            0.32 |\n",
      "| 68.36 68.36 68.36 109.5 109.5 109.5 |   245907 |           0.01 |      42678 |          84 |              0.23 |      0.07 |            0.09 |            0.33 |\n",
      "| 68.41 68.41 68.41 109.5 109.5 109.5 |   246487 |           0.01 |      41210 |          81 |              0.28 |      0.07 |            0.05 |            0.34 |\n",
      "| 68.47 68.47 68.47 109.5 109.5 109.5 |   247135 |           0.01 |      42832 |          85 |              0.23 |      0.07 |            0.11 |            0.35 |\n",
      "| 68.51 68.51 68.51 109.5 109.5 109.5 |   247491 |           0.02 |      43345 |          86 |              0.21 |      0.07 |            0.13 |            0.36 |\n",
      "| 68.27 68.27 68.27 109.5 109.5 109.5 |   244937 |           0    |      42266 |          84 |              0.25 |      0.07 |            0.12 |            0.37 |\n",
      "| 68.35 68.35 68.35 109.5 109.5 109.5 |   245798 |           0.01 |      38847 |          77 |              0.37 |      0.07 |            0.01 |            0.39 |\n",
      "| 68.41 68.41 68.41 109.5 109.5 109.5 |   246470 |           0.01 |      42992 |          85 |              0.22 |      0.08 |            0.17 |            0.4  |\n",
      "| 68.41 68.41 68.41 109.5 109.5 109.5 |   246471 |           0.01 |      39030 |          77 |              0.36 |      0.07 |            0.04 |            0.41 |\n",
      "| 68.51 68.51 68.51 109.5 109.5 109.5 |   247500 |           0.02 |      42271 |          84 |              0.25 |      0.08 |            0.16 |            0.43 |\n",
      "| 68.46 68.46 68.46 109.5 109.5 109.5 |   247018 |           0.01 |      41329 |          82 |              0.28 |      0.07 |            0.13 |            0.43 |\n",
      "| 68.40 68.40 68.40 109.5 109.5 109.5 |   246372 |           0.01 |      41473 |          82 |              0.27 |      0.08 |            0.15 |            0.43 |\n",
      "| 68.33 68.33 68.33 109.5 109.5 109.5 |   245625 |           0.01 |      39745 |          79 |              0.34 |      0.07 |            0.09 |            0.44 |\n",
      "| 68.52 68.52 68.52 109.5 109.5 109.5 |   247608 |           0.02 |      41575 |          82 |              0.27 |      0.08 |            0.15 |            0.44 |\n",
      "| 68.33 68.33 68.33 109.5 109.5 109.5 |   245592 |           0.01 |      38188 |          76 |              0.39 |      0.07 |            0.05 |            0.45 |\n",
      "| 68.34 68.34 68.34 109.5 109.5 109.5 |   245715 |           0.01 |      37730 |          75 |              0.41 |      0.07 |            0.04 |            0.46 |\n",
      "| 68.39 68.39 68.39 109.5 109.5 109.5 |   246189 |           0.01 |      38123 |          75 |              0.4  |      0.07 |            0.07 |            0.47 |\n",
      "| 68.39 68.39 68.39 109.5 109.5 109.5 |   246284 |           0.01 |      36354 |          72 |              0.46 |      0.07 |            0    |            0.47 |\n",
      "| 68.50 68.50 68.50 109.5 109.5 109.5 |   247405 |           0.02 |      38761 |          77 |              0.37 |      0.07 |            0.11 |            0.5  |\n",
      "| 68.39 68.39 68.39 109.5 109.5 109.5 |   246292 |           0.01 |      40581 |          80 |              0.31 |      0.08 |            0.19 |            0.5  |\n",
      "| 68.48 68.48 68.48 109.5 109.5 109.5 |   247237 |           0.02 |      41153 |          81 |              0.29 |      0.08 |            0.22 |            0.52 |\n",
      "| 68.47 68.47 68.47 109.5 109.5 109.5 |   247118 |           0.01 |      37188 |          74 |              0.43 |      0.07 |            0.08 |            0.53 |\n",
      "| 68.37 68.37 68.37 109.5 109.5 109.5 |   245978 |           0.01 |      39601 |          78 |              0.34 |      0.08 |            0.18 |            0.53 |\n",
      "| 68.35 68.35 68.35 109.5 109.5 109.5 |   245789 |           0.01 |      35649 |          70 |              0.49 |      0.07 |            0.11 |            0.61 |\n",
      "| 68.72 68.72 68.72 109.5 109.5 109.5 |   249845 |           0.03 |      36824 |          73 |              0.45 |      0.08 |            0.15 |            0.62 |\n",
      "| 68.46 68.46 68.46 109.5 109.5 109.5 |   247023 |           0.01 |      33452 |          66 |              0.58 |      0.07 |            0.05 |            0.65 |\n",
      "| 68.29 68.29 68.29 109.5 109.5 109.5 |   245150 |           0    |      34193 |          68 |              0.55 |      0.08 |            0.24 |            0.79 |\n",
      "| 68.44 68.44 68.44 109.5 109.5 109.5 |   246818 |           0.01 |      29691 |          59 |              0.76 |      0.07 |            0.1  |            0.87 |\n",
      "| 68.38 68.38 68.38 109.5 109.5 109.5 |   246173 |           0.01 |      39493 |          78 |              0.35 |      0.1  |            0.52 |            0.87 |\n",
      "| 68.43 68.43 68.43 109.5 109.5 109.5 |   246621 |           0.01 |      35520 |          70 |              0.5  |      0.09 |            0.43 |            0.94 |\n",
      "| 68.54 68.54 68.54 109.5 109.5 109.5 |   247873 |           0.02 |      26238 |          52 |              0.93 |      0.08 |            0.2  |            1.15 |\n",
      "| 68.73 68.73 68.73 109.5 109.5 109.5 |   249893 |           0.03 |      26671 |          53 |              0.91 |      0.08 |            0.28 |            1.22 |\n",
      "| 68.60 68.60 68.60 109.5 109.5 109.5 |   248500 |           0.02 |      24313 |          48 |              1.04 |      0.08 |            0.18 |            1.25 |\n",
      "| 68.41 68.41 68.41 109.5 109.5 109.5 |   246491 |           0.01 |      22460 |          44 |              1.16 |      0.07 |            0.08 |            1.25 |\n",
      "| 68.32 68.32 68.32 109.5 109.5 109.5 |   245438 |           0    |      24866 |          49 |              1.01 |      0.08 |            0.27 |            1.29 |\n",
      "| 68.31 68.31 68.31 109.5 109.5 109.5 |   245363 |           0    |      21514 |          43 |              1.22 |      0.07 |            0.12 |            1.34 |\n",
      "+-------------------------------------+----------+----------------+------------+-------------+-------------------+-----------+-----------------+-----------------+\n",
      "Using d_min_step 0.0\n",
      "\n",
      "Indexed crystal models:\n",
      "model 1 (49109 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.224, 79.224, 79.224, 90.000, 90.000, 90.000\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8081, -0.5787,  0.1099},\n",
      "                { 0.5890,  0.7936, -0.1523},\n",
      "                { 0.0009,  0.1879,  0.9822}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                { 0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                { 0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       49109 |          1467 | 97.1%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "\n",
      "################################################################################\n",
      "Starting refinement (macro-cycle 1)\n",
      "################################################################################\n",
      "\n",
      "\n",
      "Summary statistics for 49043 observations matched to predictions:\n",
      "+-------------------+---------+---------+---------+----------+-------+\n",
      "|                   |     Min |      Q1 |     Med |       Q3 |   Max |\n",
      "|-------------------+---------+---------+---------+----------+-------|\n",
      "| Xc - Xo (mm)      | -0.9993 | -0.3067 |  -0.213 | -0.01511 | 0.778 |\n",
      "| Yc - Yo (mm)      |  -4.479 | -0.5368 | -0.3529 |   -0.214 | 7.027 |\n",
      "| Phic - Phio (deg) |  -6.831 | 0.08669 |  0.1657 |   0.2438 | 4.574 |\n",
      "| X weights         |   251.7 |   388.2 |   401.5 |      405 | 405.6 |\n",
      "| Y weights         |   239.2 |   375.2 |   396.5 |    403.9 | 405.6 |\n",
      "| Phi weights       |   871.2 |    1199 |    1200 |     1200 |  1200 |\n",
      "+-------------------+---------+---------+---------+----------+-------+\n",
      "\n",
      "Detecting centroid outliers using the Tukey algorithm\n",
      "5135 reflections have been flagged as outliers\n",
      "43908 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 43908 observations matched to predictions:\n",
      "+-------------------+---------+---------+---------+----------+--------+\n",
      "|                   |     Min |      Q1 |     Med |       Q3 |    Max |\n",
      "|-------------------+---------+---------+---------+----------+--------|\n",
      "| Xc - Xo (mm)      | -0.9993 | -0.3094 | -0.2082 | 0.002077 | 0.7292 |\n",
      "| Yc - Yo (mm)      |  -1.197 | -0.5266 | -0.3557 |  -0.2217 | 0.3984 |\n",
      "| Phic - Phio (deg) |  -0.178 | 0.09641 |  0.1657 |    0.235 | 0.5156 |\n",
      "| X weights         |   251.7 |   390.4 |   401.9 |      405 |  405.6 |\n",
      "| Y weights         |   239.2 |   374.7 |   396.1 |    403.8 |  405.6 |\n",
      "| Phi weights       |   964.3 |    1199 |    1200 |     1200 |   1200 |\n",
      "+-------------------+---------+---------+---------+----------+--------+\n",
      "\n",
      "There are 11 parameters to refine against 4999 reflections in 3 dimensions\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |   4999 | 0.30013  | 0.45084  |   0.20487  |\n",
      "|      1 |   4999 | 0.17571  | 0.18268  |   0.14762  |\n",
      "|      2 |   4999 | 0.12687  | 0.14044  |   0.076264 |\n",
      "|      3 |   4999 | 0.083989 | 0.091684 |   0.040343 |\n",
      "|      4 |   4999 | 0.048896 | 0.052117 |   0.033191 |\n",
      "|      5 |   4999 | 0.038968 | 0.043485 |   0.029567 |\n",
      "|      6 |   4999 | 0.037705 | 0.04332  |   0.0287   |\n",
      "|      7 |   4999 | 0.037585 | 0.043398 |   0.028635 |\n",
      "|      8 |   4999 | 0.037578 | 0.043407 |   0.028632 |\n",
      "|      9 |   4999 | 0.037577 | 0.043407 |   0.028632 |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |   4999 |  0.21847 |  0.25237 |    0.28632 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "\n",
      "Refined crystal models:\n",
      "model 1 (49109 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5079(14), 79.5079(14), 79.5079(14), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8087, -0.5774,  0.1125},\n",
      "                { 0.5883,  0.7947, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       49109 |          1521 | 97.0%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "Increasing resolution to 1.41 Angstrom\n",
      "\n",
      "Indexed crystal models:\n",
      "model 1 (50550 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5079(14), 79.5079(14), 79.5079(14), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8087, -0.5774,  0.1125},\n",
      "                { 0.5883,  0.7947, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50666 |           300 | 99.4%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "\n",
      "################################################################################\n",
      "Starting refinement (macro-cycle 2)\n",
      "################################################################################\n",
      "\n",
      "\n",
      "Summary statistics for 50491 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |    Max |\n",
      "|-------------------+---------+----------+------------+---------+--------|\n",
      "| Xc - Xo (mm)      | -0.3578 | -0.02342 |   0.001258 | 0.02294 | 0.4988 |\n",
      "| Yc - Yo (mm)      | -0.5083 |  -0.0256 |   0.001955 |  0.0263 | 0.3868 |\n",
      "| Phic - Phio (deg) | -0.9721 | -0.01688 | -8.161e-05 | 0.01742 |  1.324 |\n",
      "| X weights         |   251.7 |    386.7 |      401.2 |   404.9 |  405.6 |\n",
      "| Y weights         |   239.2 |    372.9 |      395.8 |   403.8 |  405.6 |\n",
      "| Phi weights       |   871.2 |     1199 |       1200 |    1200 |   1200 |\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "\n",
      "Detecting centroid outliers using the Tukey algorithm\n",
      "2486 reflections have been flagged as outliers\n",
      "48005 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 48005 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+---------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |     Max |\n",
      "|-------------------+---------+----------+------------+---------+---------|\n",
      "| Xc - Xo (mm)      | -0.1232 | -0.02178 |   0.001674 | 0.02273 |  0.1196 |\n",
      "| Yc - Yo (mm)      | -0.1505 | -0.02437 |    0.00215 | 0.02571 |  0.1601 |\n",
      "| Phic - Phio (deg) | -0.0736 | -0.01662 | -8.828e-05 | 0.01712 | 0.07164 |\n",
      "| X weights         |   251.7 |    388.4 |      401.6 |     405 |   405.6 |\n",
      "| Y weights         |   239.2 |    374.7 |      396.5 |   403.9 |   405.6 |\n",
      "| Phi weights       |   964.3 |     1199 |       1200 |    1200 |    1200 |\n",
      "+-------------------+---------+----------+------------+---------+---------+\n",
      "\n",
      "There are 11 parameters to refine against 4999 reflections in 3 dimensions\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |   4999 | 0.035247 | 0.037583 |   0.021957 |\n",
      "|      1 |   4999 | 0.035417 | 0.037766 |   0.020655 |\n",
      "|      2 |   4999 | 0.035426 | 0.038193 |   0.020065 |\n",
      "|      3 |   4999 | 0.035392 | 0.038391 |   0.019866 |\n",
      "|      4 |   4999 | 0.035371 | 0.03841  |   0.01978  |\n",
      "|      5 |   4999 | 0.035365 | 0.038403 |   0.019746 |\n",
      "|      6 |   4999 | 0.035366 | 0.038403 |   0.01974  |\n",
      "|      7 |   4999 | 0.035367 | 0.038403 |   0.019739 |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |   4999 |  0.20562 |  0.22327 |    0.19739 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "\n",
      "Refined crystal models:\n",
      "model 1 (50550 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5201(8), 79.5201(8), 79.5201(8), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50550 |           416 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "Increasing resolution to 1.37 Angstrom\n",
      "\n",
      "Indexed crystal models:\n",
      "model 1 (50646 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5201(8), 79.5201(8), 79.5201(8), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50652 |           411 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "\n",
      "################################################################################\n",
      "Starting refinement (macro-cycle 3)\n",
      "################################################################################\n",
      "\n",
      "\n",
      "Summary statistics for 50587 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |    Max |\n",
      "|-------------------+---------+----------+------------+---------+--------|\n",
      "| Xc - Xo (mm)      | -0.3544 | -0.02244 |   0.001937 | 0.02366 | 0.5001 |\n",
      "| Yc - Yo (mm)      | -0.3488 | -0.02656 |    0.00112 | 0.02558 | 0.3832 |\n",
      "| Phic - Phio (deg) | -0.9592 | -0.01627 | -0.0006656 | 0.01571 |  1.254 |\n",
      "| X weights         |   251.7 |    386.6 |      401.2 |   404.9 |  405.6 |\n",
      "| Y weights         |   239.2 |    372.7 |      395.8 |   403.8 |  405.6 |\n",
      "| Phi weights       |   871.2 |     1199 |       1200 |    1200 |   1200 |\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "\n",
      "Detecting centroid outliers using the Tukey algorithm\n",
      "2275 reflections have been flagged as outliers\n",
      "48312 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 48312 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+---------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |     Max |\n",
      "|-------------------+---------+----------+------------+---------+---------|\n",
      "| Xc - Xo (mm)      | -0.1211 | -0.02066 |   0.002458 | 0.02356 |  0.1211 |\n",
      "| Yc - Yo (mm)      | -0.1558 | -0.02561 |   0.001264 | 0.02504 |  0.1645 |\n",
      "| Phic - Phio (deg) | -0.0678 | -0.01612 | -0.0005996 | 0.01559 | 0.06876 |\n",
      "| X weights         |   251.7 |    388.4 |      401.6 |     405 |   405.6 |\n",
      "| Y weights         |   239.2 |    374.7 |      396.5 |   403.9 |   405.6 |\n",
      "| Phi weights       |   964.3 |     1199 |       1200 |    1200 |    1200 |\n",
      "+-------------------+---------+----------+------------+---------+---------+\n",
      "\n",
      "There are 11 parameters to refine against 4999 reflections in 3 dimensions\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |   4999 | 0.036216 | 0.037805 |   0.019918 |\n",
      "|      1 |   4999 | 0.036267 | 0.03787  |   0.019785 |\n",
      "|      2 |   4999 | 0.0362   | 0.038014 |   0.019712 |\n",
      "|      3 |   4999 | 0.036136 | 0.038085 |   0.019693 |\n",
      "|      4 |   4999 | 0.036093 | 0.038118 |   0.019691 |\n",
      "|      5 |   4999 | 0.036079 | 0.038129 |   0.019691 |\n",
      "|      6 |   4999 | 0.036078 | 0.03813  |   0.019691 |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |   4999 |  0.20975 |  0.22169 |    0.19691 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "\n",
      "Refined crystal models:\n",
      "model 1 (50646 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5233(7), 79.5233(7), 79.5233(7), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50646 |           417 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "Increasing resolution to 1.32 Angstrom\n",
      "\n",
      "Indexed crystal models:\n",
      "model 1 (50678 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5233(7), 79.5233(7), 79.5233(7), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50681 |           414 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "\n",
      "################################################################################\n",
      "Starting refinement (macro-cycle 4)\n",
      "################################################################################\n",
      "\n",
      "\n",
      "Summary statistics for 50619 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |    Max |\n",
      "|-------------------+---------+----------+------------+---------+--------|\n",
      "| Xc - Xo (mm)      | -0.3525 | -0.02188 |    0.00211 | 0.02388 | 0.4988 |\n",
      "| Yc - Yo (mm)      | -0.3491 | -0.02631 |   0.001491 | 0.02597 | 0.3829 |\n",
      "| Phic - Phio (deg) | -0.9549 | -0.01633 | -0.0008644 | 0.01531 |  1.247 |\n",
      "| X weights         |   251.7 |    386.6 |      401.2 |   404.9 |  405.6 |\n",
      "| Y weights         |   239.2 |    372.7 |      395.7 |   403.8 |  405.6 |\n",
      "| Phi weights       |   871.2 |     1199 |       1200 |    1200 |   1200 |\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "\n",
      "Detecting centroid outliers using the Tukey algorithm\n",
      "2278 reflections have been flagged as outliers\n",
      "48341 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 48341 observations matched to predictions:\n",
      "+-------------------+----------+----------+------------+---------+---------+\n",
      "|                   |      Min |       Q1 |        Med |      Q3 |     Max |\n",
      "|-------------------+----------+----------+------------+---------+---------|\n",
      "| Xc - Xo (mm)      |  -0.1194 |  -0.0202 |   0.002575 | 0.02368 |  0.1209 |\n",
      "| Yc - Yo (mm)      |  -0.1565 |  -0.0254 |   0.001586 | 0.02541 |   0.164 |\n",
      "| Phic - Phio (deg) | -0.06681 | -0.01621 | -0.0008229 | 0.01519 | 0.06323 |\n",
      "| X weights         |    251.7 |    388.4 |      401.6 |     405 |   405.6 |\n",
      "| Y weights         |    239.2 |    374.6 |      396.5 |   403.9 |   405.6 |\n",
      "| Phi weights       |    964.3 |     1199 |       1200 |    1200 |    1200 |\n",
      "+-------------------+----------+----------+------------+---------+---------+\n",
      "\n",
      "There are 11 parameters to refine against 4999 reflections in 3 dimensions\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |   4999 | 0.035858 | 0.038671 |   0.01959  |\n",
      "|      1 |   4999 | 0.035965 | 0.038652 |   0.019513 |\n",
      "|      2 |   4999 | 0.035942 | 0.038717 |   0.019477 |\n",
      "|      3 |   4999 | 0.035945 | 0.03872  |   0.019467 |\n",
      "|      4 |   4999 | 0.035953 | 0.038704 |   0.019463 |\n",
      "|      5 |   4999 | 0.035956 | 0.038693 |   0.01946  |\n",
      "|      6 |   4999 | 0.035958 | 0.03869  |   0.019458 |\n",
      "|      7 |   4999 | 0.03596  | 0.038689 |   0.019457 |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |   4999 |  0.20907 |  0.22494 |    0.19457 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "\n",
      "Refined crystal models:\n",
      "model 1 (50678 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5251(6), 79.5251(6), 79.5251(6), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0030,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50678 |           417 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "Increasing resolution to 1.28 Angstrom\n",
      "\n",
      "Indexed crystal models:\n",
      "model 1 (50694 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5251(6), 79.5251(6), 79.5251(6), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0030,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50697 |           414 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "\n",
      "################################################################################\n",
      "Starting refinement (macro-cycle 5)\n",
      "################################################################################\n",
      "\n",
      "\n",
      "Summary statistics for 50635 observations matched to predictions:\n",
      "+-------------------+---------+----------+-----------+---------+--------+\n",
      "|                   |     Min |       Q1 |       Med |      Q3 |    Max |\n",
      "|-------------------+---------+----------+-----------+---------+--------|\n",
      "| Xc - Xo (mm)      |  -0.353 | -0.02216 |  0.001896 | 0.02345 | 0.4996 |\n",
      "| Yc - Yo (mm)      |   -0.35 |  -0.0266 |  0.001279 | 0.02607 | 0.3826 |\n",
      "| Phic - Phio (deg) | -0.9537 | -0.01658 | -0.001148 | 0.01489 |  1.243 |\n",
      "| X weights         |   251.7 |    386.5 |     401.2 |   404.9 |  405.6 |\n",
      "| Y weights         |   239.2 |    372.6 |     395.7 |   403.8 |  405.6 |\n",
      "| Phi weights       |   871.2 |     1199 |      1200 |    1200 |   1200 |\n",
      "+-------------------+---------+----------+-----------+---------+--------+\n",
      "\n",
      "Detecting centroid outliers using the Tukey algorithm\n",
      "2296 reflections have been flagged as outliers\n",
      "48339 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 48339 observations matched to predictions:\n",
      "+-------------------+---------+----------+-----------+---------+---------+\n",
      "|                   |     Min |       Q1 |       Med |      Q3 |     Max |\n",
      "|-------------------+---------+----------+-----------+---------+---------|\n",
      "| Xc - Xo (mm)      | -0.1194 |  -0.0204 |  0.002369 | 0.02333 |  0.1203 |\n",
      "| Yc - Yo (mm)      | -0.1593 | -0.02576 |  0.001357 |  0.0254 |  0.1681 |\n",
      "| Phic - Phio (deg) |  -0.065 | -0.01643 | -0.001105 | 0.01478 | 0.06631 |\n",
      "| X weights         |   251.7 |    388.3 |     401.6 |     405 |   405.6 |\n",
      "| Y weights         |   239.2 |    374.6 |     396.5 |   403.9 |   405.6 |\n",
      "| Phi weights       |   964.3 |     1199 |      1200 |    1200 |    1200 |\n",
      "+-------------------+---------+----------+-----------+---------+---------+\n",
      "\n",
      "There are 11 parameters to refine against 4999 reflections in 3 dimensions\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |   4999 | 0.036504 | 0.038097 |   0.01991  |\n",
      "|      1 |   4999 | 0.036233 | 0.038173 |   0.019978 |\n",
      "|      2 |   4999 | 0.036236 | 0.038059 |   0.02003  |\n",
      "|      3 |   4999 | 0.036208 | 0.03804  |   0.02005  |\n",
      "|      4 |   4999 | 0.036177 | 0.038056 |   0.020055 |\n",
      "|      5 |   4999 | 0.036163 | 0.038063 |   0.020058 |\n",
      "|      6 |   4999 | 0.036161 | 0.038063 |   0.020059 |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |   4999 |  0.21024 |   0.2213 |    0.20059 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "\n",
      "Refined crystal models:\n",
      "model 1 (50694 reflections):\n",
      "Crystal:\n",
      "    Unit cell: 79.5217(7), 79.5217(7), 79.5217(7), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "+------------+-------------+---------------+-------------+\n",
      "|   Imageset |   # indexed |   # unindexed | % indexed   |\n",
      "|------------+-------------+---------------+-------------|\n",
      "|          0 |       50694 |           417 | 99.2%       |\n",
      "+------------+-------------+---------------+-------------+\n",
      "Saving refined experiments to indexed.expt\n",
      "Saving refined reflections to indexed.refl\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.index imported.expt strong.refl space_group=199"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a655fba-d8e3-4fd8-95a1-c1b5ddb37c78",
   "metadata": {},
   "source": [
    "The indexing program produces a lot of output. For diffuse scattering, it's important to verify that all peaks are indexed well. There might be split Bragg reflections, multiple lattices, or salt crystals contaminating the signal. Or, the crystal might have slipped during data collection.\n",
    "\n",
    "Check that indexing was successful by finding the following parameters at the end of the *dials.index* output, above.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question 1:</b> Find the table with \"% indexed\".  How many strong peaks could not be indexed?\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question 2:</b> Find the table called \"RMSDs by experiment\". What is the deviation in peak position on the detector? What is deviation in frame number, or degrees of oscillation?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "517a53ff-c2f0-4bf8-8eff-6c08268cc475",
   "metadata": {},
   "source": [
    "## Refinement\n",
    "\n",
    "*DIALS* can refine a scan-varying model of the crystal geometry to improve the predicted spot locations. \n",
    "\n",
    "This can be important if the lattice constants change due to radiation damage, or if the crystal is not held rigidly in the loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bd51eb78-c766-474a-aecf-9ceee70d4fa5",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "DIALS 3.dev\n",
      "The following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  experiments = indexed.expt\n",
      "  reflections = indexed.refl\n",
      "}\n",
      "\n",
      "Configuring refiner\n",
      "Setting outlier.nproc=6\n",
      "\n",
      "Summary statistics for 50635 observations matched to predictions:\n",
      "+-------------------+---------+----------+-----------+---------+--------+\n",
      "|                   |     Min |       Q1 |       Med |      Q3 |    Max |\n",
      "|-------------------+---------+----------+-----------+---------+--------|\n",
      "| Xc - Xo (mm)      | -0.3512 | -0.02151 |  0.002323 | 0.02393 | 0.4996 |\n",
      "| Yc - Yo (mm)      | -0.3488 | -0.02623 |  0.001705 | 0.02626 | 0.3831 |\n",
      "| Phic - Phio (deg) |  -0.958 | -0.01668 | -0.001109 |  0.0151 |  1.247 |\n",
      "| X weights         |   251.7 |    386.5 |     401.2 |   404.9 |  405.6 |\n",
      "| Y weights         |   239.2 |    372.6 |     395.7 |   403.8 |  405.6 |\n",
      "| Phi weights       |   871.2 |     1199 |      1200 |    1200 |   1200 |\n",
      "+-------------------+---------+----------+-----------+---------+--------+\n",
      "\n",
      "Detecting centroid outliers using the MCD algorithm\n",
      "5287 reflections have been flagged as outliers\n",
      "45348 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 45348 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+---------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |     Max |\n",
      "|-------------------+---------+----------+------------+---------+---------|\n",
      "| Xc - Xo (mm)      | -0.1046 | -0.01683 |   0.004135 | 0.02421 |  0.1181 |\n",
      "| Yc - Yo (mm)      | -0.1679 | -0.02296 |   0.002438 | 0.02516 |  0.1757 |\n",
      "| Phic - Phio (deg) | -0.0688 | -0.01609 | -0.0009327 | 0.01479 | 0.06355 |\n",
      "| X weights         |   251.7 |    390.1 |        402 |     405 |   405.6 |\n",
      "| Y weights         |   239.2 |    376.7 |      397.5 |   404.1 |   405.6 |\n",
      "| Phi weights       |   964.3 |     1199 |       1200 |    1200 |    1200 |\n",
      "+-------------------+---------+----------+------------+---------+---------+\n",
      "\n",
      "There are 11 parameters to refine against 45348 reflections in 3 dimensions\n",
      "Performing refinement of a single Experiment...\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |  45348 | 0.031801 | 0.037464 |   0.019562 |\n",
      "|      1 |  45348 | 0.032652 | 0.036505 |   0.019494 |\n",
      "|      2 |  45348 | 0.032681 | 0.036444 |   0.019501 |\n",
      "|      3 |  45348 | 0.032722 | 0.036382 |   0.019503 |\n",
      "|      4 |  45348 | 0.032763 | 0.036335 |   0.019503 |\n",
      "|      5 |  45348 | 0.032781 | 0.036316 |   0.019503 |\n",
      "|      6 |  45348 | 0.032785 | 0.036311 |   0.019504 |\n",
      "|      7 |  45348 | 0.032785 | 0.03631  |   0.019504 |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |  45348 |  0.19061 |   0.2111 |    0.19504 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "Updating predictions for indexed reflections\n",
      "\n",
      "Scan-varying refinement\n",
      "Configuring refiner\n",
      "\n",
      "Summary statistics for 50634 observations matched to predictions:\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "|                   |     Min |       Q1 |        Med |      Q3 |    Max |\n",
      "|-------------------+---------+----------+------------+---------+--------|\n",
      "| Xc - Xo (mm)      |  -0.359 | -0.02517 | -0.0001893 | 0.02197 | 0.4985 |\n",
      "| Yc - Yo (mm)      | -0.3486 |  -0.0273 |  0.0001673 | 0.02432 | 0.3841 |\n",
      "| Phic - Phio (deg) | -0.9571 | -0.01607 | -0.0005645 | 0.01559 | 0.9406 |\n",
      "| X weights         |   251.7 |    386.5 |      401.2 |   404.9 |  405.6 |\n",
      "| Y weights         |   239.2 |    372.6 |      395.7 |   403.8 |  405.6 |\n",
      "| Phi weights       |   964.3 |     1199 |       1200 |    1200 |   1200 |\n",
      "+-------------------+---------+----------+------------+---------+--------+\n",
      "\n",
      "Detecting centroid outliers using the MCD algorithm\n",
      "5124 reflections have been flagged as outliers\n",
      "45510 reflections remain in the manager\n",
      "\n",
      "Summary statistics for 45510 observations matched to predictions:\n",
      "+-------------------+----------+----------+------------+---------+---------+\n",
      "|                   |      Min |       Q1 |        Med |      Q3 |     Max |\n",
      "|-------------------+----------+----------+------------+---------+---------|\n",
      "| Xc - Xo (mm)      |  -0.1133 | -0.02032 |   0.001799 | 0.02243 |   0.117 |\n",
      "| Yc - Yo (mm)      |   -0.154 |  -0.0242 |  0.0009423 | 0.02332 |  0.1649 |\n",
      "| Phic - Phio (deg) | -0.06599 |  -0.0155 | -0.0003895 |  0.0153 | 0.06719 |\n",
      "| X weights         |    251.7 |    389.9 |        402 |     405 |   405.6 |\n",
      "| Y weights         |    239.2 |    376.6 |      397.4 |   404.1 |   405.6 |\n",
      "| Phi weights       |    964.3 |     1199 |       1200 |    1200 |    1200 |\n",
      "+-------------------+----------+----------+------------+---------+---------+\n",
      "\n",
      "There are 15 parameters to refine against 45510 reflections in 3 dimensions\n",
      "Performing refinement of a single Experiment...\n",
      "\n",
      "Refinement steps:\n",
      "+--------+--------+----------+----------+------------+\n",
      "|   Step |   Nref |   RMSD_X |   RMSD_Y |   RMSD_Phi |\n",
      "|        |        |     (mm) |     (mm) |      (deg) |\n",
      "|--------+--------+----------+----------+------------|\n",
      "|      0 |  45510 | 0.032795 | 0.036469 |   0.019535 |\n",
      "|      1 |  45510 | 0.03033  | 0.030962 |   0.019765 |\n",
      "|      2 |  45510 | 0.030291 | 0.031014 |   0.019627 |\n",
      "|      3 |  45510 | 0.030271 | 0.031033 |   0.019522 |\n",
      "|      4 |  45510 | 0.030261 | 0.031038 |   0.019497 |\n",
      "|      5 |  45510 | 0.030255 | 0.031036 |   0.019494 |\n",
      "|      6 |  45510 | 0.030243 | 0.031028 |   0.019489 |\n",
      "|      7 |  45510 | 0.030226 | 0.031016 |   0.019485 |\n",
      "|      8 |  45510 | 0.030212 | 0.031008 |   0.019487 |\n",
      "|      9 |  45510 | 0.030207 | 0.031006 |   0.01949  |\n",
      "|     10 |  45510 | 0.030206 | 0.031006 |   0.01949  |\n",
      "+--------+--------+----------+----------+------------+\n",
      "RMSD no longer decreasing\n",
      "\n",
      "RMSDs by experiment:\n",
      "+-------+--------+----------+----------+------------+\n",
      "|   Exp |   Nref |   RMSD_X |   RMSD_Y |     RMSD_Z |\n",
      "|    id |        |     (px) |     (px) |   (images) |\n",
      "|-------+--------+----------+----------+------------|\n",
      "|     0 |  45510 |  0.17562 |  0.18026 |     0.1949 |\n",
      "+-------+--------+----------+----------+------------+\n",
      "Updating predictions for indexed reflections\n",
      "\n",
      "Final refined crystal model:\n",
      "Crystal:\n",
      "    Unit cell: 79.5227(2), 79.5227(2), 79.5227(2), 90.0, 90.0, 90.0\n",
      "    Space group: I 21 3\n",
      "    U matrix:  {{ 0.8086, -0.5775,  0.1125},\n",
      "                { 0.5884,  0.7946, -0.1495},\n",
      "                {-0.0031,  0.1871,  0.9823}}\n",
      "    B matrix:  {{ 0.0126,  0.0000,  0.0000},\n",
      "                {-0.0000,  0.0126,  0.0000},\n",
      "                {-0.0000,  0.0000,  0.0126}}\n",
      "    A = UB:    {{ 0.0102, -0.0073,  0.0014},\n",
      "                { 0.0074,  0.0100, -0.0019},\n",
      "                {-0.0000,  0.0024,  0.0124}}\n",
      "    A sampled at 501 scan points\n",
      "Saving refined experiments to refined.expt\n",
      "Saving reflections with updated predictions to refined.refl\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.refine indexed.expt indexed.refl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b84defa-9fb6-4110-bfb7-e211bfd566b6",
   "metadata": {},
   "source": [
    "Find the table of `RMSDs by experiment`. \n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question:</b> Did the spot predictions improve?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34599c0a-47d1-4714-ac23-fb50e5326418",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "*DIALS* report contains some useful plots."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5e3fdda5-003e-40c4-996b-e5a9b311c318",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "The following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  experiments = refined.expt\n",
      "  reflections = refined.refl\n",
      "}\n",
      "\n",
      "Analysing strong spots\n",
      " Selected 51111 strong reflections.........................................0.01s\n",
      "Analysing reflection centroids\n",
      " Selected 45510 refined reflections........................................0.01s\n",
      " Analysing centroid differences with I/Sigma > 0\n",
      " Analysing centroid differences in x/y with I/Sigma > 0\n",
      " Analysing centroid differences in z with I/Sigma > 0\n",
      " Analysing centroid differences vs phi with I/Sigma > 0\n",
      "Analysing reflection intensities\n",
      " Selecting only integrated reflections...Analysing reference profiles\n",
      " Skipping: following required fields not present:\n",
      "  intensity.prf.value\n",
      "  intensity.prf.variance\n",
      "  profile.correlation\n",
      "Analysing scan-varying crystal model\n",
      "Analysing scaling model\n",
      "Calculating and generating merging statistics plots\n",
      "Writing html report to: dials.report.html\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.report refined.expt refined.refl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb7975db-49f5-4f2d-942f-0f454ac50c67",
   "metadata": {},
   "source": [
    "Open the file [dials.report.html](dials.report.html).\n",
    "\n",
    "If you are unable to expand the menus, click \"Trust HTML\" in the upper left corner.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question 1:</b> Expand the tab called \"Analysis of scan varying model\". Did the unit cell axes change size during data collection? Did the crystal slip?\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question 2:</b> Expand the tab called \"Analysis of reflection centroids\". Is the error in X, Y position random? (hint: the Pilatus 6M has discrete modules). Are the spot centroid errors mostly compact, or are there possibly multiple lattices?\n",
    "</div>\n",
    "\n",
    "I think you will agree that the dataset is fairly ideal, and we can confidently apply the scan-varying model for diffuse mapping."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23743ffe-54c4-40b8-b627-50a4b7264df6",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Preparing for *mdx2*\n",
    "\n",
    "For rest of the workshop we'll be using *mdx2*. We need to do two things to prepare:\n",
    "\n",
    "1. Import background scattering images\n",
    "2. Prepare a smaller dataset for testing purposes\n",
    "\n",
    "The experiment was performed at ambient temperature. To maintain crystal hydration, it was inside a plastic capillary, which has significant scattering. To measure this scattering (and also the scattering of air) the crystal was translated out of the beam, and the same wedge of data was acquired (with 1 deg. oscillation rather than 0.1 deg.)\n",
    "\n",
    "First, import the background images and apply the mask."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f1463ccb-0efa-43d2-9730-d05adac43dae",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235\n",
      "DIALS 3.dev\n",
      "The following parameters have been modified:\n",
      "\n",
      "output {\n",
      "  experiments = \"background.expt\"\n",
      "}\n",
      "input {\n",
      "  experiments = <image files>\n",
      "}\n",
      "\n",
      "--------------------------------------------------------------------------------\n",
      "  format: <class 'dxtbx.format.FormatCBFMiniPilatusCHESS_6MSN127.FormatCBFMiniPilatusCHESS_6MSN127'>\n",
      "  num images: 50\n",
      "  sequences:\n",
      "    still:    0\n",
      "    sweep:    1\n",
      "  num stills: 0\n",
      "--------------------------------------------------------------------------------\n",
      "Writing experiments to background.expt\n",
      "\u001b[0mThe following parameters have been modified:\n",
      "\n",
      "input {\n",
      "  mask = \"pixels.mask\"\n",
      "  experiments = background.expt\n",
      "}\n",
      "output {\n",
      "  experiments = \"background.expt\"\n",
      "}\n",
      "\n",
      "Writing experiments to background.expt\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.import images/insulin_2_bkg output.experiments=background.expt\n",
    "!dials.apply_mask background.expt input.mask=pixels.mask output.experiments=background.expt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5bcf060-e832-4113-a09e-bcf9827926e2",
   "metadata": {},
   "source": [
    "Open *background.expt* using *dials.image_viewer*.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\"> \n",
    "    <b>Question:</b> What features do you see in the background images?\n",
    "</div>\n",
    "\n",
    "Finally, we'll make a smaller dataset for testing purposes. The full scan was 50 degrees (500 images). The test set will be 6 degrees (60 images)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ade1273e-5b53-40fc-a473-7d31def3323f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The following parameters have been modified:\n",
      "\n",
      "image_range = 1 60\n",
      "input {\n",
      "  experiments = refined.expt\n",
      "}\n",
      "\n",
      "Saving sliced experiments to refined_1_60.expt\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!dials.slice_sequence refined.expt \"image_range=1 60\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}