Papers.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": "# Paper summaries\n**Papers 1 to 14**",
      "metadata": {
        "tags": [],
        "cell_id": "00000-18b34093-0c2a-4d9d-b96c-7c5324cea60e",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 1: Quantifying Independently Reproducible ML Research\nNot being able to reproduce a paper's results may suggest problem with the paper. Author investigates by independently reproducing; **reproducing using only the information given in the paper**.\n\nInfluencing features for reproducability:\n- Unambigiouous:\n    - #authors, #references, etc...\n- Mild subjectivity:\n    - #tables, #equations, hyperparameters, pseudocode\n- Subjective:\n    - #conceptualization figures, rigor vs. empirical (type of research), readability, algorithms difficulty.\n\nMost important:\n- _Year of publication_: uncorrelated with reproducability\n- _Readability_: most strongly correlated with reproducability; shorter pages seem to make it less readable; suggests page limit negatively influence readability.\n- _Pseudocode_: correlated; positive for no pseudo-code, negative for in between or none.\n- _Theoretical papers_: harder to reproduce\n- _Primary topic_: significantly correlated to reproducibility; 'living' papers possibly more reproducible.\n- _code release_: uncorrelated.\n\nMany study deficiencies, like **author bias** (how experienced, topic, only implemented by one), no record of failure.",
      "metadata": {
        "tags": [],
        "cell_id": "00001-97ebca41-788e-4a26-b116-04ab09359f07",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 2: Troubling Trends in Machine Learning Scholarship\n\n1. **Failure to distinguish** between **explanation and speculation**; Speculation is like \"strong results lead to complacency\"\n2. **Sources of empirical gain**; emphasizing the wrong things; like modifications that don't do much and thus is related to hyper-parameter tuning.\n3. **Mathiness**: impress rather than clarify concepts.\n4. **Language misuse**: use wrong words to clarify the paper concepts.\n\nAuthor argue that strong results are seen as a valid excuse for weak arguments.\nCombatting these problems could make ML more accessible.",
      "metadata": {
        "tags": [],
        "cell_id": "00002-1600d10e-425d-43fe-b03c-2a7e7ea71a25",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 3: ImageNet Classifiers Generalize?\nImage classifiers do not generalize reliably. This is shown by trying to reproduce the test results of image recognition models on a **new** test set which is sampled from the **same** data source, following the **same** data cleaning protocol as described in the paper. This is tested by reproducing some of the highly ranked models in this manner.\n\nThe classification models fail to reach their original accuracy scores using the new test set (CIFAR-10 3 to 15% drop, ImageNet 11 to 14%). However, the ranking of the highly ranked models is mostly preserved.\n\nThere are three types of gaps identified:\n- **Generalization gap**: determined solely by random sampling error. Since the new test set had 10.000 data points, this should at most lead to maximally +/- 1% difference in accuracy. Thus does **not explain** the drop in accuracy.\n- **Adaptivity gap**: the adaptation of a model to the test set with regards to the true distribution. If we assume this to be true that would mean each model's hyperparameters are tuned in somewhat similar fashion with regards to the test set. Since the ranking of the models are preserved, intuitively it would also seem **unlikely to explain it**.\n- **Distribution gap**: the Systemic difference between the current true distribution proxied by the current test set and the new true distribution, which the new test set is a proxy for. This seems the **most likely explanation**.\n\nAlso the paper suggests that the gap could be explained by the inability to generalize to slightly “harder” images than those found in the original test sets.\n\n\n**Question**: Models trained on existing datasets may not generalize to new test sets sampled from the same distribution. What was the paper’s hypothesis for that?\n- “The existing test sets have been used too many times and existing models are tuned to the particular test set.” - This is what is initially assumed but later debunked by the fact that ranking is preserved.\n",
      "metadata": {
        "tags": [],
        "cell_id": "00003-85bca771-897f-4db0-86bd-5a53b2279a0b",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 4: Scaling down Deep learning\n\nProjects run at large scale require enormous amounts of **time**, **money**, and **electricity**. MNIST-1D is a minimalist, low-memory, low-compute alternative to classic deep learning benchmarks. This dataset differentiates more clearly between linear, non-linear, and convolutional models in comparison with the original MNIST dataset. In addition, MNIST is somewhat large for a toy dataset, and hard to hack (researchers cannot easily vary parameters).\n\nThe authors state that small scale research is important in the DL field, because it permits creativity, allows deep understanding, improves interpretability, reproducibility and iteration speed. It is easy to perform ablation studies to isolate causal mechanisms of results (e.g. for finding what attributes to a lottery ticket’s success). MNIST-1D has rapid iteration as a priority. However, large scale research is also required: to expose fertile new research territory.\n\nProperties of MNIST-1D\n- Extremely small (smaller than MNIST)\n- Able to identify models with (spatial) inductive biases\n- Easy to hack, extend or modify\n- Analogous to large-scale problems\n- Differentiates clearly between linear, non-linear, and convolutional models (e.g. logistic vs MLP vs CNN vs GRU) \n\n",
      "metadata": {
        "tags": [],
        "cell_id": "00004-c0479da6-9c55-4128-8df7-2e6cccc8541e",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 5: Deep Convolutional Really need to be Deep and Convolutional?\n**Goal**: check if shallow networks can be as accurate as deeper ones.\n\n**Main contributions**: \n- Still substantial performance gap between deep CNN's and shallow CNN's/NN's (image recognition)\n- Training via distillation yields higher model accuracy than conventional training\n\n**Distillation technique**:\n- Use a teacher models real values output scores and mimick those.\n\n**Main finding**: \nIf student models have a similar number of parameters as the deep teacher models, high accuracy can not be achieved without multiple layers of convolution even when the student models are trained via distillation\n- deep convolutional nets do need to be both deep and convolutional even when trained to mimic very accurate models via distillation\n\n**Conclusion**:\nYes, we need Convolutions and Deep. Although shallow can learn Deep for Speech, not for Image recognition. \n- **A network with a single hidden layer cannot approximate every decision boundary**.\n- Accuracy can be significantly be improved using several layers of convolution.\n- On high dimensional data shallow neural networks perform worse than deep neural networks.\n- Distillation for models can lead to accurate results.\n\n",
      "metadata": {
        "tags": [],
        "cell_id": "00005-83c63c0b-d9ce-4e8e-95c7-c12ceb1d2c8f",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 6: Residual Networks Behave Like Ensembles of Relatively Shallow Networks\n**Goal**:\n- Investigate the impact of the following to confirm whether the Residual Network behaves like _\"Ensemble\"_.\n    - identity **skip-connections**\n    - **paths** are **not dependent** on each other\n    - skip connections give **rise to large networks**\n- Shortcut path -> create more complex model without vanishing gradient\n\n**Ensemble**:\n- Ensemble means that arranging a committee of neural networks in a simple voting scheme, then the ﬁnal output predictions are averaged. And it has the following features.\n    - **Feature 1**: A path does not depend on each other\n    - **Feature 2**: Performance increase from additional ensemble members gets smaller with increasing ensemble\n\n**Method**:\n- Introduce the unraveled view\n    - residual networks can be viewed as a collection of many paths instead of a single deep network\n- Do a lesion study\n- Investigate the depth of residual networks\n\n**Conclusion**:\n- Unraveled view reveals that **residual networks can be viewed as a collection of many paths**, instead of a single ultra deep network.\n- Lesion studies show that, although paths are trained jointly, they do **not strongly depend on each other**, but show ensemble-like behavior.\n- The paths through the network that contribute gradient during training are shorter than expected. _Longer paths do not contribute to the gradient_.",
      "metadata": {
        "tags": [],
        "cell_id": "00006-cb60f661-b0aa-4399-b27d-8b286ec28184",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 7: Deep Image Prior\n**Goal**:\n- Recover original image when having a corruption. Instead of searching for the answer in the image space we now search for it in the space of neural network's parameters.\n\n**Conclusion**:\n- It thus shows that the approach of constructing a implicit prior inside deep convolution neural network architectures with randomized weights is well-suited for image restoration tasks.\n- A randomly-initialized neural network can be used as a handcrafted prior with excellent results in standard inverse problems such as denoising, super-resolution, and inpainting.",
      "metadata": {
        "tags": [],
        "cell_id": "00007-888161f2-aa24-48d9-b037-ffbd8ef6626b",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 8: Approximating CNN's with Bag-of-local-Features\n\n**Main contributions**:\n- Solving ImageNet is much simpler than many have thought.\n- The findings allow us to build much more interpretable and transparent image classification pipelines.\n- CNNs has a bias towards texture.\n- BagNet is easier to explain (Hybrid of DNN and BoF)\n    - Ex: Medical imaging, autonomous vehicles\n- DNN:\n    - Have better fine tuning rather than qualitatively different decision strategies\n    - Not taking into account spatial ordering\n    - Bad at distribution shifts\n\n**Conclusion**:\n- Deep Neural Networks can still recognise scrambled images well",
      "metadata": {
        "tags": [],
        "cell_id": "00008-14efc387-d1e6-4e29-a1a3-d7cab637a435",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 9: Group Normalization\n\n**Goal**:\n- Current normalisation techniques (used for layer normalisation) used in deep learning include:\n    - BN (batch normalization) which normalizes over batches.\n    - LN (layer normalization) which normalizes over all input channels.\n    - IN (instance normalization) which normalizes over each individual channel.\n- Make a normalisation technique **insensitive to batch size**.\n    - Does not mean it is independent of batch size, it just is insentive to it.\n\n** Contribution**:\n- GN (group normalisation) is between LN and IN where it uses a **given number of channels grouped together to normalise over** (as opposed to LN which does all at once and IN which does one channel at a time). \n- Useful when training with limited batch sizes (due to memory requirement):\n    - image and video classification\n    - segmentation problems\n- BN:\n    - pre-computed statistics may differ **between training vs testing**.\n    - when used in a real life scenario the inputs are single images so no batch normalization can take place. \n        - Issue because the network learned with normalized data. \n        - **Fix**? This is solved by GN\n            - _IN_ and _LN_ perform poorly on image based learning.\n    - reliance on the batch dimension called BR (Batch renormalisation)\n        - improves performance, it still relies on batch dimension.  \n        - Hardware scaling; but increases complexity and does not fix the underlying issue of reliance on the batch size.\n- **Large batch size**: BN better\n- **Small batch size**: GN better\n- **GN outperforms both LN and IN**.",
      "metadata": {
        "tags": [],
        "cell_id": "00009-6ecef7b7-de31-4374-9621-611edc90e9b2",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 10: Empirical Comparisons of Optimizers for DL\n**Finding**\n- Many optimizers, however, unsure how an optimizer will generalize to new workloads (datasets).\n    - **SGD** sometimes outperforms an **ADAM** optimizer. \n        - _is this due to a bad optimization schedule?_ \n- An optimizer is a **combination of optimizer(s)** (N(Adam), SGD, Momentum, Nesterov) and hyperparameters. \n\n<img src=\"./image/optimizers_comparison.png\" height=\"75\" />\n\n- The author: \n    - proposes an inclusion hierarchy / taxonomy.\n    - wants to research whether this inclusion taxonomy holds in practice, so, provide an optimization protocol to do parameter search for an optimizer such that a more general optimizer will always match or outperform a simpler optimizer.\n    - finds that with his proposed optimization protocol this taxonomy holds for all of his tested datasets with different models. \n- General optimizers never underperform specialized ones\n- Hyperparameter search may be the most important factor in optimizer rankings\n- Downside of this is that one would **need enough computational power** to follow his proposed learning schedule.\n",
      "metadata": {
        "tags": [],
        "cell_id": "00010-2495fa9f-53d5-42da-80fa-e141358b4ebf",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 11: Attention Is All You Need\n\n**Contributions**:\n- Word2vec turns discrete word into a d-dimensional one-hot-encoded vector. \n- Calc context via:\n    - Skip gram (from center word to k surrounding) \n    - CBOW (continuous bag-of-words, from surrounding k words to center)\n- Self-attention is a parallelizable alternative to training RNN. Its a seq2seq operation where each input vector will be converted (learned) to have all samples contain information of other samples from the sequence.\n- Uses similarities between words\n    - n^2 number of comparisons via dot-prroduct, softmax and resulting matrix used to take a weighted sum of original word vector.\n- Three have the following functions:\n    - value is used to for the weighted sum\n    - query is used to compare the words to the other vectors\n    - key is used by other weights to compare to\n- Transformers because support parallelization, variable length input, distant relationships\n- Has encoder, decoder, pre and post processing\n- Faster than RNN and thus does not use RNN, GRU or LSTM",
      "metadata": {
        "tags": [],
        "cell_id": "00011-e73f77bf-06b8-4994-8836-51415324cfc2",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 12: Single Headed Attention RNN: Stop Thinking With Your Head\n**Background**:\n- Current NLP research rely a lot on Transformer\n\n**Motivation**\n- Challenges trend by proposing new research direction, which needs less computational power and simpler model, but nearly reach SOTA; thus **there should still exist competition and variety in the types of models for a task.**\n- Investigate a language modelling technique that does not rely on “Transformers” (current SOTA)\n- Lone GPU vs Expensive computation\n- Single head vs multi-headed attention (memory intensive)\n\n**Used**: \n- SHA-RNN: LSTM + single head attention\n\n**Contribution**:\n- Single headed performance better than without attention head (4 headed slightly better, but twice as long training)",
      "metadata": {
        "tags": [],
        "cell_id": "00012-fd545780-5c50-4faf-9b31-4ea16f9273a9",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 13: Unpaired Img2Img Translation using Cycle-Consistent Adversarial Networks\n**Contributions**:\n- Neural style transfer\n- Cycle consistency\n    - Convert Zebra to Horse and then back to Zebra should be the same image.\n- Mode collapse\n    - produce same label maps regardless of input photo / generator produces limities varieties of samples\n- Lacks in transferring shapes",
      "metadata": {
        "tags": [],
        "cell_id": "00013-92158df4-b356-4daf-b55a-9c1654994f75",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Paper 14: Critical analysis of self-supervision\n**Contributions**:\n- BiGAN:\n    - extension of GAN\n- RotNet:\n    - predict \"upright\" direction of image\n- One image is sufficient to train early layers\n- Strong supervision superior to self-supervision in deeper layers",
      "metadata": {
        "tags": [],
        "cell_id": "00014-4a57e97e-59c2-41c3-94f6-ca0c6451df95",
        "deepnote_cell_type": "markdown"
      }
    },
    {
      "cell_type": "markdown",
      "source": "<a style='text-decoration:none;line-height:16px;display:flex;color:#5B5B62;padding:10px;justify-content:end;' href='https://deepnote.com?utm_source=created-in-deepnote-cell&projectId=de0be7a9-29e1-4ab6-9ce7-607fa646094e' target=\"_blank\">\n<img alt='Created in deepnote.com' style='display:inline;max-height:16px;margin:0px;margin-right:7.5px;' src='data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iODBweCIgaGVpZ2h0PSI4MHB4IiB2aWV3Qm94PSIwIDAgODAgODAiIHZlcnNpb249IjEuMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiAgICA8IS0tIEdlbmVyYXRvcjogU2tldGNoIDU0LjEgKDc2NDkwKSAtIGh0dHBzOi8vc2tldGNoYXBwLmNvbSAtLT4KICAgIDx0aXRsZT5Hcm91cCAzPC90aXRsZT4KICAgIDxkZXNjPkNyZWF0ZWQgd2l0aCBTa2V0Y2guPC9kZXNjPgogICAgPGcgaWQ9IkxhbmRpbmciIHN0cm9rZT0ibm9uZSIgc3Ryb2tlLXdpZHRoPSIxIiBmaWxsPSJub25lIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiPgogICAgICAgIDxnIGlkPSJBcnRib2FyZCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTEyMzUuMDAwMDAwLCAtNzkuMDAwMDAwKSI+CiAgICAgICAgICAgIDxnIGlkPSJHcm91cC0zIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMjM1LjAwMDAwMCwgNzkuMDAwMDAwKSI+CiAgICAgICAgICAgICAgICA8cG9seWdvbiBpZD0iUGF0aC0yMCIgZmlsbD0iIzAyNjVCNCIgcG9pbnRzPSIyLjM3NjIzNzYyIDgwIDM4LjA0NzY2NjcgODAgNTcuODIxNzgyMiA3My44MDU3NTkyIDU3LjgyMTc4MjIgMzIuNzU5MjczOSAzOS4xNDAyMjc4IDMxLjY4MzE2ODMiPjwvcG9seWdvbj4KICAgICAgICAgICAgICAgIDxwYXRoIGQ9Ik0zNS4wMDc3MTgsODAgQzQyLjkwNjIwMDcsNzYuNDU0OTM1OCA0Ny41NjQ5MTY3LDcxLjU0MjI2NzEgNDguOTgzODY2LDY1LjI2MTk5MzkgQzUxLjExMjI4OTksNTUuODQxNTg0MiA0MS42NzcxNzk1LDQ5LjIxMjIyODQgMjUuNjIzOTg0Niw0OS4yMTIyMjg0IEMyNS40ODQ5Mjg5LDQ5LjEyNjg0NDggMjkuODI2MTI5Niw0My4yODM4MjQ4IDM4LjY0NzU4NjksMzEuNjgzMTY4MyBMNzIuODcxMjg3MSwzMi41NTQ0MjUgTDY1LjI4MDk3Myw2Ny42NzYzNDIxIEw1MS4xMTIyODk5LDc3LjM3NjE0NCBMMzUuMDA3NzE4LDgwIFoiIGlkPSJQYXRoLTIyIiBmaWxsPSIjMDAyODY4Ij48L3BhdGg+CiAgICAgICAgICAgICAgICA8cGF0aCBkPSJNMCwzNy43MzA0NDA1IEwyNy4xMTQ1MzcsMC4yNTcxMTE0MzYgQzYyLjM3MTUxMjMsLTEuOTkwNzE3MDEgODAsMTAuNTAwMzkyNyA4MCwzNy43MzA0NDA1IEM4MCw2NC45NjA0ODgyIDY0Ljc3NjUwMzgsNzkuMDUwMzQxNCAzNC4zMjk1MTEzLDgwIEM0Ny4wNTUzNDg5LDc3LjU2NzA4MDggNTMuNDE4MjY3Nyw3MC4zMTM2MTAzIDUzLjQxODI2NzcsNTguMjM5NTg4NSBDNTMuNDE4MjY3Nyw0MC4xMjg1NTU3IDM2LjMwMzk1NDQsMzcuNzMwNDQwNSAyNS4yMjc0MTcsMzcuNzMwNDQwNSBDMTcuODQzMDU4NiwzNy43MzA0NDA1IDkuNDMzOTE5NjYsMzcuNzMwNDQwNSAwLDM3LjczMDQ0MDUgWiIgaWQ9IlBhdGgtMTkiIGZpbGw9IiMzNzkzRUYiPjwvcGF0aD4KICAgICAgICAgICAgPC9nPgogICAgICAgIDwvZz4KICAgIDwvZz4KPC9zdmc+' > </img>\nCreated in <span style='font-weight:600;margin-left:4px;'>Deepnote</span></a>",
      "metadata": {
        "tags": [],
        "created_in_deepnote_cell": true,
        "deepnote_cell_type": "markdown"
      }
    }
  ],
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {
    "orig_nbformat": 2,
    "deepnote": {
      "is_reactive": false
    },
    "deepnote_notebook_id": "6c0b5a59-ac1a-4aea-b7d0-7857e7c8dd53",
    "deepnote_execution_queue": []
  }
}