[skip tests] [skip docs] start joss paper draft

.github/workflows/JOSSPaper.yml

@@ -0,0 +1,23 @@
+on: [push]
+
+jobs:
+  paper:
+    runs-on: ubuntu-latest
+    name: Paper Draft
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+      - name: Build draft PDF
+        uses: openjournals/openjournals-draft-action@master
+        with:
+          journal: joss
+          # This should be the path to the paper within your repo.
+          paper-path: joss/paper.md
+      - name: Upload
+        uses: actions/upload-artifact@v1
+        with:
+          name: paper
+          # This is the output path where Pandoc will write the compiled
+          # PDF. Note, this should be the same directory as the input
+          # paper.md
+          path: joss/paper.pdf

joss/paper.bib

@@ -0,0 +1,71 @@
+@article{bai2019deep,
+  title   = {Deep equilibrium models},
+  author  = {Bai, Shaojie and Kolter, J Zico and Koltun, Vladlen},
+  journal = {arXiv preprint arXiv:1909.01377},
+  year    = {2019}
+}
+
+@inproceedings{enzyme:2020,
+  author    = {Moses, William and Churavy, Valentin},
+  booktitle = {Advances in Neural Information Processing Systems},
+  editor    = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
+  pages     = {12472--12485},
+  publisher = {Curran Associates, Inc.},
+  title     = {Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients},
+  url       = {https://proceedings.neurips.cc/paper/2020/file/9332c513ef44b682e9347822c2e457ac-Paper.pdf},
+  volume    = {33},
+  year      = {2020}
+}
+
+@inproceedings{enzyme:2021,
+  author    = {Moses, William S. and Churavy, Valentin and Paehler, Ludger and H\"{u}ckelheim, Jan and Narayanan, Sri Hari Krishna and Schanen, Michel and Doerfert, Johannes},
+  title     = {Reverse-Mode Automatic Differentiation and Optimization of GPU Kernels via Enzyme},
+  year      = {2021},
+  isbn      = {9781450384421},
+  publisher = {Association for Computing Machinery},
+  address   = {New York, NY, USA},
+  url       = {https://doi.org/10.1145/3458817.3476165},
+  doi       = {10.1145/3458817.3476165},
+  booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis},
+  articleno = {61},
+  numpages  = {16},
+  keywords  = {CUDA, LLVM, ROCm, HPC, AD, GPU, automatic differentiation},
+  location  = {St. Louis, Missouri},
+  series    = {SC '21}
+}
+
+@misc{innes2018fashionable,
+  title         = {Fashionable Modelling with Flux},
+  author        = {Michael Innes and Elliot Saba and Keno Fischer and Dhairya Gandhi and Marco Concetto Rudilosso and Neethu Mariya Joy and Tejan Karmali and Avik Pal and Viral Shah},
+  year          = {2018},
+  eprint        = {1811.01457},
+  archiveprefix = {arXiv},
+  primaryclass  = {cs.PL}
+}
+
+@misc{pal2023continuous,
+  title         = {Continuous Deep Equilibrium Models: Training Neural ODEs Faster by Integrating Them to Infinity},
+  author        = {Avik Pal and Alan Edelman and Christopher Rackauckas},
+  year          = {2022},
+  eprint        = {2201.12240},
+  archiveprefix = {arXiv},
+  primaryclass  = {cs.LG}
+}
+
+@misc{simplechains,
+  author    = {Elrod, Chris},
+  title     = {SimpleChains.jl},
+  year      = {2021},
+  publisher = {GitHub},
+  journal   = {GitHub repository},
+  url       = {https://github.com/PumasAI/SimpleChains.jl}
+}
+
+@inproceedings{yuret2016knet,
+  title     = {Knet: beginning deep learning with 100 lines of julia},
+  author    = {Yuret, Deniz},
+  booktitle = {Machine Learning Systems Workshop at NIPS},
+  volume    = {2016},
+  pages     = {5},
+  year      = {2016}
+}

joss/paper.md

@@ -0,0 +1,72 @@
+---
+title: 'Lux.jl: Bridging Scientific Computing & Deep Learning'
+tags:
+  - Julia
+  - Deep Learning
+  - Scientific Computing
+  - Neural Ordinary Differential Equations
+  - Deep Equilibrium Models
+authors:
+  - name: Avik Pal
+    orcid: 0000-0002-3938-7375
+    affiliation: "1"
+affiliations:
+ - name: Electrical Engineering and Computer Science, CSAIL, MIT
+   index: 1
+date: 2 October 2023
+bibliography: paper.bib
+---
+
+# Summary
+
+Combining Machine Learning and Scientific Computing have recently led to development of
+methods like Universal Differential Equations, Neural Differential Equations, Deep Equilibrium Models, etc.,
+which have been pushing the boundaries of physical sciences. However, every major deep learning
+framework requires the numerical softwares to be rewritten to satisfy their specific requirements.
+Lux.jl is a deep learning framework written in Julia with the correct abstractions to provide seamless
+composability with scientific computing softwares. Lux uses pure functions to provide a
+compiler and automatic differentiation friendly interface without compromising on the performance.
+
+# Statement of Need
+
+Julia already has quite a few well established Neural Network Frameworks –
+Flux [@innes2018fashionable] and KNet [@yuret2016knet]. However, similar to Pytorch,
+Tensorflow, etc. these frameworks were designed for typical Deep Learining workflows and
+Scientific Computing workflows had to be tailored to fit into these frameworks.
+
+Having to rewrite these workflows, which are often highly optimized, is a major barrier for
+research in this domain.
+
+## Switching Automatic Differentiation Frameworks
+
+## Support for CPU, NVIDIA GPUs and AMD GPUs
+
+## Composability with Scientific Computing Softwares
+
+In this section, we will go over a couple of examples to show how Lux.jl can be used with
+other scientific computing softwares. Lux.jl has an extensive
+[manual](https://lux.csail.mit.edu/dev/manual/interface),
+[tutorials](https://lux.csail.mit.edu/dev/tutorials/), and
+[API Reference](https://lux.csail.mit.edu/dev/api/), showcasing the composability in more
+details.
+
+### Neural Ordinary Differential Equations
+
+### Deep Equilibrium Models
+
+Deep Equilibrium Models [@bai2019deep; @pal2023continuous] are a class of neural networks
+where the output of the model is the steady state of a dynamical system defined by an
+internal neural network.
+
+## Ecosystem
+
+# Limitations
+
+Lux.jl is still in its early days of development and has the following known limitations:
+
+* Training Small Neural Networks on CPUs are not optimized yet. For small networks,
+  [SimpleChains.jl](https://github.com/PumasAI/SimpleChains.jl) [@simplechains] is the fastest option!
+* Nested Automatic Differentiation is current not well supported. We hope to fix this soon,
+  with a migration to Enzyme Automatic Differentiation Framework [@enzyme:2020; @enzyme:2021].
+
+# References