ImageCore 1.0 -- the scope of ImageCore and ImageBase

[johnnychen94]

We used to have one single package ImageCore that keeps all the common small functions so that our other toolboxes can reuse them. But it's a growing need to also keep non-trivial functions somewhere without adding more loading latency to ImageCore. -- I constantly see people blaming ImageCore too slow to load. This made me create the ImageBase package.

The next question (also asked by many people) is: what are the scopes of ImageCore and of ImageBase? Or in other words, for function f, should it be kept in ImageCore or ImageBase?

I'm proposing to draw a clean line here: if the function is used to build the image concept and/or convert from different concepts, it's part of ImageCore. Otherwise, it goes to ImageBase if it's expected to be used widely.

---

Generally speaking, JuliaImages takes AbstractArray{<:Colorant} as an image and this is our first-class abstraction -- everything else is built around it. But there comes down to many situations:

• indexed image -- For this, we have IndirectArray from IndirectArrays
• sparse point cloud -- For this there is dictionary-based array type SparseArray from SparseArrayKit
• an image pyramid or image overlay?
• image with physical units and custom axes meanings -- AxisArrays?
• image with metadata -- ImageMetadata?
• on-disk mmaped-image? MultifileArrays?

There are also memory layout differences (#180): RR...RGG...GBB...B SoA (struct of arrays) vs RGBRGB...RGB AoS (array of structs). For this StructArray is a very nice abstraction.

There are also JuliaImages vs other frameworks differences. For instance, people keep wondering how to convert Flux's image batch from/to JuliaImage's representation (I've even written a page explaining this FluxML/Flux.jl#1705). We also have colorview/channelview to support conversion from/to raw numeric array.

Taking these into consideration, there are mainly three types of functions(structs) included in ImageCore:

• the core types to build the image concept. For instance, IndirectArrays, StructArray.
• the conversion functions between different image concept and layouts. For instance, from dense image to indexed image, colorview, channelview, MappedArrays
• the trait types that extract some properties of the given image. For instance, coords_spatial that extracts the non-time dimensions

If a function is not expected to be used to support the overall image concepts, then it's part of the ImageBase. For instance, MosaicViews.mosaic is a visualization tool and it adds nothing to our image concept, thus mosaic should be part of ImageBase. (or maybe even ImageShow). But MosaicViews.promote_wrapped_type provides a good helper to unify different colorant types and I would expect this to be part of Colors or ColorTypes or glued using Requires.

---

About latency, ImageCore will be unavoidably become bigger, but that's something I expect us to live with for quite a long time (but I would expect it to be no more than 1.5s for a decent CPU). We could use Requires to make them optional, or we can introduce ImageCoreIndexedImage to glue Colors with IndirectArrays. I'm in favor of Requires because I believe fist-class conditional package loading is the right way, and if we want to solve the unnecessary loading latency issue, we should try to solve it at the language level instead of spliting more atomic packages.

---

How far are we from 1.0 stage?

I'm not expecting 1.0 to be a stage that everything I mentioned above is done. Overall, I would expect 1.0 to be a stage that we've cleaned up the legacy dead codes, and designed well to leave some space for future additional features (the protocols and dispatches).

Because we reexport everything in Colors and FixedPointNumbers, every breaking release of them triggers a breaking release of ImageCore. ImageCore 1.0 won't be possible unless Colors and FixedPointNumbers reached their 1.0 stage, but we can get a "stable" release before the 1.0 stage to ease the migration (like the Julia 0.7).

[timholy]

This is very much how I think about the split between the two. The other role for ImageCore is in defining traits that help with writing generic algorithms (pixelspacing, nimages, etc.). And your thoughts about what remains are again in agreement with my own.

A few additional thoughts about heading for 1.0 (which I think is not far off now):

• I want to replace our use of AxisArrays with NamedDims & AxisKeys (probably)
• We probably need a careful pass through the various traits and make sure we still like the definitions.