Code

feat: implement virtual arrays#3364

Code

Open

ikrommyd wants to merge 101 commits intoscikit-hep:mainscikit-hep/awkward:mainfrom ikrommyd:virtual-arraysikrommyd/awkward:virtual-arraysCopy head branch name to clipboard

+7,430-147

Commits

Commits on Jan 23, 2025

• first try to implement virtual arrays at (least close to) the buffer level

  

  pfackeldey

  authored and

  ikrommyd

  committed
  8407680

  Copy full SHA for 8407680

• lint

  

  pfackeldey

  authored and

  ikrommyd

  committed
  938b883

  Copy full SHA for 938b883

• prepare virtual arrays for Iason (now really at the buffer level)

  

  pfackeldey

  authored and

  ikrommyd

  committed
  eccb9a8

  Copy full SHA for eccb9a8

• fix .T for virtual arrays

  

  pfackeldey

  authored and

  ikrommyd

  committed
  960f7c3

  Copy full SHA for 960f7c3

• minimal changes to get an example of a listoffsetarray with virtual buffers working in a notebook

  ikrommyd

  committed
  ecb883b

  Copy full SHA for ecb883b

• what if we start like this?

  ikrommyd

  committed
  40824c7

  Copy full SHA for 40824c7

• add repr

  ikrommyd

  committed
  7b2809c

  Copy full SHA for 7b2809c

• restore _data <-> data to original

  ikrommyd

  committed
  85d8a83

  Copy full SHA for 85d8a83

• an attempt at implementing some methods

  ikrommyd

  committed
  3b5ac6a

  Copy full SHA for 3b5ac6a

• style: pre-commit fixes

  

  pre-commit-ci[bot]

  authored and

  ikrommyd

  committed
  6b8cbc4

  Copy full SHA for 6b8cbc4

• tiny changes

  ikrommyd

  committed
  d0e4947

  Copy full SHA for d0e4947

• add ones_like and full_like to play around a bit

  ikrommyd

  committed
  a91dad0

  Copy full SHA for a91dad0

• no need to implement ufuncs separately. They are covered by apply_ufunc

  ikrommyd

  committed
  40841a2

  Copy full SHA for 40841a2

• add materialization in array_module.py

  ikrommyd

  committed
  72babd0

  Copy full SHA for 72babd0

• same repr for placeholders

  ikrommyd

  committed
  95e9e3c

  Copy full SHA for 95e9e3c

• adhere to ArrayLike interface

  ikrommyd

  committed
  6a25b1b

  Copy full SHA for 6a25b1b

• add TODO comment

  ikrommyd

  committed
  c1ff9ea

  Copy full SHA for c1ff9ea

• try kernels

  ikrommyd

  committed
  166fcb0

  Copy full SHA for 166fcb0

• never create new virtual arrays for now, let them materialize

  ikrommyd

  committed
  c5020d0

  Copy full SHA for c5020d0

• ruff fix

  ikrommyd

  committed
  7a8a074

  Copy full SHA for 7a8a074

Commits on Jan 25, 2025

• bring back creation of virtual arrays with a local cache

  ikrommyd

  committed
  b2bb788

  Copy full SHA for b2bb788

• forgot this

  ikrommyd

  committed
  6e41bda

  Copy full SHA for 6e41bda

• Merge branch 'main' into virtual-arrays

  ikrommyd

  committed
  a84d84e

  Copy full SHA for a84d84e

• maybe rapidjson submodule fix

  ikrommyd

  committed
  4137bf9

  Copy full SHA for 4137bf9

• adjust _get_item_at_placeholder for virtual arrays as well

  ikrommyd

  committed
  c773329

  Copy full SHA for c773329

Commits on Jan 26, 2025

• new _get_item_at_virtual function and different VirtualValue vs PlaceholderValue

  ikrommyd

  committed
  391fe17

  Copy full SHA for 391fe17

• remove some comments and docstrings that we decided to use in a different manner

  ikrommyd

  committed
  80f40ae

  Copy full SHA for 80f40ae

• let iterators of virtual buffers materialize

  ikrommyd

  committed
  93db3a4

  Copy full SHA for 93db3a4

Commits on Jan 27, 2025

• inherit from NDArrayOperatorsMixin as well

  ikrommyd

  committed
  4c36c7d

  Copy full SHA for 4c36c7d

Commits on Jan 31, 2025

• Merge branch 'main' into virtual-arrays

  ikrommyd

  committed
  7c59dba

  Copy full SHA for 7c59dba

Commits on Feb 5, 2025

• first attempt at cupy nplike for virtual arrays

  ikrommyd

  committed
  261b1ac

  Copy full SHA for 261b1ac

Commits on Feb 6, 2025

• we should not instantiate cupy for everyone :)

  ikrommyd

  committed
  01fa4dc

  Copy full SHA for 01fa4dc

Commits on Feb 7, 2025

• I wanna try like this first

  ikrommyd

  committed
  773e339

  Copy full SHA for 773e339

• Merge branch 'main' into virtual-arrays

  ikrommyd

  committed
  c66db68

  Copy full SHA for c66db68

• add preliminary implementations of ak.materialize and ak.dematerialize

  ikrommyd

  committed
  4e1275e

  Copy full SHA for 4e1275e

• duplicate import

  ikrommyd

  committed
  8c5f0d3

  Copy full SHA for 8c5f0d3

• whoops

  ikrommyd

  committed
  2efab45

  Copy full SHA for 2efab45

• delete dematerialize

  ikrommyd

  committed
  5f620c7

  Copy full SHA for 5f620c7

Commits on Feb 8, 2025

• make views return VirtualArray's only

  ikrommyd

  committed
  536a0d2

  Copy full SHA for 536a0d2

Commits on Feb 11, 2025

• numpy metadata are unused

  ikrommyd

  committed
  92aeb14

  Copy full SHA for 92aeb14

• we need .ctypes

  ikrommyd

  committed
  1964234

  Copy full SHA for 1964234

Commits on Feb 12, 2025

• better use classes instead of strings

  ikrommyd

  committed
  80130f2

  Copy full SHA for 80130f2

• materialize should keep the parameters in the new NumpyArray

  ikrommyd

  committed
  9bc1a63

  Copy full SHA for 9bc1a63

• Merge branch 'main' into virtual-arrays

  pfackeldey

  committed
  3fa6123

  Copy full SHA for 3fa6123

• add ak.from_virtual

  pfackeldey

  committed
  59829e0

  Copy full SHA for 59829e0

• remove extra length argument from from_virtual

  ikrommyd

  committed
  4e5f8a5

  Copy full SHA for 4e5f8a5

Commits on Feb 13, 2025

• use ak.materialize inside to_something calls

  ikrommyd

  committed
  d79bd42

  Copy full SHA for d79bd42

• docstring for materialize

  ikrommyd

  committed
  fed02cb

  Copy full SHA for fed02cb

• add comment to not forget

  ikrommyd

  committed
  ba4c666

  Copy full SHA for ba4c666

• Merge branch 'main' into virtual-arrays

  ikrommyd

  authored
  ddd1974

  Copy full SHA for ddd1974

• some changs to contiguousness and backends

  ikrommyd

  committed
  2b91312

  Copy full SHA for 2b91312

• forgot a breakpoint

  ikrommyd

  committed
  beeb033

  Copy full SHA for beeb033

Commits on Feb 14, 2025

• Do not use recursively_apply for ak.materialize. Write our own recursive function

  ikrommyd

  committed
  375094c

  Copy full SHA for 375094c

• Merge branch 'main' into virtual-arrays

  ikrommyd

  authored
  aebfcfc

  Copy full SHA for aebfcfc

• add is_materialized property to layouts

  ikrommyd

  committed
  c0b2810

  Copy full SHA for c0b2810

• is_materialized it should work with records to

  ikrommyd

  committed
  987d812

  Copy full SHA for 987d812

Commits on Feb 17, 2025

• some touches

  ikrommyd

  committed
  5cebd74

  Copy full SHA for 5cebd74

• ruff fix

  ikrommyd

  committed
  970af04

  Copy full SHA for 970af04

• update ak.from_virtual

  pfackeldey

  committed
  4301dbf

  Copy full SHA for 4301dbf

• style: pre-commit fixes

  pre-commit-ci[bot]

  committed
  e28ddf6

  Copy full SHA for e28ddf6

• allow unknown shape slices in virtual arrays

  ikrommyd

  committed
  028336f

  Copy full SHA for 028336f

Commits on Feb 18, 2025

• remove ak.from_virtual; make ak.from_buffers compatible with VirtualArrays

  pfackeldey

  committed
  fa44c59

  Copy full SHA for fa44c59

• style: pre-commit fixes

  pre-commit-ci[bot]

  committed
  4df11eb

  Copy full SHA for 4df11eb

• remove ak.from_virtual

  pfackeldey

  committed
  73aa6fb

  Copy full SHA for 73aa6fb

• Merge branch 'virtual-arrays' of github.com:ikrommyd/awkward into virtual-arrays

  pfackeldey

  committed
  5c648ef

  Copy full SHA for 5c648ef

Commits on Feb 20, 2025

• drop support for unknown_length

  ikrommyd

  committed
  a3d733d

  Copy full SHA for a3d733d

• style: pre-commit fixes

  pre-commit-ci[bot]

  committed
  c5a44f4

  Copy full SHA for c5a44f4

• make reprs of placeholder and virtual consistent

  ikrommyd

  committed
  da20cda

  Copy full SHA for da20cda

• better make those typeerros

  ikrommyd

  committed
  8911f91

  Copy full SHA for 8911f91

• make ak.from_buffers materialize offsets and then free the memory again immediately

  pfackeldey

  committed
  cde5b1b

  Copy full SHA for cde5b1b

• add tolist() to typetracer array

  pfackeldey

  committed
  4081597

  Copy full SHA for 4081597

• remove form_key from virtual arrays

  ikrommyd

  committed
  27f994b

  Copy full SHA for 27f994b

Commits on Feb 21, 2025

• address some comments

  ikrommyd

  committed
  9cd1268

  Copy full SHA for 9cd1268

• add is_any_materialized and is_all_materialized

  ikrommyd

  committed
  b4073a6

  Copy full SHA for b4073a6

• fix integer check

  ikrommyd

  committed
  9602f42

  Copy full SHA for 9602f42

• was this a bug all along? The .data are Virtual/Placeholder, not the masks/index

  ikrommyd

  committed
  be31ebb

  Copy full SHA for be31ebb

• pre-commit

  ikrommyd

  committed
  cbfd8c4

  Copy full SHA for cbfd8c4

• similar errors for both nplikes

  ikrommyd

  committed
  acb6196

  Copy full SHA for acb6196

• if it's virtual, the virtual.nplike.ndarray is going to go in there so it's fine

  ikrommyd

  committed
  add4f79

  Copy full SHA for add4f79

• pre-commit fixes

  ikrommyd

  committed
  6c2701b

  Copy full SHA for 6c2701b

• remove from_virtual from docs

  ikrommyd

  committed
  3785159

  Copy full SHA for 3785159

• recursive from_buffers for virtual arrays

  ikrommyd

  committed
  1133d97

  Copy full SHA for 1133d97

• raise error if non-fully materialized arraystry to get into an arraybuilder kernel

  ikrommyd

  committed
  7b5d44e

  Copy full SHA for 7b5d44e

Commits on Feb 25, 2025

• Merge branch 'main' into virtual-arrays

  ikrommyd

  committed
  d561389

  Copy full SHA for d561389

Commits on Feb 26, 2025

• add class VirtualArray tests

  ikrommyd

  committed
  c48651d

  Copy full SHA for c48651d

• specify dtypes in the test

  ikrommyd

  committed
  ce39720

  Copy full SHA for ce39720

• maybe this fixes windows tests?

  ikrommyd

  committed
  001143b

  Copy full SHA for 001143b

• array module tests

  ikrommyd

  committed
  1202da2

  Copy full SHA for 1202da2

• we should be able to convert virtual to typetracer

  ikrommyd

  committed
  7e5ce5a

  Copy full SHA for 7e5ce5a

Commits on Feb 28, 2025

• to_rdataframe should materialize

  ikrommyd

  committed
  c298c0b

  Copy full SHA for c298c0b

• allow changing dtype of virtual arrays in asarray

  ikrommyd

  committed
  01584c7

  Copy full SHA for 01584c7

• style: pre-commit fixes

  pre-commit-ci[bot]

  committed
  211443c

  Copy full SHA for 211443c

• mypy fix

  ikrommyd

  committed
  4a5996c

  Copy full SHA for 4a5996c

• .max() will not work for virtual arrays

  ikrommyd

  committed
  480e2d9

  Copy full SHA for 480e2d9

• numpyarray tests

  ikrommyd

  committed
  033fd32

  Copy full SHA for 033fd32

• listoffsetarray tests

  ikrommyd

  committed
  630a7ca

  Copy full SHA for 630a7ca

• listarray tests

  ikrommyd

  committed
  8f3e8d6

  Copy full SHA for 8f3e8d6

• pre-commit fixes

  ikrommyd

  committed
  c8a0ee3

  Copy full SHA for c8a0ee3

• recordarray tests

  ikrommyd

  committed
  3b6d73e

  Copy full SHA for 3b6d73e

• add copying

  ikrommyd

  committed
  2b91ae8

  Copy full SHA for 2b91ae8

• fix pylint?

  ikrommyd

  committed
  43c03d6

  Copy full SHA for 43c03d6