Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Developing User Extensions through Dynamic Linking #3850

Open
scienceplease opened this issue Oct 17, 2024 · 0 comments
Open

Developing User Extensions through Dynamic Linking #3850

scienceplease opened this issue Oct 17, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@scienceplease
Copy link

scienceplease commented Oct 17, 2024

Currently, the process of developing Arkouda extensions (e.g. user defined functions) is roughly:

  1. Modify and recompile version of Arkouda containing extension
  2. Restart Arkouda server
  3. Reconnect Arkouda client
  4. Re-run client notebook to reach prior state
  5. Test functionality

Leveraging Chapel's existing dynamic linking facilities (chapel-lang/chapel#26024), Arkouda could support the following workflow:

  1. Modify and recompile standalone Chapel module containing extension
  2. Arkouda server automatically detects and loads library (by watching the build directory)
  3. Test functionality (no need to reconnect or re-run client notebook)

This would dramatically improve iteration time, getting the feedback loop closer to what Python users are accustomed to. With the existing level of support for dynamic linking, this could be used to implement scalar routines that are run in parallel by the Arkouda server. This should be faster than issuing individual operations and also remove some instances of creating temporary arrays to hold intermediate results.

As Chapel's dynamic linking facilities mature, more ambitious extensions to Arkouda like Arachne could potentially be developed and distributed as "plugins" instead of requiring a fork of the Arkouda codebase.

@scienceplease scienceplease added the enhancement New feature or request label Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant