Refactor picker/menu logic into an options manager which can be fed w…

…ith multiple different item sources (async, sync, refetch)

This PR unifies the functionality of the `Picker` and the `Menu` in the struct `OptionsManager` (i.e. matching/filtering, sorting, cursor management etc.), and extends the logic of `score` by being able to retain the cursor position (which is useful, if new entries are added async).

The main reason for this PR though is to have multiple option `ItemSource`s, which can be async and which may be refetched (see #5055) to provide the options for the `OptionsManager`. It currently allows 3 different input sources: synchronous data, asynchronous data, and asynchronous functions which provide the data, which are reexecuted after an `IdleTimeout` occured (similar as #5055).
This PR is or will be a requirement for #2507 (currently provided by #3082), which I will rebase onto this PR soon.

Noticeable effects right now:

* The `Menu` now has the more sophisticated logic of the picker (mostly better caching while changing the query)
* The `Menu` currently *doesn't* reset the cursor position, if `score()` is called (effectively different pattern query), I'm not sure  if we should keep this behavior, I guess it needs some testing and feedback. I have kept the original behavior of the Picker (everytime `score()` changes it resets the cursor)

This PR is an alternative to #3082, which I wasn't completely happy with (adding options async was/is kind of a hack to keep it simple) thus this PR will supersede it. It was originally motivated by #5055 and how to integrate that logic into #2507 (which is not done yet). The PR unfortunately grew a little bit more than I've anticipated, but in general I think it's the cleaner and more future proof way to be able to have async item sources, which may change the options overtime.
For example the same logic of #5055 could be simply implemented for the completion menu with different multiple completion sources, whether it makes sense or not (though also one of the motivations of this PR to have this possibility).
This should also cleanup and greatly (hopefully) simplify #2507.

To make reviewing easier (I hope) I provide a technical summary of this PR:

* `OptionsManager` contains all logic that is necessary to manage options (d'oh), that means:
    * Filtering/matching of options with `score()` which takes a few more parameters now:
        * *Optional* pattern to be able to recalculate the score without providing a pattern (necessary for async addition of the options)
        * Boolean flag to retain the cursor, since it will likely be annoying, if an item source takes so long that the user is already selecting options of a different item source and it gets resetted because score has to be called again for matches that respect the new option source).
        * force_recalculation, kind of an optimization, if nothing external has changed (except new async options)
    * The `OptionsManager` is fed by `ItemSource`s
        * Fetching of items from the item source is started by the creation of the `OptionsManager` (`create_from_item_sources`), and as soon one item source provides some items a construction callback is executed (see below)
        * Async fetching is done by looping over `FuturesUnordered` async requests and adding the options via an `mpsc`, I wanted to avoid any `Mutex` or something alike to keep the rest of the logic as is (and likely to be more performant as well). Thus instead the editor `redraw_handle` is notified when new options are available and in the render functions new options are polled (via `poll_for_new_options`). Although not optimal, it was IMHO the best tradeoff (I tried different solutions, and came to that conclusion).
        * Options are stored with an additional index to the item source to be able to easily change the options on the fly, it provides some iterator wrapper functions to access the options/matches outside as usual.
    * When using the `OptionManager` with async sources, I have decided (to mostly mirror current logic) to use a "constructor callback" in the `create_from_item_sources` for creating e.g. the actual `Menu` or `Picker` when at least one item is available (this callback is only then executed), otherwise an optional other callback can show e.g. editor status messages, when there was no item source provided any data.
    * The `(File-)Picker` and `Menu` now takes the `OptionsManager` instead of the items or editor data (this is now provided in the `ItemSource`)
    * For sync data it's also possible to directly create the `OptionsManager` without a callback using `create_from_items` but only with this function.
* I've imported the `CompletionItem` from #2507 to provide additional information where the item came from (which language server id and offset encoding) when multiple language servers are used as Item source, it's currently an enum to avoid unnecessary refactoring if other completion sources are added (such as #2608 or #1015), though it's not really relevant for this PR currently (as the doc has just one language server anyway), but I think it may be easier to review it here separated from #2507 and it's also the (only) dependency of #2507 for #2608
* The `DynamicPicker` is removed as this behavior can now be modeled with a `ItemSource::AsyncRefetchOnIdleTimeoutWithPattern` which is done for the `workspace_symbol_picker`

helix-lsp/src/lib.rs

@@ -44,7 +44,7 @@ pub enum Error {
     Other(#[from] anyhow::Error),
 }
 
-#[derive(Clone, Copy, Debug, Default)]
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
 pub enum OffsetEncoding {
     /// UTF-8 code units aka bytes
     Utf8,

helix-term/src/application.rs

@@ -348,11 +348,11 @@ impl Application {
                     self.handle_signals(signal).await;
                 }
                 Some(callback) = self.jobs.futures.next() => {
-                    self.jobs.handle_callback(&mut self.editor, &mut self.compositor, callback);
+                    self.jobs.handle_callback(&mut self.editor, &mut self.compositor, &self.jobs, callback);
                     self.render().await;
                 }
                 Some(callback) = self.jobs.wait_futures.next() => {
-                    self.jobs.handle_callback(&mut self.editor, &mut self.compositor, callback);
+                    self.jobs.handle_callback(&mut self.editor, &mut self.compositor, &self.jobs, callback);
                     self.render().await;
                 }
                 event = self.editor.wait_event() => {

helix-term/src/commands.rs

@@ -52,11 +52,16 @@ use crate::{
     filter_picker_entry,
     job::Callback,
     keymap::ReverseKeymap,
-    ui::{self, overlay::overlayed, FilePicker, Picker, Popup, Prompt, PromptEvent},
+    ui::{
+        self,
+        menu::{ItemSource, OptionsManager},
+        overlay::overlayed,
+        FilePicker, Picker, Popup, Prompt, PromptEvent,
+    },
 };
 
 use crate::job::{self, Jobs};
-use futures_util::StreamExt;
+use futures_util::{FutureExt, StreamExt};
 use std::{collections::HashMap, fmt, future::Future};
 use std::{collections::HashSet, num::NonZeroUsize};
 
@@ -2097,9 +2102,9 @@ fn global_search(cx: &mut Context) {
                     return;
                 }
 
+                let option_manager = OptionsManager::create_from_items(all_matches, current_path);
                 let picker = FilePicker::new(
-                    all_matches,
-                    current_path,
+                    option_manager,
                     move |cx, FileResult { path, line_num }, action| {
                         match cx.editor.open(path, action) {
                             Ok(_) => {}
@@ -2473,13 +2478,16 @@ fn buffer_picker(cx: &mut Context) {
         is_current: doc.id() == current,
     };
 
-    let picker = FilePicker::new(
+    let option_manager = OptionsManager::create_from_items(
         cx.editor
             .documents
             .values()
             .map(|doc| new_meta(doc))
             .collect(),
         (),
+    );
+    let picker = FilePicker::new(
+        option_manager,
         |cx, meta, action| {
             cx.editor.switch(meta.id, action);
         },
@@ -2551,7 +2559,7 @@ fn jumplist_picker(cx: &mut Context) {
         }
     };
 
-    let picker = FilePicker::new(
+    let option_manager = OptionsManager::create_from_items(
         cx.editor
             .tree
             .views()
@@ -2562,6 +2570,9 @@ fn jumplist_picker(cx: &mut Context) {
             })
             .collect(),
         (),
+    );
+    let picker = FilePicker::new(
+        option_manager,
         |cx, meta, action| {
             cx.editor.switch(meta.id, action);
             let config = cx.editor.config();
@@ -2623,7 +2634,8 @@ pub fn command_palette(cx: &mut Context) {
                 }
             }));
 
-            let picker = Picker::new(commands, keymap, move |cx, command, _action| {
+            let option_manager = OptionsManager::create_from_items(commands, keymap);
+            let picker = Picker::new(option_manager, move |cx, command, _action| {
                 let mut ctx = Context {
                     register: None,
                     count: std::num::NonZeroUsize::new(1),
@@ -4169,6 +4181,7 @@ pub fn completion(cx: &mut Context) {
         None => return,
     };
 
+    let language_server_id = language_server.id();
     let offset_encoding = language_server.offset_encoding();
     let text = doc.text().slice(..);
     let cursor = doc.selection(view.id).primary().cursor(text);
@@ -4191,39 +4204,52 @@ pub fn completion(cx: &mut Context) {
     let offset = iter.take_while(|ch| chars::char_is_word(*ch)).count();
     let start_offset = cursor.saturating_sub(offset);
 
-    cx.callback(
-        future,
-        move |editor, compositor, response: Option<lsp::CompletionResponse>| {
-            if editor.mode != Mode::Insert {
-                // we're not in insert mode anymore
-                return;
-            }
+    let completion_item_source = ItemSource::from_async_data(
+        async move {
+            let json = future.await?;
+            let response: helix_lsp::lsp::CompletionResponse = serde_json::from_value(json)?;
 
-            let items = match response {
-                Some(lsp::CompletionResponse::Array(items)) => items,
+            let mut items = match response {
+                lsp::CompletionResponse::Array(items) => items,
                 // TODO: do something with is_incomplete
-                Some(lsp::CompletionResponse::List(lsp::CompletionList {
+                lsp::CompletionResponse::List(helix_lsp::lsp::CompletionList {
                     is_incomplete: _is_incomplete,
                     items,
-                })) => items,
-                None => Vec::new(),
+                }) => items,
             };
 
-            if items.is_empty() {
-                // editor.set_error("No completion available");
+            // Sort completion items according to their preselect status (given by the LSP server)
+            items.sort_by_key(|item| !item.preselect.unwrap_or(false));
+            Ok(items
+                .into_iter()
+                .map(|item| ui::CompletionItem::LSP {
+                    item,
+                    language_server_id,
+                    offset_encoding,
+                })
+                .collect())
+        }
+        .boxed(),
+        (),
+    );
+
+    OptionsManager::create_from_item_sources(
+        vec![completion_item_source],
+        cx.editor,
+        cx.jobs,
+        move |editor, compositor, option_manager| {
+            if editor.mode != Mode::Insert {
+                // we're not in insert mode anymore
                 return;
             }
+
             let size = compositor.size();
             let ui = compositor.find::<ui::EditorView>().unwrap();
-            ui.set_completion(
-                editor,
-                items,
-                offset_encoding,
-                start_offset,
-                trigger_offset,
-                size,
-            );
+            ui.set_completion(editor, option_manager, start_offset, trigger_offset, size);
         },
+        Some(Box::new(|editor: &mut Editor| {
+            editor.set_error("No completion available")
+        })),
     );
 }
 

helix-term/src/commands/dap.rs

@@ -2,9 +2,15 @@ use super::{Context, Editor};
 use crate::{
     compositor::{self, Compositor},
     job::{Callback, Jobs},
-    ui::{self, overlay::overlayed, FilePicker, Picker, Popup, Prompt, PromptEvent, Text},
+    ui::{
+        self,
+        menu::{ItemSource, OptionsManager},
+        overlay::overlayed,
+        FilePicker, Picker, Popup, Prompt, PromptEvent, Text,
+    },
 };
 use dap::{StackFrame, Thread, ThreadStates};
+use futures_util::FutureExt;
 use helix_core::syntax::{DebugArgumentValue, DebugConfigCompletion, DebugTemplate};
 use helix_dap::{self as dap, Client};
 use helix_lsp::block_on;
@@ -61,21 +67,30 @@ fn thread_picker(
     let debugger = debugger!(cx.editor);
 
     let future = debugger.threads();
-    dap_callback(
+
+    let thread_states = debugger!(cx.editor).thread_states.clone();
+    let item_source = ItemSource::from_async_data(
+        async move {
+            let response: dap::requests::ThreadsResponse = serde_json::from_value(future.await?)?;
+
+            Ok(response.threads)
+        }
+        .boxed(),
+        thread_states,
+    );
+
+    OptionsManager::create_from_item_sources(
+        vec![item_source],
+        cx.editor,
         cx.jobs,
-        future,
-        move |editor, compositor, response: dap::requests::ThreadsResponse| {
-            let threads = response.threads;
-            if threads.len() == 1 {
-                callback_fn(editor, &threads[0]);
+        move |editor, compositor, option_manager| {
+            if option_manager.options_len() == 1 {
+                callback_fn(editor, option_manager.options().next().unwrap().0);
                 return;
             }
-            let debugger = debugger!(editor);
 
-            let thread_states = debugger.thread_states.clone();
             let picker = FilePicker::new(
-                threads,
-                thread_states,
+                option_manager,
                 move |cx, thread, _action| callback_fn(cx.editor, thread),
                 move |editor, thread| {
                     let frames = editor.debugger.as_ref()?.stack_frames.get(&thread.id)?;
@@ -90,6 +105,7 @@ fn thread_picker(
             );
             compositor.push(Box::new(picker));
         },
+        None,
     );
 }
 
@@ -270,9 +286,9 @@ pub fn dap_launch(cx: &mut Context) {
 
     let templates = config.templates.clone();
 
+    let option_manager = OptionsManager::create_from_items(templates, ());
     cx.push_layer(Box::new(overlayed(Picker::new(
-        templates,
-        (),
+        option_manager,
         |cx, template, _action| {
             let completions = template.completion.clone();
             let name = template.name.clone();
@@ -681,9 +697,9 @@ pub fn dap_switch_stack_frame(cx: &mut Context) {
 
     let frames = debugger.stack_frames[&thread_id].clone();
 
+    let option_manager = OptionsManager::create_from_items(frames, ());
     let picker = FilePicker::new(
-        frames,
-        (),
+        option_manager,
         move |cx, frame, _action| {
             let debugger = debugger!(cx.editor);
             // TODO: this should be simpler to find