[commit ffac420, pquery, portageq] Error messages from auto-complete

[blackteahamburger]

ble/widget/display-shell-version:
GNU bash, version 5.2.32(1)-release (x86_64-pc-linux-gnu) [Gentoo Linux]
ble.sh, version 0.4.0-devel4+70b89e5e (noarch) [git 2.46.0, GNU Make 4.4.1, GNU Awk 5.3.0, API 4.0]
bash-completion, version 2.14.0 (hash:480ffcc6a751e55621ec526eb5dea7a0d86d9e72, 17877 bytes) (noarch)
locale: LANG=C
terminal: TERM=xterm-256color wcwidth=auto-auto/15.1-2+ri, konsole:220380 (1;115;0)
options: +extglob +histappend -hostcomplete +inherit_errexit +no_empty_cmd_completion

Input something=, some error messages will be shown, e.g. (input LANG=):

$ LANG=bash: comp_words: bad array subscript
bash: comp_words: bad array subscript
bash: comp_words: bad array subscript

And another strange bug with pquery (See https://pkgcore.github.io/pkgcore/man/pquery.html#pquery) (maybe there's more commands with the same problem)

Input pquery something, some error messages will be shown, e.g. (input pquery s):

$ pquery sfatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
s

But no error message will be shown if the current working directory is a git repository.

[akinomyoga]

Input something=, some error messages will be shown, e.g. (input LANG=):

Thank you for the report. Confirmed. This is a regression introduced in commit ffac420.

---

And another strange bug with pquery (See https://pkgcore.github.io/pkgcore/man/pquery.html#pquery) (maybe there's more commands with the same problem)

$ pquery sfatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
s

This must be caused in the programmable completions called from auto-complete.

• Q1: Does the problem persist after you set bleopt complete_auto_complete=?
• Q2: Does the problem happen in the detached state of ble.sh on attempting the TAB completion?

$ ble-detach
[ble: detached]
Please run `stty sane' to recover the correct TTY state.
$  stty sane;[RET]
$ pquery s[TAB] <-- Does the problem reproduce here?

[akinomyoga]

Input something=, some error messages will be shown, e.g. (input LANG=):

Thank you for the report. Confirmed. This is a regression introduced in commit ffac420.

I pushed a fix 9270b52 for this problem.

[akinomyoga]

$ pquery sfatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
s

This is an issue in the upstream. This line is always executed without redirecting stderr, which is the problem.

[blackteahamburger]

• Q1: Does the problem persist after you set bleopt complete_auto_complete=?

No. Error msgs disappeared.

• Q2: Does the problem happen in the detached state of ble.sh on attempting the TAB completion?

Yes with the same error msg.

[akinomyoga]

Thank you, so it is not the issue with ble.sh. It's an upstream bug in pkgcore, which provides the pquery completion.

[blackteahamburger]

Thank you for finding the source of the bug and the PR!

[blackteahamburger]

There is one more command with error messages from auto-complete.

Input portageq envvar something (see here for bashcomp), some error messages will be shown, e.g.:

$ portageq envvar s!!! "--ask" should only be used in a terminal. Exiting.
!!! "--ask" should only be used in a terminal. Exiting.
s

I set EMERGE_DEFAULT_OPTS="--ask" in /etc/portage/make.conf, and removing it causes error messages to disappear.

Related: https://forums.gentoo.org/viewtopic-t-1068540-start-0.html

---

• Q1: Does the problem persist after you set bleopt complete_auto_complete=?

No. Error msgs disappeared.

• Q2: Does the problem happen in the detached state of ble.sh on attempting the TAB completion?

No.

$ portageq envvar[TAB]
Display all 169 possibilities? (y or n)[y]
ABI                                      CHOST_x86                                GSETTINGS_BACKEND                        OFFICE_IMPLEMENTATION                    POSTGRES_TARGETS
ABI_X86                                  CLEAN_DELAY                              GUILE_SINGLE_TARGET                      PATH                                     PROFILE_ONLY_VARIABLES
ACCEPT_KEYWORDS                          COLLECTD_PLUGINS                         GUILE_TARGETS                            PHP_TARGETS                              PYTHONDONTWRITEBYTECODE
ACCEPT_LICENSE                           COLLISION_IGNORE                         INFOPATH                                 PKGDIR                                   PYTHON_SINGLE_TARGET
ACCEPT_PROPERTIES                        CONFIG_PROTECT                           INPUT_DEVICES                            PORTAGE_ARCHLIST                         PYTHON_TARGETS
ACCEPT_RESTRICT                          CONFIG_PROTECT_MASK                      IUSE_IMPLICIT                            PORTAGE_BIN_PATH                         RESUMECOMMAND
...

I believe this is an inner bug because there's no related bug report in Gentoo's Bugzilla.

[akinomyoga]

I believe this is an inner bug because there's no related bug report in Gentoo's Bugzilla.

This is a compatibility issue, or just the completion setting is "not yet supported" by ble.sh. Since Bash doesn't offer the feature of autosuggestions, existing completion settings are designed to be used with the TAB completion. They are not designed to be used by autosuggestions. If you say this is a bug, the entire auto-complete feature based on the existing completion settings is a bug, but I do not want to remove the auto-complete feature based on the completion settings. It is impractical to prepare the dedicated auto-completion settings for all the commands in the wild by ourselves, so it is still beneficial to use the existing completion settings.

The present case is related to something that ble.sh disables for the completion settings. Some completion settings, on the user's attempt at the TAB completion, try to open a TUI window and wait for the user's input from TTY/PTY on stdin. Although it seems like hijacking of the TAB completion of Readline, Bash technically doesn't prohibit or discourage such a completion setting. If such a completion setting is naively called from the auto-complete feature, it would open the TUI window on every keystroke of the user and randomly steals the user's input. This would make the shell almost unusable. At worst, such a completion setting called in an abnormal setup by ble.sh would block the session, and you will lose control of the session. It is almost impossible for ble.sh to robustly judge whether a given completion setting would cause a problem in auto-complete without prior knowledge, so it is not possible to block such a completion setting before calling it. Instead, ble.sh tries to call the completion setting with stdin redirected to /dev/null, expecting such a completion setting would immediately fail. This probably works fine, but this is merely an expectation. The existence of completion settings that do not work with ble.sh's auto-complete is anticipated, and we are going to patch such completion settings one by one to officially "support" them. That is my choice. There is no better way other than disabling auto-complete based on the completion settings. Fortunately, most of the completion settings do not try to access TTY/PTY, so it seems we only need to patch finite number of completion settings.

It should also be noted that even for the TAB completion, other limitations exist in the custom terminal state set up by ble.sh and also in the signal handling while calling the completion settings. For this reason, ble.sh actually disables stdin even for the TAB completion by default. ble.sh wants to limit the access to the TTY to the completion settings patched by ble.sh. This is also related to the special setup needed for fzf's completion settings notified in ble.sh's README. The fzf completion is a typical completion that opens a TUI window.

[akinomyoga]

So I only see env -i PATH="${PATH}" emerge -v --info in the completion settings, where I don't see the option --ask or -a. Does that try to access TTY or attempt to read from stdin? Does running the following command in the command line produce the same message?

$ env -i PATH="${PATH}" emerge -v --info </dev/null

[akinomyoga]

Neither option -v nor --info seems to imply --ask, but it seems possible to globally configure the default options in /etc/portage/make.conf. Do you have --ask in EMERGE_DEFAULT_OPTS?

[blackteahamburger]

Neither option -v nor --info seems to imply --ask, but it seems possible to globally configure the default options in /etc/portage/make.conf. Do you have --ask in EMERGE_DEFAULT_OPTS?

I set EMERGE_DEFAULT_OPTS="--ask" in /etc/portage/make.conf, and removing it causes error messages to disappear.

I believe it's quite common among Gentoo users because the default behavior of emerge is that it does not ask anything and run by itself. It didn't cause any problems before.

[akinomyoga]

I set EMERGE_DEFAULT_OPTS="--ask" in /etc/portage/make.conf, and removing it causes error messages to disappear.

That means that --ask is actually not the flag requested by the completion setting. The completion setting can work without the --ask flag and is designed so by default.

• A question is whether the call of emerge -v --info, with --ask supplied by the default setting, has a chance to ask a user to respond. Option --info seems to print just information, and I guess it wouldn't do any destructive operation, so the user's confirmation about whether it should print the information is unlikely to be needed/useful.
  • If the answer is no, we may ask if they can specify --ask=n or redirect stderr for the call of emerge -v --info.
  • If the answer is yes, the next question is whether it would be expected and useful in the middle of the completion. I don't see any use case. If --info would try to do destructive operations, the answer to the confirmation should always be "no" in the middle of completions, so the confirmation could be skipped.
    • If the answer is no, --ask=n should actually be specified to the call of emergy -v --info.
    • If the answer is yes, what is that?

I believe it's quite common among Gentoo users because the default behavior
of emerge is that it does not ask anything and run by itself.

Yeah, and in fact, that is suggested by emerge - Gentoo Wiki. However, I see some inconsistency in the design of the behavior of --ask and the description there.

• First, I think we can assume that emerge command is intended to be able to be used also in scripts (where the terminal is not necessarily available) with proper command-line options¹.
• Besides, the Gentoo wiki suggests specifying --ask to EMERGE_DEFAULT_OPTS, but this would cause all the scripts using emerge print the error messages when executed in an environment without the terminal.

These two points appear to conflict with each other by design. We cannot safely use the emergy command in scripts. In particular, it would be a problem that --ask outputs the error message even when the specific usage of emerge in a script never requires confirmation.

• One way to resolve this issue may be to change the behavior of --ask so that it prints the error message only when it actually triggers asking the user's confirmation. Then, a command that just prints information (such as emergy -v --info wouldn't print the error message).
• Another way could be to change emergy to ignore --ask when stdin is not connected to the terminal and --ask is specified through EMERGE_DEFAULT_OPTS since --ask in EMERGE_DEFAULT_OPTS must be intended for the interactive use by definition. However, this becomes a problem when the user redirects stdin to a different stream in an interactive session by mistake. The operation will be performed without asking the confirmation.
• Or another way is to update the script side. All the scripts needs to specify --ask=n to every call of emerge regardless of whether that specific call of emerge would cause the confirmation. The Gentoo wiki page actually states "Default options can be overridden on the command line, for example --ask=n.", though I'm not sure why it limits the situation as "on the command line".

I wondered if there weren't any problems with the --ask behavior. A quick search finds many problems caused by this behavior:

• Gentoo Forums :: View topic - [SOLVED] Syncing portage with script while having --ask
• Gentoo Forums :: View topic - --ignore-default-opts not working???
• Gentoo Forums :: View topic - [solved] Overriding EMERGE_DEFAULT_OPTS while using eix-sync
• Gentoo Forums :: View topic - [SOLVED] MAKEOPTS and EMERGE_DEFAULT_OPTS
• [install.sh] override --ask in EMERGE_DEFAULT_OPTS for gentoo install · Issue #4354 · tailscale/tailscale
• 548814 – sys-apps/portage: suggestion to make --ask behaviour depend on os.isatty()

This must be a design issue of --ask when specified in EMERGE_DEFAULT_OPTS. In the first link above, specifying --ask=n or --ignore-default-opts at the script side is suggested as a solution. In the second link, no one seems to have replied, but the OP tried to fix the problem by --ignore-default-opts, which didn't work.

It didn't cause any problems before.

That is just because your situation has changed.

Footnotes

1. If it's not intended to be used in scripts at all and is required to be run only in the interactive shell sessions, it should always print the error message whenever it is called in scripts without the terminal. However, the emergy command prints the error message only when --ask is printed. In addition, the completion setting is technically not an interactive command, yet
   emerge -v --info is called. So, the use in scripts is also within a scope of some emerge calls. ↩

[blackteahamburger]

• A question is whether the call of emerge -v --info, with --ask supplied by the default setting, has a chance to ask a user to respond. Option --info seems to print just information, and I guess it wouldn't do any destructive operation, so the user's confirmation about whether it should print the information is unlikely to be needed/useful.

  • If the answer is no, we may ask if they can specify --ask=n or redirect stderr for the call of emerge -v --info.

  • If the answer is yes, the next question is whether it would be expected and useful in the middle of the completion. I don't see any use case. If --info would try to do destructive operations, the answer to the confirmation should always be "no" in the middle of completions, so the confirmation could be skipped.

    • If the answer is no, --ask=n should actually be specified to the call of emergy -v --info.
    • If the answer is yes, what is that?

It simply print the information, nothing else is done, and without any confirmation even with --ask.

In the second link, no one seems to have replied, but the OP tried to fix the problem by --ignore-default-opts, which didn't work.

I don't know what happened 18 years ago but --ignore-default-opts now works well on emerge --sync.

[akinomyoga]

• A question is whether the call of emerge -v --info, with --ask supplied by the default setting, has a chance to ask a user to respond. Option --info seems to print just information, and I guess it wouldn't do any destructive operation, so the user's confirmation about whether it should print the information is unlikely to be needed/useful.

  • If the answer is no, we may ask if they can specify --ask=n or redirect stderr for the call of emerge -v --info.

It simply print the information, nothing else is done, and without any confirmation even with --ask.

Thank you for the information. Then, I would probably ask them if they can include --ask=n in the call of emerge -v --info.

In the second link, no one seems to have replied, but the OP tried to fix the problem by --ignore-default-opts, which didn't work.

I don't know what happened 18 years ago but --ignore-default-opts now works well on emerge --sync.

Thank you also for this information.

Just to confirm the behavior, what are the results of the following commands (with --ask specified to EMERGE_DEFAULT_OPTS?

$ env -i PATH="${PATH}" emerge -v --info < /dev/null
$ env -i PATH="${PATH}" emerge -v --info --ask=n < /dev/null
$ env -i PATH="${PATH}" emerge -v --info --ignore-default-opts < /dev/null

Maybe specifying --ignore-default-opts in the completion setting should be safer than just specifying --ask=n because there might be other options that can interfere with it. Do you think specifying --ignore-default-opts would make sense? Or is there a possibility that it affects the set of completions for envvar in an unexpected way? Sorry for asking many things, but I'm not familiar with emerge because I'm not a user of Gentoo.

[blackteahamburger]

$ env -i PATH="${PATH}" emerge -v --info < /dev/null
!!! "--ask" should only be used in a terminal. Exiting.
$ env -i PATH="${PATH}" emerge -v --info --ask=n < /dev/null
...
ABI="amd64"
ABI_X86="64"
ACCEPT_KEYWORDS="amd64 ~amd64"
ACCEPT_LICENSE="*"
ACCEPT_PROPERTIES="*"
ACCEPT_RESTRICT="*"
ADA_TARGET="gcc_12"
APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_anon authn_dbm authn_file authz_dbm authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir env expires ext_filter file_cache filter headers include info log_config logio mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias"
ARCH="amd64"
AUTOCLEAN="no"
BINPKG_COMPRESS="zstd"
BINPKG_FORMAT="xpak"
BINPKG_GPG_SIGNING_BASE_COMMAND="/usr/bin/flock /run/lock/portage-binpkg-gpg.lock /usr/bin/gpg --sign --armor [PORTAGE_CONFIG]"
BINPKG_GPG_SIGNING_DIGEST="SHA512"
BINPKG_GPG_VERIFY_BASE_COMMAND="/usr/bin/gpg --verify --batch --no-tty --no-auto-check-trustdb --status-fd 2 [PORTAGE_CONFIG] [SIGNATURE]"
BINPKG_GPG_VERIFY_GPG_HOME="/etc/portage/gnupg"
BOOTSTRAP_USE="unicode pkg-config split-usr xml python_targets_python3_12 python_single_target_python3_12 gil multilib zstd cet systemd sysv-utils udev"
BROOT=""
CALLIGRA_FEATURES="karbon sheets words"
CAMERAS=""
CARGO_PROFILE_RELEASE_CODEGEN_UNITS="1"
CARGO_PROFILE_RELEASE_LTO="fat"
CARGO_PROFILE_RELEASE_OPT_LEVEL="3"
CARGO_PROFILE_RELEASE_PANIC="abort"
CARGO_PROFILE_RELEASE_STRIP="symbols"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=auto -fisolate-erroneous-paths-attribute -pipe"
CFLAGS_amd64="-m64"
CFLAGS_x32="-mx32"
CFLAGS_x86="-m32 -mfpmath=sse"
CHOST="x86_64-pc-linux-gnu"
CHOST_amd64="x86_64-pc-linux-gnu"
CHOST_x32="x86_64-pc-linux-gnux32"
CHOST_x86="i686-pc-linux-gnu"
CLEAN_DELAY="5"
COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog"
COLLISION_IGNORE="/boot/dtbs/* /lib/modules/*"
CONFIG_PROTECT="/etc /usr/lib64/libreoffice/program/sofficerc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d"
CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt rdrand sse sse2 sse3 sse4_1 sse4_2 ssse3"
CXXFLAGS="-march=native -O3 -fgraphite-identity -floop-nest-optimize -fdevirtualize-at-ltrans -fipa-pta -fno-semantic-interposition -flto=auto -fisolate-erroneous-paths-attribute -pipe -fforce-emit-vtables -fstrict-vtable-pointers -fwhole-program-vtables"
DEFAULT_ABI="amd64"
DISTDIR="/var/cache/distfiles"
EDITOR="nano"
ELIBC="glibc"
EMERGE_DEFAULT_OPTS="-av" # note here
...
$ env -i PATH="${PATH}" emerge -v --info --ignore-default-opts < /dev/null
... # exactly the same output as env -i PATH="${PATH}" emerge -v --info --ask=n < /dev/null
EMERGE_DEFAULT_OPTS="-av" # same here
...

So --ignore-default-opts is good.

[akinomyoga]

According to Manpage of EMERGE, there seems to be an option --prefix that affects the environment variable EPREFIX. I'm afraid that it might affect the list of envvars when --prefix is specified in the default options. What are the results of the following commands (/tmp can be adjusted if necessary)?

$ env -i PATH="${PATH}" emerge -v --info               | grep -o '^EPREFIX'
$ env -i PATH="${PATH}" emerge -v --info --prefix=/tmp | grep -o '^EPREFIX'

[blackteahamburger]

It really differs:

$ env -i PATH="${PATH}" emerge -v --info
...
EPREFIX=""
EROOT="/"
ESYSROOT="/"
...

$ env -i PATH="${PATH}" emerge -v --info --prefix=/tmp
...
EPREFIX="/tmp"
EROOT="/tmp/"
ESYSROOT="/tmp/"
...

[akinomyoga]

Thank you. So EPREFIX etc. exist even if --prefix is not specified. The differences in the values don't matter for the completion setting of portageq because any strings after = are anyway stripped on this line. Then, I think we don't have to care about this point for --ignore-default-opts.