Skip to content

Platform Acceptance Test for Windows

Jump to bottom
Makara edited this page Apr 1, 2022 · 20 revisions

Keyman for Windows Acceptance Test Procedures

  1. These test procedures are to be run before moving from alpha to beta, or beta to stable, or before PRs are merged into stable branches.

  2. Results are annotated with results, use > at the start of a new line under the checkbox/Test to note the result.

User Testing

Gather Assets for Testing

  • Windows 7 (x86) - VM or physical okay
  • Windows 7 (x64) - VM or physical okay
  • Windows 8.1 (x64) - VM or physical okay
  • Windows 10 (x64) - VM or physical okay
  • Windows 11 (x64) - VM or physical okay
  • Keyboards for Caps Lock test sets - The test keyboard layouts are found in the keyman repo at app/windows/src/test/manual-tests/caps-lock-stores.

Setup Steps

  1. Uninstall previous versions of Keyman Desktop and Keyman Developer on the system
  2. Restart system, install updates
  3. Make sure all additional Windows languages are removed
  4. Make sure no debug flags are set in HKCU/Software/Keyman/Debug

SUITE_INSPECT: Visual Inspection of master/beta branch

  • TEST_CI_MASTER: Verify latest CI build of master is successful
  • TEST_CI_BETA: Verify latest CI build of beta is successful
  • TEST_VERIFY_HISTORY: Verify /HISTORY.md contains all the current changes

SUITE_BASELINE: Keyman for Windows Base Line Tests

  • GROUP_WIN7_x86
  • GROUP_WIN7_x64
  • GROUP_WIN8_1
  • GROUP_WIN10
  • GROUP_WIN11

Test cases

click to expand

  • TEST_INSTALL:

    • Install test build keyman-a.b.c.exe
    • Make sure Keyman is started and visible in the Notification area

  • TEST_KEYBOARD_INSTALLATION_REMOTE:

    • Start Keyman Configuration
    • Click Download Keyboard, select a keyboard from the download dialog and install it
    • If the package includes a readme file, verify that it appears in the install window.
    • If the package includes a welcome file, verify that it appears after installation.
    • Click OK to close Keyman Configuration
    • Start Notepad
    • Verify that the newly installed keyboard appears in the Keyman menu and can be selected

  • TEST_INSTALL_PKG_DISK:

    • Install a package from disk ##
    • Locate a .kmp file on disk and double-click to install it
    • If the package includes a readme file, verify that it appears in the install window.
    • If the package includes a welcome file, verify that it appears after installation.
    • Start Notepad
    • Verify that the keyboard appears in the Keyman menu

  • TEST_KEYBOARD_OUTPUT:

    • Start Notepad and select a Keyman keyboard (testing Legacy integration)
    • Verify that various test sequences are output correctly
    • Start Word and select a Keyman keyboard (testing TSF integration)
    • Verify that various test sequences are output correctly

  • TEST_ON_SCREEN_KEYBOARD:

    • Click Keyman icon in Notification area, select On Screen Keyboard
    • Verify that OSK displays correctly for selected Keyman keyboard
    • Verify that typing characters by clicking on OSK produces expected output

  • TEST_WHATS_NEW: Refer to the new changes in /HISTORY.md and verify functionality

    Note: The following tests are extra created for this release (15) based on the the review of /History.md

SUITE_CAPSLOCK:

  • GROUP_WIN7_x86
  • GROUP_WIN7_x64
  • GROUP_WIN8_1
  • GROUP_WIN10
  • GROUP_WIN11

Caps Lock

The test keyboard layouts are found in the keyman repo at app/windows/src/test/manual-tests/caps-lock-stores. There is a project file for the 3 keyboards used in this test. The project file can be used to build the keyboard packages required for the following tests.

The test cases below expect the usage of the capslock.kmp keyboard. That keyboard outputs pass or fail if following the test cases.

Prerequisites before each test

  • System keyboard layout is en-US
  • Install a keyboard that doesn't use any of the caps lock stores, e.g. capslock.kmp.
  • CapsLock is currently on
  • Currently active keyboard is the capslock.kmp keyboard

Test cases

click to expand

  • TEST_CAPSLOCK-1: uppercase with virtual key

    • press and release a

    Expected result:

    • pass. (with other keyboards uppercase A)

  • TEST_CAPSLOCK-2: lowercase with virtual key

    • press and hold 'Shift'
    • press and release b
    • release Shift

    Expected result:

    • pass. (with other keyboards lowercase b)

  • TEST_CAPSLOCK-3: capslock ignored for numbers

    • press and hold 'Shift'
    • press and release 3
    • release Shift

    Expected result:

    • pass. (with other keyboards #)

  • TEST_CAPSLOCK-4: uppercase

    • press and release c

    Expected result:

    • pass. (with other keyboards uppercase C)

  • TEST_CAPSLOCK-5: lowercase

    • press and hold 'Shift'
    • press and release d
    • release Shift

    Expected result:

    • pass. (with other keyboards lowercase d)

CapsAlwaysOff

For these tests, use a keyboard with the CapsAlwaysOff store set. We call this keyboard capsalwaysoff below.

Any keyboard with that store set will work; if you don't have one at hand you can use the caps_always_off.kmp keyboard. The caps_always_off.kmp keyboard will prevent switching caps lock on. As a sanity check to verify that Keyman is actually active, pressing the key a will output ncaps_little_a, and Shift+a will output ncaps_shift_A.

Note: When testing in a virtual machine, use an on-screen keyboard (in VirtualBox: Input/Keyboard/Soft Keyboard) and observe the caps lock indicator of the on-screen keyboard. Using the hardware keyboard might show side effects with caps lock.

Prerequisites before each test

  • Install a keyboard that has CapsAlwaysOff store set, e.g. caps_always_off.kmp.
  • CapsLock is currently off
  • Currently active keyboard is a non-Keyman keyboard

Test cases

click to expand

  • TEST_CAPSOFF-1: sanity check

    • switch to capsalwaysoff keyboard
    • press and release a

    Expected result:

    • output: ncaps_little_a

  • TEST_CAPSOFF-2: caps lock stays off

    • switch to capsalwaysoff keyboard
    • press and release CapsLock key
    • press and release a

    Expected result:

    • caps lock indicator stays turned off
    • output: ncaps_little_a

  • TEST_CAPSOFF-3: no caps lock while holding capslock key

    • switch to capsalwaysoff keyboard
    • press and hold CapsLock key
    • press and release a
    • release CapsLock key

    Expected result:

    • output: ncaps_little_a

  • TEST_CAPSOFF-4: no caps lock while holding capslock key

    • switch to capsalwaysoff keyboard
    • press and hold CapsLock key
    • press and hold Shift key
    • press and release a
    • release CapsLock and Shift keys

    Expected result:

    • output: ncaps_shift_A

  • TEST_CAPSOFF-5: switching turns off caps lock

    • turn on caps lock
    • switch to capsalwaysoff keyboard
    • press and release a

    Expected result:

    • caps lock indicator turned off
    • output: ncaps_little_a

SHIFT: CapsOnOnly/ShiftFreesCaps

For these tests, use a keyboard with the CapsOnOnly and ShiftFreesCaps stores set. We call this keyboard shift_frees_caps below.

Any keyboard with these stores set will work; if you don't have one at hand you can use the shift_frees_caps.kmp keyboard.

The shift_frees_caps.kmp keyboard will enable caps lock by pressing the CapsLock key, and will turn capslock off by pressing the Shift key. The keyboard outputs pass or fail if following the test cases.

Note: When testing in a virtual machine, use an on-screen keyboard (in VirtualBox: Input/Keyboard/Soft Keyboard) and observe the caps lock indicator of the on-screen keyboard. Using the hardware keyboard might show side effects with caps lock.

Prerequisites before each test

  • Install a keyboard that has the CapsOnOnly and ShiftFreesCaps stores set, e.g. shift_frees_caps.kmp.
  • CapsLock is currently off
  • Currently active keyboard is shift_frees_caps keyboard

Test cases

click to expand

  • TEST_CAPSONLY-1: no caps

    • press and release 1

    Expected result:

    • output: pass.

  • TEST_CAPSONLY-2: caps

    • press and release CapsLock
    • press and release 2

    Expected result:

    • caps lock indicator turned on
    • output: pass.

  • TEST_CAPSONLY-3: caps doesn't toggle

    • press and release CapsLock
    • press and release CapsLock
    • press and release 6

    Expected result:

    • caps lock indicator turned on
    • output: pass.

  • TEST_CAPSONLY-4: shift turns off

    • press and release CapsLock
    • press and hold Shift
    • press and release 3
    • release Shift

    Expected result:

    • caps lock indicator turned off
    • output: pass.

  • TEST_CAPSONLY-5: shift by itself turns off

    • press and release CapsLock
    • press and release Shift

    Expected result:

    • caps lock indicator turned off
    • (no output)

SUITE_TSF_APP:

Type in-app that uses the windows TSF (MS Word, Fieldworks windows search field on the taskbar).

To run these tests the EuroLatin (SIL) Keyboard from here.

  • GROUP_WIN7_x86
  • GROUP_WIN7_x64
  • GROUP_WIN8_1
  • GROUP_WIN10
  • GROUP_WIN11

Test cases: double processing

  • TEST_DOUBLE_PROCESSING_SEARCHBAR:

    1. Start Keyman
    2. Select the EuroLatin Keyboard
    3. Type in to the windows search bar a b c d Backspace
    4. Expected result is abc

    Search_bar

  • TEST_ DOUBLE_PROCESSING_NOTEPAD:

    1. Open Notepad
    2. Start Keyman
    3. Select the EuroLatin Keyboard
    4. Type a sequence of letters a b c d
    5. Expected result: abcd

SUITE_IMX_KEYBOARDS

Test IMX keyboards still work after core integration. see tests suite on #5936 and also #6187

This is the keyboard to be tested. imsample.zip If an IMTest keyboard is already installed you will need to uninstall it first, restart windows and then install the Imsample keyboard attached to this PR.

I have copied the tests from #5936 and made a few modifications.
The overall test we are aiming for is that in doing the tests below the keyboard does not cause a keyman or app crash.

  • GROUP_WIN7_x86
  • GROUP_WIN7_x64
  • GROUP_WIN8_1
  • GROUP_WIN10
  • GROUP_WIN11

Test cases

click to expand

IMSAMPLE_KEYBOARD_INLINE_MENU:

This keyboard uses the letters aeom to allow IMX input. This keyboard is a bit flakey being a demo and not polished. It also takes some setting up the following registry keys need to be added and set once installed. The location of the registry keys are at. Computer\HKEY_CURRENT_USER\SOFTWARE\Keyman\Keyman Engine\Active Keyboards\imsample and are of type REG_DWORD The keys are ShowIMWindow and ShowIMWindowAlways they both need to be set to 0.

In this mode IM window is not shown rather the app text is updated with a menu when either aeom is pressed. For example when type he the text will become h[1ɛ 2ɜ 3ə 4e 5ɘ] then you type the number 3 the text displayed should now be hə

  • TEST_IMSAMPLE_INPUT:

    • Install Imsample keyboard attached to this PR.
    • Select the keyboard its name is IMTest
    • Type some keys that are not the special keys e.g. t r y
      • Type the space key then a then type 1 to get the 1st option
    • Expected Result: try æ
    • continue to the next test

  • TEST_IMSAMPLE_INPUT_CONT: ... contd from above

    • Type 'm' and type 3 to get ### option
    • Expected Result: try æ###

  • TEST_IMSAMPLE_BACKSPACE: Using Notepad or Libreoffice

    • Type f
    • Type e
    • The screen should have f[1ɛ 2ɜ 3ə 4e 5ɘ]
    • Now press Backspace. the whole option menu including the square brackets [ ] should be deleted.
    • Expected result f

IMSAMPLE_KEYBOARD_IM_WINDOW:

This keyboard uses the letters aeom to allow IMX input. This time a IM window should display. image

The first time you use the keyboard with an application the registry keys will be created and by default set to 1.

If the IM window is not appearing you will need to verify the value of two registry keys. The location of the registry keys are at. Computer\HKEY_CURRENT_USER\SOFTWARE\Keyman\Keyman Engine\Active Keyboards\imsample and are of type REG_DWORD The keys are ShowIMWindow and ShowIMWindowAlways they both need to be set to 1.

Note for the IM window you need to use the mouse pointer to select the options, you can't type the number.

  • TEST_IMSAMPLE_INPUT_IMW:

    • Install Imsample keyboard attached to this PR (if not done already)
    • Select the keyboard its name is IMTest
    • Type some keys that are not the special keys e.g. t r y
    • Type the space key then a then click the 1st option
    • Expected Result: try æ
    • continue to the next test

  • TEST_IMSAMPLE_INPUT_IMW_CONT: ... contd from above

    • Type 'm' and select the 3rd ### option
    • Expected Result: try æ###

SIMPLIFIED_CHINESE:

This keyboard will display the IMX window as soon as a string the matches the pinyin for one or more characters are typed.

For all these tests install the Simplified Chinese Keyboard cs-pinyin.cmp found here

  • TEST_ SIMPLIFIED_CHINESE_SINGLE: Using Notepad or equivalent.

    • Type o - The IMX window should appear
    • Choose the 4th option
    • Expected result: 喔

  • TEST_ SIMPLIFIED_CHINESE_MULTIPLE: Using Notepad or equivalent.

    • Type h a n z i - The IMX window will appear and in the top left the letters hanzi will be present
    • Choose the 5th option
    • Expected result: 汉字变换

  • TEST_ SIMPLIFIED_CHINESE_BACKSPACE_1: Using Notepad or equivalent. This is to test the backspace occurring mid pinyin sequence before a character is output to the app

    • Type h a n z i - The IMX window will appear and in the top left the letters hanzi will be present
    • Press Backspace twice - The IMX window should now have the top left letters of han.
    • Choose the 5th option
    • Expected result: 汗

  • TEST_ SIMPLIFIED_CHINESE_BACKSPACE_2: Using Notepad or equivalent. This is to test the backspace of already output characters.

    • Type h a n z i - The IMX window will appear and in the top left the letters hanzi will be present.
    • Choose the 5th option
    • Expected result: 汉字变换
    • Press Backspace twice
    • Expected result: 汉字
Clone this wiki locally