Timers and enhanced media player control

[synesthesiam]

Hi @home-assistant/language-leaders 👋

The 2024.6 version of HA brings us one step closer to support all of the common tasks people need with voice satellites! This version will add support for timers in Assist and more ways of controlling media players 🚀

For timers, we have 6 new intents and 3 new error messages:

• HassStartTimer - start a new timer (requires area context)
• HassCancelTimer - cancel a timer by name, area, or start time
• HassTimerStatus - report status of timer(s) (see the English response template)
• HassIncreaseTimer - add time to a timer
• HassDecreaseTimer - remove time from a timer
• HassPauseTimer - pause a running timer
• HassUnpauseTimer - resume a paused timer

See #2179 for an example of adding timers to the test fixtures.

The new error messages are:

• timer_not_found - error when no timer matches constraints (name, duration)
• multiple_timers_matched - error when more than one timers matches constraints (timer intents can only target one timer for now)
• no_timer_support - the voice satellite does not support timers

For enhanced media player control, the existing intents have been expanded so that a media player name is not required. Area context is required so that sentences like "pause" and "next" will target the media player(s) that are playing in the voice satellite's area. Only media players in the correct state (playing/paused) and supporting the required features (setting volume) will be targeted.

There are 2 new error messages:

• entity_wrong_state - an entity is not in expected state (e.g., playing for a media player when "pause" is spoken)
• feature_not_supported - entity does not support the required feature (e.g., VOLUME_SET for a media player when "set volume..." is spoken)

Let me know if you have any questions. Thanks! 🙂

[synesthesiam]

One addition to timers: there is now a conversation_command slot on HassStartTimer that may contain the text of a command to execute when the timer is finished. This will let us do things like "open the garage door in 5 minutes" 🙂

Example:

intents/sentences/en/homeassistant_HassStartTimer.yaml

Line 19 in c3a53fc

- sentences:

[cibernox]

That is great. It's not something I use very often but I have used in Alexa a few times.

[cibernox]

The only remaining feature for the wife-approval-factor to be good enough to switch alexa with local voice control right now is the "brief mode" (home-assistant/architecture#1047)

[synesthesiam]

I think brief mode will be something that could be enabled in ESPHome or on the Wyoming satellite itself. It would still get the text, but could just do an acknowledgement "beep" instead of speaking it.

[cibernox]

I leaned towards having it in home assistant because it's not so cut and dry to just replace any answer with a beeping sound. Sometimes you do want the answer even when bried mode is enables. Responses to queries, error messages. Sometimes the response can be a shorter version of the one you'd get when brief mode is not enabled.

I have the hunch that it would make more sense in HA itself.

[mitrokun]

Is it possible to remove the playback of the timer signal when executing a delayed command in the future? It is enough to use the standard Done.wav. Or add another sound setting for this case.

[TheFes]

Trying to translate the intents for timer to Dutch, but I run into some errors.
I have a draft PR now #2194

The error are about unsupported slots or missing slots, and I don't really understand why or how. Can someone shed a light on this?

======================================================================================================================================== FAILURES ========================================================================================================================================
_________________________________________________________________________________________________________________________ test_homeassistant_HassCancelTimer[nl] _________________________________________________________________________________________________________________________

intent_schemas = {'HassCancelTimer': {'description': 'Cancels a timer', 'domain': 'intent', 'slots': {'area': {'description': 'Area of ...inutes', 'required': False}, 'name': {'description': 'Name attached to the timer', 'required': False}, ...}, ...}, ...}
language_sentences_yaml = {'_common.yaml': {'expansion_rules': {'afdekking': '[de|het] (<awning>|<blind>|<curtain>|<door>|<garage>|<gate>|<shade...Temperature.yaml': {'intents': {'HassClimateSetTemperature': {'data': [{'sentences': [...]}]}}, 'language': 'nl'}, ...}
language_sentences_common = Intents(language='nl', intents={}, slot_lists={'brightness': RangeSlotList(start=0, stop=100, step=1, type=<RangeType....}, skip_words=['alstublieft', 'alsjeblieft', 'aub', 'ajb', 'graag'], settings=IntentsSettings(ignore_whitespace=False))

    def test_func(
        intent_schemas: dict[str, Any],
        language_sentences_yaml: dict[str, Any],
        language_sentences_common: Intents,
    ) -> None:
>       do_test_language_sentences(
            test_file,
            intent_schemas,
            language_sentences_yaml,
            language_sentences_common,
        )

tests/test_language_intents.py:215: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/test_language_intents.py:108: in do_test_language_sentences
    _verify(
tests/test_language_intents.py:188: in _verify
    _verify(
tests/test_language_intents.py:188: in _verify
    _verify(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

expression = ListReference(list_name='timer_seconds', prefix=None, suffix=None, _slot_name='seconds')
intents = Intents(language='nl', intents={'HassCancelTimer': Intent(name='HassCancelTimer', data=[IntentData(sentence_texts=['<t...}, skip_words=['alstublieft', 'alsjeblieft', 'aub', 'ajb', 'graag'], settings=IntentsSettings(ignore_whitespace=False))
intent_name = 'HassCancelTimer'
slot_schema = {'area': {'description': 'Area of the device used to start the timer', 'required': False}, 'name': {'description': 'Na...h', 'required': False}, 'start_minutes': {'description': 'Minutes the timer was started with', 'required': False}, ...}
visited_rules = {'timer_duration_seconds', 'timer_start'}, found_slots = set()
expansion_rules = {'afdekking': Sentence(items=[Sequence(items=[Sequence(items=[TextChunk(text='de', original_text='de')], type=<Sequenc...enceType.GROUP: 'group'>)], type=<SequenceType.ALTERNATIVE: 'alternative'>, text='(luifel[s]|[zonne]scherm[en])'), ...}

    def _verify(
        expression: Expression,
        intents: Intents,
        intent_name: str,
        slot_schema: dict[str, Any],
        visited_rules: set[str],
        found_slots: set[str],
        expansion_rules: dict[str, Sentence],
    ) -> None:
        if isinstance(expression, ListReference):
            list_ref: ListReference = expression
    
            # Ensure list exists
            assert (
                list_ref.list_name in intents.slot_lists
            ), f"Missing slot list: {{{list_ref.list_name}}}. Are you missing a 'lists' entry in _common.yaml?"
    
            # Ensure slot is part of intent
>           assert (
                list_ref.slot_name in slot_schema
            ), f"Intent {intent_name} does not support slot '{list_ref.slot_name}'. See intents.yaml for supported slots"
E           AssertionError: Intent HassCancelTimer does not support slot 'seconds'. See intents.yaml for supported slots
E           assert 'seconds' in {'area': {'description': 'Area of the device used to start the timer', 'required': False}, 'name': {'description': 'Na...h', 'required': False}, 'start_minutes': {'description': 'Minutes the timer was started with', 'required': False}, ...}
E            +  where 'seconds' = ListReference(list_name='timer_seconds', prefix=None, suffix=None, _slot_name='seconds').slot_name

tests/test_language_intents.py:168: AssertionError
_________________________________________________________________________________________________________________________ test_homeassistant_HassCancelTimer[nl] _________________________________________________________________________________________________________________________

language = 'nl', intent_schemas = {'HassCancelTimer': {'description': 'Cancels a timer', 'domain': 'intent', 'slots': {'area': {'description': 'Area of ...inutes', 'required': False}, 'name': {'description': 'Name attached to the timer', 'required': False}, ...}, ...}, ...}
slot_lists = {'area': TextSlotList(values=[TextSlotValue(text_in=TextChunk(text='keuken', original_text='keuken'), value_out='Keuke...t_in=TextChunk(text='rover', original_text='rover'), value_out='Rover', context={'domain': 'vacuum'}, metadata=None)])}
states = [State(entity_id='light.slaapkamer_lamp', name='Slaapkamerlampje', hass_state='off', attributes={}, area_id='slaapkame...nkamerlamp', hass_state='on', attributes={}, area_id='woonkamer', human_state='aan', aliases=set(), _domain=None), ...]
areas = [AreaEntry(id='keuken', name='Keuken', aliases=set()), AreaEntry(id='woonkamer', name='Woonkamer', aliases=set()), Are...et()), AreaEntry(id='hal', name='Hal', aliases=set()), AreaEntry(id='bijkeuken', name='Bijkeuken', aliases=set()), ...]
timers = [Timer(is_active=True, start_hours=1, start_minutes=None, start_seconds=None, rounded_hours_left=0, rounded_minutes_le...ounded_hours_left=0, rounded_minutes_left=3, rounded_seconds_left=0, name=None, area='keuken', total_seconds_left=190)]
language_sentences = Intents(language='nl', intents={'HassGetState': Intent(name='HassGetState', data=[IntentData(sentence_texts=['<is> [de...}, skip_words=['alstublieft', 'alsjeblieft', 'aub', 'ajb', 'graag'], settings=IntentsSettings(ignore_whitespace=False))
language_responses = {'HassCancelTimer': {'default': 'Timer geannuleerd'}, 'HassClimateGetTemperature': {'default': "{% set temperature = s...ClimateSetTemperature': {'default': 'Temperatuur ingesteld'}, 'HassDecreaseTimer': {'default': 'Timer aangepast'}, ...}

    def test_func(
        language: str,
        intent_schemas: dict[str, Any],
        slot_lists: dict[str, SlotList],
        states: List[State],
        areas: List[AreaEntry],
        timers: List[Timer],
        language_sentences: Intents,
        language_responses: dict[str, Any],
    ) -> None:
>       do_test_language_sentences_file(
            language,
            test_file_stem,
            intent_schemas,
            slot_lists,
            states,
            areas,
            timers,
            language_sentences,
            language_responses,
        )

tests/test_language_sentences.py:218: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

language = 'nl', test_file = 'homeassistant_HassCancelTimer'
intent_schemas = {'HassCancelTimer': {'description': 'Cancels a timer', 'domain': 'intent', 'slots': {'area': {'description': 'Area of ...inutes', 'required': False}, 'name': {'description': 'Name attached to the timer', 'required': False}, ...}, ...}, ...}
slot_lists = {'area': TextSlotList(values=[TextSlotValue(text_in=TextChunk(text='keuken', original_text='keuken'), value_out='Keuke...t_in=TextChunk(text='rover', original_text='rover'), value_out='Rover', context={'domain': 'vacuum'}, metadata=None)])}
states = [State(entity_id='light.slaapkamer_lamp', name='Slaapkamerlampje', hass_state='off', attributes={}, area_id='slaapkame...nkamerlamp', hass_state='on', attributes={}, area_id='woonkamer', human_state='aan', aliases=set(), _domain=None), ...]
areas = [AreaEntry(id='keuken', name='Keuken', aliases=set()), AreaEntry(id='woonkamer', name='Woonkamer', aliases=set()), Are...et()), AreaEntry(id='hal', name='Hal', aliases=set()), AreaEntry(id='bijkeuken', name='Bijkeuken', aliases=set()), ...]
timers = [Timer(is_active=True, start_hours=1, start_minutes=None, start_seconds=None, rounded_hours_left=0, rounded_minutes_le...ounded_hours_left=0, rounded_minutes_left=3, rounded_seconds_left=0, name=None, area='keuken', total_seconds_left=190)]
language_sentences = Intents(language='nl', intents={'HassGetState': Intent(name='HassGetState', data=[IntentData(sentence_texts=['<is> [de...}, skip_words=['alstublieft', 'alsjeblieft', 'aub', 'ajb', 'graag'], settings=IntentsSettings(ignore_whitespace=False))
language_responses = {'HassCancelTimer': {'default': 'Timer geannuleerd'}, 'HassClimateGetTemperature': {'default': "{% set temperature = s...ClimateSetTemperature': {'default': 'Temperatuur ingesteld'}, 'HassDecreaseTimer': {'default': 'Timer aangepast'}, ...}

    def do_test_language_sentences_file(
        language: str,
        test_file: str,
        intent_schemas: dict[str, Any],
        slot_lists: dict[str, SlotList],
        states: List[State],
        areas: List[AreaEntry],
        timers: List[Timer],
        language_sentences: Intents,
        language_responses: dict[str, Any],
    ) -> None:
        """Tests recognition all of the test sentences for a language"""
        if not get_test_path(language, test_file).exists():
            return
    
        testing_domain, testing_intent = test_file.rsplit("_", maxsplit=1)
    
        seen_sentences = set()
        template_env = Environment(loader=BaseLoader())
    
        for test in load_test(language, test_file)["tests"]:
            intent = test["intent"]
            assert (
                intent["name"] == testing_intent
            ), f"File {test_file}: Unexpected intent {intent['name']}. Only test for intent {testing_intent}"
    
            if not test["sentences"]:
                continue
    
            if testing_domain != "homeassistant":
                # Domain specific files (ie light_HassTurnOn.yaml) should only test
                # sentences for the light domain.
                if intent_schemas[testing_intent]["domain"] == testing_domain:
                    assert ("domain" not in intent.get("slots", {})) and (
                        "domain" not in intent.get("context", {})
                    ), f"File {test_file}: tests should not have a domain slot"
                else:
                    assert (intent.get("slots", {}).get("domain") == testing_domain) or (
                        intent.get("context", {}).get("domain") == testing_domain
                    ), f"File {test_file}: tests should have domain slot set to {testing_domain}"
    
            intent_context = intent.get("context", {})
            expected_response_texts = test.get("response")
            if expected_response_texts:
                if isinstance(expected_response_texts, str):
                    expected_response_texts = {expected_response_texts.strip()}
                else:
                    expected_response_texts = set(
                        t.strip() for t in expected_response_texts
                    )
    
            for sentence in test["sentences"]:
                assert (
                    sentence not in seen_sentences
                ), f"Duplicate sentence found: {sentence}"
                seen_sentences.add(sentence)
    
                result = recognize(
                    sentence,
                    language_sentences,
                    slot_lists=slot_lists,
                    intent_context=intent_context,
                )
                assert result is not None, f"Recognition failed for '{sentence}'"
                assert (
                    result.intent.name == intent["name"]
                ), f"For '{sentence}' expected intent {intent['name']}, got {result.intent.name}"
    
                matched_slots = {slot.name: slot.value for slot in result.entities.values()}
                actual_slots = intent.get("slots") or {}
    
                # Check for all match slots
                for match_name, match_value in matched_slots.items():
                    actual_value = actual_slots.get(match_name)
>                   assert (
                        actual_value is not None
                    ), f"Missing slot {match_name} for: {sentence} (value={match_value})"
E                   AssertionError: Missing slot minutes for: annuleer 5 minuten timer (value=5)
E                   assert None is not None

tests/test_language_sentences.py:132: AssertionError
================================================================================================================================ short test summary info =================================================================================================================================
FAILED tests/test_language_intents.py::test_homeassistant_HassCancelTimer[nl] - AssertionError: Intent HassCancelTimer does not support slot 'seconds'. See intents.yaml for supported slots
FAILED tests/test_language_sentences.py::test_homeassistant_HassCancelTimer[nl] - AssertionError: Missing slot minutes for: annuleer 5 minuten timer (value=5)
============================================================================================================================ 2 failed, 91 deselected in 1.40s ============================================================================================================================

[synesthesiam]

You need to use start_hours, start_minutes, and start_seconds in all of the intents that target existing timers (everything except HassStartTimer. These refer to the starting time of the timer as set by the user.

HassIncreaseTimer, HassDecreaseTimer, and HassStartTimer all have hours, minutes, and seconds slots that refer to how much time to add/remove/start with.

[synesthesiam]

This is why English has separate timer_duration and timer_start expansion rules.

[TheFes]

I was stupid, and thought the expansion rules for start were the same as for duration. So I used anchors.
Of course there were differences I overlooked (seconds vs start_seconds etc).
Changing that fixed it.

I have it ready for review now, asked Paulus to have a look

[tetele]

I think there is a bug or inconsistency. @synesthesiam could you take a look, please?

# tests/ro/_fixtures.yaml
areas:
  - name: "Bucatarie"
    id: kitchen # <--- area_id different from name
    floor: Parter
  - name: "Sufragerie"
    id: living_room # <--- area_id different from name
    floor: Parter
timers:
  - area: kitchen # <--- area_id provided
    start_minutes: 5
    total_seconds_left: 190
    rounded_hours_left: 0
    rounded_minutes_left: 3
    rounded_seconds_left: 0

# tests/ro/homeassistant_HassPauseTimer.yaml
tests:
  - sentences:
      - "opreste putin cronometrul din sufragerie"
    intent:
      name: HassPauseTimer
      slots:
        area: Sufragerie # <--- area name provided - WORKS!
    response: Am pus pe pauză temporizatorul

# tests/ro/homeassistant_HassTimerStatus.yaml
tests:
  - sentences:
      - "care e statusul temporizatorului din bucatarie?"
    intent:
      name: HassTimerStatus
      slots:
        area: Bucatarie # <--- area name provided - DOES NOT PASS TEST
    response: |
      Mai durează 3 minute

The test for HassTimerStatus responds with the equivalent of No timers found, which corresponds to this branch of responses/ro/HassTimerStatus.yaml:

{% if num_timers == 0: %}

It's easily fixable by providing the area name instead of ID to the timer in _fixtures.yaml, but that would be inconsistent with other entity fixtures.

It's easy to miss this in English, where most IDs ar the same as the names of areas.

[jlpouffier]

Same exact thing in French ;)

[synesthesiam]

Fix here: #2210

[mib1185]

when doing any time command (besides start) without any context - which timer is addressed?

example

1. start a timer in the living room (via area context): start a 5 minute timer
2. start a timer in the bath room (via area context): start a 1 hour timer
3. issue command cancel timer via assistant in the app

which timer gets canceled?

[synesthesiam]

It depends. If you are speaking to a device in one of those areas, it will cancel the timer for that area. If you're in a different area (or there's no area context), it will fail since we don't support targeting multiple timers.

[mib1185]

ok - so we should make sentences which does not refer to an area or name (eq. "<timer_cancel>[ the| my] timer") area-aware (requiring area context)?

[mib1185]

after some thoughts about it - there might be some valid cases which will break with that approach 🤔

[synesthesiam]

Timer sentences that don't mention an area should have an area context required. This is because a device with timer support is required for the intents to work.

The only exception to this is HassStartTimer with a conversation_command slot, which can be set from anywhere.

[mib1185]

another question about the syntax - especial whitspaces within or outside of brackets - in #2183 we have the following syntax:

          - "<timer_cancel>[ the| my] timer"
          - "<timer_cancel>[ the| my] <timer_start> timer"

but i would expect something like

          - "<timer_cancel> [the|my] timer"
          - "<timer_cancel> [the|my] <timer_start> timer"

is this just a question of preference, or are there specific reasons for it?

[tetele]

<timer_cancel>[ (the|my)] timer is an in-between variant in terms of efficiency, but less readable.

[synesthesiam]

It's not a big deal for efficiency. We can probably do an automated clean up at some point.

[mib1185]

• [DE] Optimize usage of spaces in optional sentence parts #2213

[ViViDboarder]

How is matching done today? Is it full string matches or is everything tokenized first? I'm curious how the extra spaces impact performance.

[tetele]

Everything is tokenized and then the matching is done on tokens. You can look here for some details.

[mitrokun]

I found a small bug. If the device alias (possibly name too) ends with a dot, the conversation_command is not executed. Although for ordinary intents these names are not an issue.
Can anyone else confirm this?
I discovered this while attempting to improve the quality of piper tts output for endings in some names.
This use case is quite rare and can only be achieved with a deterministic dictionary for stt,