Specification review by DVW

[cconcolato]

Comments received from Peter Gillespie from https://descriptivevideoworks.com/ and reported here with permission:

I have gone through the DAPT specification and made some notes on my observations (see below). Overall I found it to be very comprehensive and well thought out!

1. General Comments

• good job overall – most of it is very understandable!

[Additional comment split out into individual comment thread below]

2. Specific Comments

[These have been split out into individual comment threads below.]

[nigelmegitt]

• I note that in all the examples on seconds are used and not frames. I see there is allowance for using frames in 5.7.3 but I don’t see it in the examples. Is that a global setting (applies to the entire file) or a per event setting? Also, in 5.7.5 it seems that ‘clock expressions which look similar to time-code are prohibited. However, in your example in E3, it looks like clock modes are supported?

	<feature value="optional">#clockMode-gps</feature>
    	<feature value="optional">#clockMode-local</feature>
   	 <feature value="optional">#clockMode-utc</feature>

[nigelmegitt]

An example of the use of frames would be begin="3254f" - I'm not sure how useful that would be generally, but we could certainly modify one or more of the examples to use this syntax.

§E3 is not an example, it's a formal part of the specification. It says that it is optional for a processor conforming to the profile to support those clock modes. We do not prohibit any feature in a processor profile: that would unreasonably invalidate processors that happen to support a feature even if the input documents never exercise it. The lease processor support we specify is "optional", which means that nobody can rely on it being present in a conformant processor; therefore it is unwise to use it in a document. More strongly though, if you look at §E2, which defines the content profile, use of those features in DAPT documents is in fact prohibited.

[nigelmegitt]

• Final sentence of the last paragraph of section 2.1 – “That mixing can occur prior to distribution, or in the client directly.”
  • not quite sure by the meaning of ‘in the client directly’. In the client’s portal directly or in a software client directly, or something else? I see later you refer to this as the ‘client’s player’.

[nigelmegitt]

This is supposed to indicate that it is possible to send the DAPT document all the way to the player in the user's UA/browser and code in the player can parse the document and do the audio mixing locally. An example of this approach is prototyped at https://bbc.github.io/Adhere/

Happy to have suggestions for clearer wording. I'm not sure I understand the distinction between "client's portal" and "software client".

[cconcolato]

From DVW:

I understand what you mean after this clarification and I’m not sure my alternate wording would be
much better. Perhaps just change the work ‘client’ to ‘player’?

[nigelmegitt]

• @nigelmegitt to raise this wording tweak as an issue.

[nigelmegitt]

Raised as #202

[nigelmegitt]

• Paragraph 1 of section 2.1.2 - “It is the result of the audio rendition of one or more descriptions mixed with the audio associated with the programme prior to any mixing with audio description (sometimes referred to as main programme audio)”
  • this is super confusing to me. What is the audio associated with the program if it is not the actual program audio? If that is the case, then this sentence says that we are mixing the audio descriptions with the programme audio, prior to mixing with the main programme audio, which of course doesn’t make sense.

[nigelmegitt]

Thanks for this, I will try to reword to make this easier to read. What it's trying to say is:

Audio description = as-received programme audio + descriptions rendered as audio

Would this be better:

It is the result of mixing the "main programme audio" with the audio rendition of each description, where "main programme audio" refers to the audio associated with the programme prior to any mixing.

?

[cconcolato]

From DVW:

Yes, that is much clearer, thanks!

[nigelmegitt]

• @nigelmegitt to make this into an issue.

[nigelmegitt]

Raised as #203.

[nigelmegitt]

• Section 2.1.2 in the list – “defining mixing instructions (applied using [TTML2] audio styling) for combining the audio with the programme audio.”
  • there are two distinct cases here; one where we are actually mixing the audio descriptions with the program audio, and one where the client’s player (portal) facilitates the mixing. It sounds like the sentence above is only concerned with defining mixing parameters for automated mixing?

[nigelmegitt]

Sorry, I don't understand this question: the mixing is happening either way, just in different parts of the chain. Is this a reference to the idea that during client side mixing the user might somehow vary the parameters of the mixing to suit their own preferences? Even then, the mixing instructions provide the baseline behaviour.

[cconcolato]

from DVW

Yes, mixing instructions would definitely need to be defined but interestingly enough, our AD writers
never provide them. However, these may be automatically applied somewhere/somehow else in the
process – just not sure where and how.

[nigelmegitt]

• Example 2 section 2.2.1 – “The first example shows a script where timed opportunities for descriptions or transcriptions have been identified but no text has been written.”
  • it would be interesting to know how these are being derived. Are these spaces that the audio waveform has identified as being without main program audio?

[nigelmegitt]

I do not think we should be prescriptive about this in the specification. One way would be to analyse the audio waveform. Another might be to invert the times where a transcript of the dialogue identifies there is some speech. There may be other mechanisms, I guess.

[cconcolato]

From DVW

Understood

[nigelmegitt]

• Example 4 Section 2.2.1 – “After creating audio recordings, if not using text to speech, instructions for playback mixing can be inserted. For example, the gain of "received" audio can be changed before mixing in the audio played from inside the span, smoothly animating the value on the way in and returning it on the way out”
  • good explanation of what is happening in this example, but I’m not sure what the span parameter does (from .3 to 2.7)?

[nigelmegitt]

The <span> contains the text of the description and its audio representation. Its timing specifies when playback of the associated audio begins, relative to the parent element's begin time, i.e. in this case, 25 + 0.3 = 25.3s.

That's what the text after the example is trying to explain, especially:

Then the <span> is timed to begin only after the first audio dip has finished.

Happy to take suggestions for editorial improvement of the wording so this reads better.

[cconcolato]

From DVW

This is a good explanation as it stands, thanks.

[nigelmegitt]

• Section 4.1.6.1 – Timing Properties – “DAPT Scripts begin at time zero on the media timeline.”
  • Scripts may begin at time zero but what if the program/proxy does not?

[nigelmegitt]

I think that's a logical impossibility, because of the way that media time is defined. Assume a media timeline on which the program/proxy is placed: the DAPT Script's times have to be on that timeline to match the program content.

For a standalone complete program, the first frame of video is at time zero on the media timeline, unless there's some other local context that changes that. For a program in the context of a live stream, for example, the media timeline begins at some epoch time, like UNIX time that is relative to the beginning of the year 1970. In that case, the timestamps needed to synchronise with the programme at the point it appears on that media timeline.

Either way, time is ultimately expressed as a positive non-integer number of seconds on an externally defined media timeline.

[cconcolato]

From DVW:

I understand the context and even if the proxy time and the script time didn’t align you are correct,
that it not for the DAPT script to sort out.

[nigelmegitt]

• Section 4.2 – Character
  • I didn’t get at first, that the entire Character section only applies to dubbing (even though the first sentence generally states this). This might benefit from a note at the top of the section stating that ‘Character attributes do not apply to Audio Description scripts). Then we AD people can just skip that section 😉.

[nigelmegitt]

Happy to add a note.

[nigelmegitt]

Raised as #188 - by the way, I would prefer not to rule out adding characters entirely for AD scripts. There could be a use for them, e.g. to identify different AD voices for different parts of the programme, though I understand that is not generally done today.

[cconcolato]

From DVW

Yes, this is a good point that I overlooked.

[nigelmegitt]

• Section 4.9.1 – Audio Recording – Linking to external audio sources.
  • as per the editor’s note, we haven’t run into any situations where we would actually be using differing audio sources so in our case, the fragment identifier would not be useful – perhaps they run across this in dubbing though?

[nigelmegitt]

I think this is mainly an AD use case since the dubbing use case generally is likely to involve recording and mixing externally to the DAPT script, as per common practice today.

I'm not sure which note this refers to - is this in response to one of the embedded issues? If so, which one?

[cconcolato]

From DVW:

The subsequent discussion on embedding has taken this discussion beyond my original observation.
Also, in my original interpretation I didn’t quite understand that ‘embedding’ meant embedding audio
data in the document, as opposed to ‘embedded audio’ which we work with (where the audio is
embedded with the video in a single file).

[nigelmegitt]

• Section 4.10 – Mixing Instructions – “Zero or more Gain properties. The gain acts as a multiplier to be applied to the related Audio.”
  • from the examples at the beginning of the document it looked like these values were absolute and not multipliers?

[nigelmegitt]

I don't follow this: how do the examples suggest gain values being absolute? If that suggestion is present, we should change the wording to make it clearer. I can't find it though.

In case it helps, the semantics are defined ultimately by https://www.w3.org/TR/webaudio/#gainnode but with the additional constraint from TTML2 that gain is clamped to the range 0..1.

[cconcolato]

From DVW:

Yes and specifically in 5.7.5 this is spelled out so I was off base on this one.

[nigelmegitt]

• Section 4.10 – Mixing Instructions
  • I’m curious to know what happens when a gain attribute has been applied to an event with fill = freeze and that event ends without returning the audio to normal. Does the gain automatically return to normal at the end of the event or will it be frozen at the new gain parameter?

[nigelmegitt]

That's an excellent question. My interpretation has been that when the div or p element corresponding to the event ends the audio node that corresponds to that element is removed from the tree, so there's nothing for the gain to apply to, and therefore any gain applied to the main programme audio, for example, would revert.

I'd be interested to know if there's other implementer feedback about this, and if we need to make any specification changes to clarify it: is there another possible interpretation than mine? (surely that's a yes!)