fix: change shape of filelist list data instead of re-reading it

[roedoejet]

this allows for any processes that take place on the filelist list data (i.e. normalization) to be preserved

fixes #504

PR Goal?

Ensures that we are outputting a set of symbols without duplicates

Fixes?

#504

Feedback sought?

sanity. code clarity

Priority?

medium

Tests added?

tests updated

How to test?

You could try re-running on your dataset to see how the config changes

Confidence?

medium

Version change?

none. this isn't a breaking change. no version change needed really.

Related PRs?

[semanticdiff-com]

Review changes with SemanticDiff.

Analyzed 6 of 8 files.

Overall, the semantic diff is 12% smaller than the GitHub diff.

	Filename	Status
✔️	everyvoice/wizard/dataset.py	21.11% smaller
✔️	everyvoice/utils/__init__.py	0.0% smaller
✔️	everyvoice/tests/test_text.py	Analyzed
✔️	everyvoice/tests/test_wizard.py	2.39% smaller
❔	everyvoice/tests/data/unit-test-case1.psv	Unsupported file format
❔	everyvoice/tests/data/relative/config/everyvoice-shared-text.yaml	Unsupported file format
✔️	everyvoice/model/e2e/config/__init__.py	12.5% smaller
✔️	everyvoice/config/text_config.py	21.31% smaller

[codecov]

Codecov Report

Attention: Patch coverage is 88.57143% with 4 lines in your changes missing coverage. Please review.

Project coverage is 74.48%. Comparing base (83742ff) to head (9ab79eb).
Report is 1 commits behind head on main.

Files	Patch %	Lines
everyvoice/wizard/dataset.py	83.33%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #515      +/-   ##
==========================================
+ Coverage   74.40%   74.48%   +0.07%     
==========================================
  Files          45       45              
  Lines        3004     3029      +25     
  Branches      479      491      +12     
==========================================
+ Hits         2235     2256      +21     
- Misses        676      679       +3     
- Partials       93       94       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

CLI load time: 0:00.24
Pull Request HEAD: 9ab79eba4a9a99f9bb90c888bdb3b91889a8eb00
Imports that take more than 0.1 s:
import time: self [us] | cumulative | imported package

[SamuelLarkin]

LGTM

[SamuelLarkin]

What is dropped in state["filelist_data_list"][0]?
Why are we dropping it?

[wiitt]

The first element of state["filelist_data_list"] is the header

[SamuelLarkin]

yeah forgot about the header. Got it.
I guess I got confused with ilne 679 which gets the headers.

[marctessier]

I ran some tests. List of characters is much shorter now without the duplicates.

ex: NEW: ( sdte note I manually added ':' for moh )

` TEST_characters: [' ', ':' , a, e, h, i, k, l, m, n, o, r, s, t, w, y, à, á, è, é,
    ì, í, ò, ó, ě, ό, –, ’]
  TEST_phones: [d, d͡z, d͡ʒ, e, èː, é, éː, ě, f, h, i, ìː, í, íː, j, k,
    kʷ, l, m, n, n̥, o, òː, ó, óː, s, t, t͡s, ũ, ũ̀ː, ṹ, ṹː, w, z, ɑ, ɑ̀ː,
    ɑ́, ɑ́ː, ɡ, ɡʷ, ɽ, ɽ̥, ʌ̃, ʌ̃̀ː, ʌ̃́, ʌ̃́ː, ʔ, ό]`

Previous version:

`TEST_characters: [' ', '!', '"', '''', (, ), ',', ., /, ':', ;, '?', a, á, b, e,
    é, h, i, ì, í, k, m, n, o, ó, p, r, s, t, v, w, x, y, a, à, á, b, c, d, e, è,
    é, g, h, i, ì, í, k, l, m, n, o, ò, ó, p, r, s, t, u, v, w, x, y, ' ', á, è, é,
    ì, í, ò, ó, à, á, è, é, ì, í, ò, ó, ě, ό, –, —, ’, “, ”]
  TEST_phones: [b, c, d, d͡z, d͡ʒ, e, èː, é, éː, ě, f, g, h, i, ìː, í, íː, j, k, kʷ,
    l, m, n, n̥, o, òː, ó, óː, p, s, t, t͡s, t͡ʃ, u, ũ, ũ̀ː, ṹ, ṹː, v, w, x, z, ɑ,
    ɑ̀ː, ɑ́, ɑ́ː, ɡ, ɡʷ, ɽ, ɽ̥, ʃ, ʌ̃, ʌ̃̀ː, ʌ̃́, ʌ̃́ː, ʔ, ό]` 

Training worked with no issues.

[wiitt]

Everything looks good to me except the pre-commit part in git workflows which I don't feel confident about.