Restructure SEC company information tables

[katie-lamb]

Overview

Closes #4078 . The first of two (or more) SEC table restructuring PRs - this handles the quarterly filings table and company information tables. It doesn't include the ownership tables.

What problem does this address?

Makes changes to the SEC table structures to be more well normalized and usable.

What did you change?

core_sec10k__quarterly_filings
This table is largely the same. Only minor changes were made.

• Ensure that the CIK of each filing matches the CIK from the filename string - make a test for this?
• Update filing_date column description to indicate that it's daily frequency
• Indicate that report_date is the quarter that the filing_date pertains to
• Enforce format of exhibit_21_version with a regex - enforce this with a constraint in the field metadata?

Company Information Tables

raw_sec10k__quarterly_company_information

• Rename core_sec10k__quarterly_company_information to be a raw table

core_sec10k__quarterly_company_information

• Pivot the raw table so that field values are columns
• Make report_date and central_index_key the primary key
• Strip leading "[" from field values
• Add field metadata for new columns
• Investigate how many CIKs from the quarterly filings table don't have corresponding harvested company information in this table.
  • I see 257/39183 filer CIKs that aren't harvested into this company information table coming from 360 filings. It looks like ~200 of these filings are pre-2000 report dates. See this comment.
• Update row counts in the ETL fast and full row counts CSVs

Questions:

• central_index_key and report_date are not the natural primary key because there are slight differences in headers harvested from different filings. I prioritized headers from filings where the filer is the same as the record we're harvesting but we do lose some records by forcing the CIK + report date primary key. Any reason to leave filename as the primary key?

out_sec10k__quarterly_company_information

• Merge utility_id_eia and utility_name_eia onto the core company information table.

core_sec10k__changelog_company_name

• Pull central_index_key, report_date, name_change_date, former_conformed_name, current_name out of raw company information table and see if central_index_key, report_date, name_change_dateis a natural primary key
• make FK relationship back to core table

Documentation

Make sure to update relevant aspects of the documentation.

Tasks

Preview Give feedback

1. Update the release notes: reference the PR and related issues.
2. Update relevant Data Source jinja templates (see docs/data_sources/templates).
3. Update relevant table or source description metadata (see src/metadata).
4. Review and update any other aspects of the documentation that might be affected by this PR.

Testing

How did you make sure this worked? How can a reviewer verify this?

To-do list

Preview Give feedback

1. If updating analyses or data processing functions: make sure to update or write data validation tests (e.g. test_minmax_rows()).
2. Run make pytest-coverage locally to ensure that the merge queue will accept your PR.
3. Review the PR yourself and call out any questions or issues you have.
4. For minor ETL changes or data additions, once make pytest-coverage passes, make sure you have a fresh full PUDL DB downloaded locally, materialize new/changed assets and all their downstream assets and run relevant data validation tests using pytest and --live-dbs.
5. For bigger ETL or data changes run the full ETL locally and then run the data validations using make pytest-validate.
6. Alternatively, run the build-deploy-pudl GitHub Action manually.