Overview

Closes #4078 . The first of two (or more) SEC table restructuring PRs - this handles the quarterly filings table and company information tables. It doesn't include the ownership tables.

What problem does this address?

Makes changes to the SEC table structures to be more well normalized and usable.

What did you change?

core_sec10k__quarterly_filings
This table is largely the same. Only minor changes were made.

• Ensure that the CIK of each filing matches the CIK from the filename string - make a test for this?
• Update filing_date column description to indicate that it's daily frequency
• Indicate that report_date is the quarter that the filing_date pertains to
• Enforce format of exhibit_21_version with a regex - enforce this with a constraint in the field metadata?

Company Information Tables

raw_sec10k__quarterly_company_information

• Rename core_sec10k__quarterly_company_information to be a raw table

core_sec10k__quarterly_company_information

• Pivot the raw table so that field values are columns
• Make report_date and central_index_key the primary key
• Strip leading "[" from field values
• Add field metadata for new columns
• Investigate how many CIKs from the quarterly filings table don't have corresponding harvested company information in this table.
  • I see 257/39183 filer CIKs that aren't harvested into this company information table. It looks like ~200 of these are pre-2000 records.
• Update row counts in the ETL fast and full row counts CSVs

Questions:

• central_index_key and report_date are not the natural primary key because there are slight differences in headers harvested from different filings. I prioritized headers from filings where the filer is the same as the record we're harvesting but we do lose some records by forcing the CIK + report date primary key. Any reason to leave filename as the primary key?

out_sec10k__quarterly_company_information

• Merge utility_id_eia and utility_name_eia onto the core company information table.

Documentation

Make sure to update relevant aspects of the documentation.

Testing

How did you make sure this worked? How can a reviewer verify this?