Code

80 recommend movies#81

Code

Merged

jhanley634 merged 70 commits intomainnoisebridge/MediaBridge:mainfrom 80-recommend-moviesnoisebridge/MediaBridge:80-recommend-moviesCopy head branch name to clipboardFeb 11, 2025

+1,202-260

Commits

Commits on Jan 27, 2025

• Flattening returns dict[str, str]

  jhanley634

  committed
  b9f671b

  Copy full SHA for b9f671b

• Annotate bulk_insert(). Given that nothing calls the function, it's not entirely clear what its signature ought to be, but this seems like a decent guess

  jhanley634

  committed
  0bd5cf4

  Copy full SHA for 0bd5cf4

• Enable --strict mode for mypy linting

  jhanley634

  committed
  5ad14ba

  Copy full SHA for 5ad14ba

• Merge branch 'main' into 76_enable_--strict_mode

  jhanley634

  authored
  780829e

  Copy full SHA for 780829e

Commits on Jan 28, 2025

• LightFM is a recommender, so move it into the recommender/ folder

  jhanley634

  committed
  2446c0e

  Copy full SHA for 2446c0e

• Add test_two_movies_rec()

  jhanley634

  committed
  9a70872

  Copy full SHA for 9a70872

• Merge branch 'main' of https://github.com/noisebridge/MediaBridge into 80-recommend-movies

  jhanley634

  committed
  489ac6f

  Copy full SHA for 489ac6f

Commits on Jan 30, 2025

• Add pandas + sqlalchemy

  jhanley634

  committed
  74e8078

  Copy full SHA for 74e8078

• Add pandas dep

  jhanley634

  committed
  43d3a75

  Copy full SHA for 43d3a75

Commits on Jan 31, 2025

• We use tuple for a C struct, and list for arbitrary number of "same thing". Also, DRY up the docstring so it doesn't repeat what the signature already told us.

  jhanley634

  committed
  8a7dd92

  Copy full SHA for 8a7dd92

Commits on Feb 1, 2025

• Deal with "unknown year"

  jhanley634

  committed
  a64b527

  Copy full SHA for a64b527

• Dump dataframe to sqlite table

  jhanley634

  committed
  aedb699

  Copy full SHA for aedb699

• Add _etl_user_rating

  jhanley634

  committed
  35305a3

  Copy full SHA for 35305a3

Commits on Feb 2, 2025

• Read files in lexical order

  jhanley634

  committed
  5d5509e

  Copy full SHA for 5d5509e

Commits on Feb 3, 2025

• Avoid ETL of movie titles if they're not available, e.g. during a CI run

  jhanley634

  committed
  84cfbec

  Copy full SHA for 84cfbec

Commits on Feb 4, 2025

• Push silence_logging() down a level

  jhanley634

  committed
  36543c5

  Copy full SHA for 36543c5

• etl() now takes a glob parameter

  jhanley634

  committed
  c422549

  Copy full SHA for c422549

• Add Rating table

  jhanley634

  committed
  41eb3e9

  Copy full SHA for 41eb3e9

• Insert movie ratings

  jhanley634

  committed
  343a6a5

  Copy full SHA for 343a6a5

Commits on Feb 5, 2025

• Write to compressed rating.csv.gz file

  jhanley634

  committed
  c334c4c

  Copy full SHA for c334c4c

• use pv | gzip compression pipeline (child won't hold the GIL)

  jhanley634

  committed
  2bd9816

  Copy full SHA for 2bd9816

• Omit the pv child, as buffering only buys us a 10% reduction in elapsed time

  jhanley634

  committed
  bbb60f7

  Copy full SHA for bbb60f7

• Don't re-create CSV if it's already there

  jhanley634

  committed
  899f5ad

  Copy full SHA for 899f5ad

• Measure the time to write rating_temp

  jhanley634

  committed
  1891445

  Copy full SHA for 1891445

• Prefer the ORM bulk insert API

  jhanley634

  committed
  e6ecf89

  Copy full SHA for e6ecf89

• Use a class_mapper()

  jhanley634

  committed
  d0a7b80

  Copy full SHA for d0a7b80

• Order first by movie, then by user

  jhanley634

  committed
  0a43209

  Copy full SHA for 0a43209

• Mention why we go to the trouble of spawning a gzip child

  jhanley634

  committed
  b119fb5

  Copy full SHA for b119fb5

• Add timing remarks

  jhanley634

  committed
  a1396eb

  Copy full SHA for a1396eb

• Looks like the DB belongs in the out/ folder

  jhanley634

  committed
  bbbdd83

  Copy full SHA for bbbdd83

• Add max_rows parameter to etl()

  jhanley634

  committed
  71494d7

  Copy full SHA for 71494d7

• Simplify: delete the rating_temp table, as it does not appear to offer an elapsed time win

  jhanley634

  committed
  c12fd90

  Copy full SHA for c12fd90

• Ensure that OUTPUT_DIR exists, even in CI

  jhanley634

  committed
  2c940aa

  Copy full SHA for 2c940aa

Commits on Feb 6, 2025

• Choose a better identifier

  jhanley634

  committed
  58463e0

  Copy full SHA for 58463e0

• Call a spade a spade: rename to "etl"

  jhanley634

  committed
  e193923

  Copy full SHA for e193923

• Add PopularMovie and ProlificUser tables

  jhanley634

  committed
  596e583

  Copy full SHA for 596e583

• Only spend time INSERTing rating rows if we need to

  jhanley634

  committed
  77fb994

  Copy full SHA for 77fb994

• We can't recycle a ForeignKey() object, so make a new one

  jhanley634

  committed
  adfa8d7

  Copy full SHA for adfa8d7

• Add _gen_reporting_tables

  jhanley634

  committed
  56ebf56

  Copy full SHA for 56ebf56

• Make output a little less chatty

  jhanley634

  committed
  1562e7e

  Copy full SHA for 1562e7e

• Add recommend()

  jhanley634

  committed
  fac22fb

  Copy full SHA for fac22fb

• Add _get_ratings()

  jhanley634

  committed
  df7741c

  Copy full SHA for df7741c

• Add _get_test_movie_ids()

  jhanley634

  committed
  88ee791

  Copy full SHA for 88ee791

• Prefer an ORM-style query

  jhanley634

  committed
  5190bc7

  Copy full SHA for 5190bc7

• Add WIP discussion

  jhanley634

  committed
  aeac2ed

  Copy full SHA for aeac2ed

• Rename

  jhanley634

  committed
  8c7fb2d

  Copy full SHA for 8c7fb2d

• Relax a threshold test

  jhanley634

  committed
  a9b6802

  Copy full SHA for a9b6802

• Look before you leap: only run tests in CI when we know the prerequisite data is available

  jhanley634

  committed
  0abe9d2

  Copy full SHA for 0abe9d2

• Fix a "None" bug revealed by "pipenv run dev process"

  jhanley634

  committed
  1a96f17

  Copy full SHA for 1a96f17

• Suggest zero-padding upon ingest

  jhanley634

  committed
  178c54e

  Copy full SHA for 178c54e

• Merge branch 'main' into 80-recommend-movies

  jhanley634

  authored
  48ff825

  Copy full SHA for 48ff825

Commits on Feb 7, 2025

• Add discussion item 4, which suggests "pipenv run dev recommend"

  jhanley634

  committed
  0cdc5d8

  Copy full SHA for 0cdc5d8

• Delete the magic number 4

  jhanley634

  committed
  62e925e

  Copy full SHA for 62e925e

Commits on Feb 8, 2025

• Bugfix: avoid trying to convert y-m-d to an integer

  jhanley634

  committed
  a3b6051

  Copy full SHA for a3b6051

• Bugfix: avoid changing the sparsity structure of a csr_matrix, as it is expensive

  jhanley634

  committed
  06bfb94

  Copy full SHA for 06bfb94

• Add tqdm progress bar

  jhanley634

  committed
  30e90bb

  Copy full SHA for 30e90bb

• Adjust docstring description of "txt_file" parameter

  jhanley634

  committed
  d66d161

  Copy full SHA for d66d161

• Rename module to data_processing/etl.py

  jhanley634

  committed
  c77012e

  Copy full SHA for c77012e

• Elide ORDER BY cnt DESC, as the PK overrules it

  jhanley634

  committed
  2b1fa5c

  Copy full SHA for 2b1fa5c

• Expand the docstring critique to mention the integer ID comparison is against large_movie_id

  jhanley634

  committed
  fe5272d

  Copy full SHA for fe5272d

• Make a range check a little looser

  jhanley634

  committed
  d131f2a

  Copy full SHA for d131f2a

• Add _normalize_rating

  jhanley634

  committed
  ffe34a0

  Copy full SHA for ffe34a0

• Add comments

  jhanley634

  committed
  410b8b7

  Copy full SHA for 410b8b7

• The ETL glob parameter, for rapid test runs, has outlived its usefulness; replace with a constant

  jhanley634

  committed
  94e6b89

  Copy full SHA for 94e6b89

• Default to full 101 M row ETL, skipping it if rows are already there

  jhanley634

  committed
  4df2937

  Copy full SHA for 4df2937

• LightFM is returning somewhat deterministic results now, so test that the expected recommended titles came back

  jhanley634

  committed
  84d84f2

  Copy full SHA for 84d84f2

• Ignore Private Ryan noise

  jhanley634

  committed
  8bc94c6

  Copy full SHA for 8bc94c6

• Add pipenv script to browse through ETL'd data.

  jhanley634

  committed
  f4b008a

  Copy full SHA for f4b008a

• Add EDA convenience view: rating_v

  jhanley634

  committed
  2a3fcd1

  Copy full SHA for 2a3fcd1

Commits on Feb 9, 2025

• Rename to RATING_V_DDL (data definition language)

  jhanley634

  committed
  1aa1f5a

  Copy full SHA for 1aa1f5a