Revise travel day #82

sgreenbury · 2025-01-08T18:34:10Z

This PR:

Adds Configurable common day of travel for households and revise the filtering of NTS trips to before matching to increase matches with non-missing trips
Updates modelling of choosing a sample travel day for individuals given pwkstat in SPC
Adds initial approach for matching for unmatched individuals after household matching
Revises the variable used for region to "PSUStatsReg_B01ID" as this is included for 2021 and 2022

sgreenbury · 2025-01-16T17:56:16Z

scripts/2_match_households_and_individuals.py

+
+    # Ensure that the households have at least one day in `nts_days_of_week` that
+    # all household members have trips for
+    if config.parameters.common_household_day:


The new parameter boolean common_household_day determines whether all individuals of the household need to have a TravDay in common.

sgreenbury · 2025-01-16T17:57:43Z

scripts/2_match_households_and_individuals.py

+    # Ensure that the households have at least one day in `nts_days_of_week` that
+    # all household members have trips for
+    if config.parameters.common_household_day:
+        hids = households_with_common_travel_days(


Gets the subset of households where all individuals have a common TravDay that is in the set of configured days (config.parameters.nts_days_of_week)

sgreenbury · 2025-01-16T17:58:40Z

scripts/2_match_households_and_individuals.py

+            nts_trips, config.parameters.nts_days_of_week
+        )
+    else:
+        hids = households_with_travel_days_in_nts_weeks(


Gets the subset of households where all individuals have any TravDay that is in the set of configured days (config.parameters.nts_days_of_week)

sgreenbury · 2025-01-16T18:00:13Z

scripts/2_match_households_and_individuals.py

+            nts_trips, config.parameters.nts_days_of_week
+        )
+
+    # Subset individuals and households given filtering of trips


Subset to the households subset above before matching to ensure matches have required TravDays

sgreenbury · 2025-01-16T18:02:21Z

scripts/2_match_households_and_individuals.py

+        # match remaining individuals
+        remaining_ids = spc_edited.loc[
+            ~spc_edited.index.isin(matches_ind.keys()), "id"
+        ].to_list()
+        matches_remaining_ind = match_remaining_individuals(
+            df1=spc_edited,
+            df2=nts_individuals,
+            matching_columns=["age_group", "sex"],
+            remaining_ids=remaining_ids,
+            show_progress=True,
+        )
+        matches_ind.update(matches_remaining_ind)
+


Add matching for any remaining individuals that were part of unmatched households. It might be worth considering if this should be more configurable.

src/acbm/preprocessing.py

sgreenbury · 2025-01-16T18:06:01Z

src/acbm/preprocessing.py

+    # PSUGOR_B02ID but does not have values for 2021 and 2022
+    # region_dict = {
+    #     -10.0: "DEAD",
+    #     -9.0: "DNA",
+    #     -8.0: "NA",
+    #     1.0: "North East",
+    #     2.0: "North West",
+    #     3.0: "Yorkshire and the Humber",
+    #     4.0: "East Midlands",
+    #     5.0: "West Midlands",
+    #     6.0: "East of England",
+    #     7.0: "London",
+    #     8.0: "South East",
+    #     9.0: "South West",
+    #     10.0: "Wales",
+    #     11.0: "Scotland",
+    # }
+
+    # PSUStatsReg_B01ID but does not have values for 2021 and 2022


Could be made configurable or to removed before merging.

sgreenbury · 2025-01-16T18:06:26Z

src/acbm/preprocessing.py

    # In the PSU table, create a column with the region names
-    psu["region_name"] = psu["PSUGOR_B02ID"].map(region_dict)
+    # psu["region_name"] = psu["PSUGOR_B02ID"].map(region_dict)


sgreenbury · 2025-01-16T18:07:28Z

src/acbm/preprocessing.py

+        1.0: "Northern, Metropolitan",
+        2.0: "Northern, Non-metropolitan",
+        3.0: "Yorkshire / Humberside, Metropolitan",
+        4.0: "Yorkshire / Humberside, Non-metropolitan",
+        5.0: "East Midlands",
+        6.0: "East Anglia",
+        7.0: "South East (excluding London Boroughs)",
+        8.0: "London Boroughs",
        9.0: "South West",
-        10.0: "Wales",
-        11.0: "Scotland",
+        10.0: "West Midlands, Metropolitan",
+        11.0: "West Midlands, Non-metropolitan",
+        12.0: "North West, Metropolitan",
+        13.0: "North West, Non-metropolitan",
+        14.0: "Wales",
+        15.0: "Scotland",


Revised regions using field "PSUStatsReg_B01ID" as this provides values for 2021 and 2022.

sgreenbury · 2025-01-16T18:12:34Z

scripts/3.1_assign_primary_feasible_zones.py

-    ]
+
+    # Generate random sample of days by household
+    get_chosen_day(config).to_parquet(


Get a chosen day for each individual to represent a "sample" day given the configured days of the week and whether the household is configured to share a common day.

sgreenbury · 2025-01-16T18:15:40Z

src/acbm/assigning/utils.py

+            # If pwkstat = 1 (full time)
+            # and a work travel day is available
+            pl.when(pl.col("pwkstat").eq(1) & pl.col("TravDayWork").is_not_null())
+            .then(pl.col("TravDayWork"))
+            .otherwise(
+                # If pwkstat = 1 (full time)
+                # and a work travel day is NOT available
+                pl.when(pl.col("pwkstat").eq(1) & pl.col("TravDayWork").is_null())
+                .then(pl.col("TravDayAny"))
+                .otherwise(
+                    # If pwkstat = 2 (part time)
+                    # and a work travel day is available
+                    # and a non-work travel day is available
+                    pl.when(
+                        pl.col("pwkstat").eq(2)
+                        & pl.col("TravDayWork").is_not_null()
+                        & pl.col("TravDayNonWork").is_not_null()
+                    )
+                    .then(
+                        # Sample either TravDayWork or TravDayNonWork
+                        # stochastically given config
+                        pl.col("TravDayWork")
+                        # TODO: update from config
+                        if np.random.random() < 1
+                        else pl.col("TravDayNonWork")
+                    )
+                    .otherwise(pl.col("TravDayAny"))
+                )
+            )
+            .alias("ChosenTravDay")


Samples a chosen day given an individual's pwkstat value to increase the likelihood of choosing a day that includes a work trip.

sgreenbury · 2025-01-17T10:44:45Z

scripts/2_match_households_and_individuals.py

+        matches_remaining_ind = match_remaining_individuals(
+            df1=spc_edited,
+            df2=nts_individuals,
+            matching_columns=["age_group", "sex"],


Could update the matching_columns here to enable more precision when not using households: e.g. for employment status and urban rural classification.

sgreenbury changed the base branch from main to 53-paths-with-79 January 8, 2025 18:34

sgreenbury added 6 commits January 14, 2025 18:39

Adjust matching to account for common travel day

6164a28

Add 'DayID' to outputs

0cea61f

Fix missing subset of nts_trips

88f4911

Add interim output with chosen random day for households

e0fc2f9

Add todo

ed45b92

Handle missing data and revise test for common trav days

ce5a261

sgreenbury force-pushed the 53-paths-with-79-revise-trav-day branch from c122be5 to ce5a261 Compare January 14, 2025 18:39

sgreenbury added 9 commits January 14, 2025 18:42

Fix test

ced648d

Add chosen travel day modelling using pwkstat

5f17c56

Revise subsetting of households given trav day config

4b6db8c

Fix test

1f5e0b9

Add logging for NTS filtering

4ce4b1a

Fix merge columns

46946d1

Add matching for remaining individuals

4301493

Revise region variable to 'PSUStatsReg_B01ID'

be2eb6b

Fix logging

748260a

sgreenbury commented Jan 16, 2025

View reviewed changes

src/acbm/preprocessing.py Outdated Show resolved Hide resolved

Remove obsolete comment

5105dab

sgreenbury commented Jan 16, 2025

View reviewed changes

Uncomment code

32dfe63

sgreenbury marked this pull request as ready for review January 16, 2025 20:51

Limit the number of processes to proportion of cpu_count()

6ebd5be

sgreenbury commented Jan 17, 2025

View reviewed changes

Only consider working not from home

1dd281b

sgreenbury force-pushed the 53-paths-with-79-revise-trav-day branch from b210e29 to 1dd281b Compare January 22, 2025 18:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revise travel day #82

Revise travel day #82

sgreenbury commented Jan 8, 2025 •

edited

Loading

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 16, 2025

sgreenbury Jan 17, 2025

Revise travel day #82

Are you sure you want to change the base?

Revise travel day #82

Conversation

sgreenbury commented Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sgreenbury commented Jan 8, 2025 •

edited

Loading