pep_sex_2024 changes made #1110
Conversation
|
@kurus21 Can you remove input & output folder and confirm? |
| skiprows=7, | ||
| skipfooter=102, | ||
| header=None) | ||
| df.columns = [ |
There was a problem hiding this comment.
pls use df.rename() instead of assuming column order.
There was a problem hiding this comment.
As discussed over chat rename method doesn't gives an upper hand since that approach has also demands to assume the rows and cols position.
| skipfooter=102, | ||
| header=None) | ||
| df.columns = [ |
There was a problem hiding this comment.
As discussed over chat rename method doesn't gives an upper hand since that approach has also demands to assume the rows and cols position.
| 'White Total', 'White Male', 'White Female', 'NonWhite Total', | ||
| 'NonWhite Male', 'NonWhite Female' | ||
| ] | ||
| df = df.drop(columns=[ |
There was a problem hiding this comment.
more readable to list columns of interest to be retained:
df.drop(columns=df.columns.difference(['Count_Person_Male', 'Count_Person_Female']), inplace=True)
Then it can be moved outside the if/else block
There was a problem hiding this comment.
It has been modified accordingly.
| # adding geoid, year and measurement method | ||
| df['Year'] = year | ||
| df.insert(0, 'geo_ID', 'country/USA', True) | ||
| df['Measurement_Method'] = 'dcAggregate/CensusPEPSurvey_PartialAggregate' |
There was a problem hiding this comment.
This seems common to both if and else and can be moved out.
There was a problem hiding this comment.
It has been modified accordingly.
| for col in float_col.columns.values: | ||
| df[col] = df[col].astype('int64') | ||
| df[col] = df[col].astype("str").str.replace("-1", "") | ||
| df.rename(columns={'SEX': 'Year'}, inplace=True) |
There was a problem hiding this comment.
Why is the column 'SEX' being renamed to 'Year' here and in functions below.
There was a problem hiding this comment.
It has been renamed accordingly to match the data frame after modification.
| 'POPEST_FEM': 'Count_Person_Female', | ||
| 'YEAR': 'Year' | ||
| }) | ||
| df = df.drop(columns=[ |
There was a problem hiding this comment.
may be easier to to do df.drop(columns=df.columns.difference([])..)
| 'Count_Person_Male', 'Count_Person_Female' | ||
| ] | ||
| df = pd.read_excel(file_path, skiprows=5, skipfooter=7, header=None) | ||
| df.columns = column_name |
| 'July2022Female', | ||
| 'July2023Male', | ||
| 'July2023Female', | ||
| '2023Total', |
There was a problem hiding this comment.
Can we generalize this to 2024 and future years?
There was a problem hiding this comment.
It has been generalized for future years
| "sc-est2023-syasex-": _state_2023, | ||
| "sc-est2023-agesex-": _state_2023, | ||
| "cc-est2023-agesex-": _county_2023, | ||
| "cc-est2023-agesex-a": _county_2023 |
There was a problem hiding this comment.
can we also extend to handle future years assuming the same format?
| return df | ||
| except Exception as e: | ||
| logging.fatal(f"Error processing the file {file_path}: {e}") | ||
| except Exception as e: |
There was a problem hiding this comment.
multiple except block are there. Remove the duplicate ones.
| return df | ||
| try: | ||
| df = pd.read_csv(file_path, thousands=',', skiprows=4, header=None) | ||
| df.columns = [ |
There was a problem hiding this comment.
Consider implementing a more dynamic approach to identify the required columns instead of hardcoding their order.?
There was a problem hiding this comment.
We have to assume the position of the columns [rows and cols] anyway. So it has been hard coded like other places.
Please be informed it has been handled with try and catch block anyway.
There was a problem hiding this comment.
Kuru, could you add a comment to the script explaining the reason for fixing the column order? This will help future developers understand the rationale behind the change
|
|
Please be informed that the comments has been updated. |
No description provided.