Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history
commit a218426
Author: jchristel <[email protected]>
Date:   Sun Dec 8 11:24:13 2024 +1100

    updated samples to work with new files_combine

commit 342f01e
Author: jchristel <[email protected]>
Date:   Sun Dec 8 11:23:45 2024 +1100

    refactor of files combined functions

    trying to fix empty row bug
    removed basic function since almost the same as the default file_combine
    returns now a result object...
    added custom delimeter support
    added quoting support
    changed input parameter name from out_put_file_name to output_file_name

commit c29f81a
Author: Jan <[email protected]>
Date:   Sat Dec 7 17:57:20 2024 +1100

    Update family_base_data_missing_families.py

    code formatting update

commit ec3220a
Author: jchristel <[email protected]>
Date:   Sat Dec 7 17:51:45 2024 +1100

    Update FamilyFindMissingTests.cs

    some additional log output added

commit 62ed4b1
Author: jchristel <[email protected]>
Date:   Sat Dec 7 17:51:30 2024 +1100

    Update family_base_data_missing_families.py

    fixed bug in getting direct root families of missing families

commit 24b6d89
Author: Jan Christel <[email protected]>
Date:   Fri Dec 6 15:39:33 2024 +1100

    code clean up

commit 8e4bd02
Author: Jan Christel <[email protected]>
Date:   Fri Dec 6 09:23:24 2024 +1100

    updated is perpendicular tests

commit 774a7a7
Merge: 114f59b e125202
Author: jchristel <[email protected]>
Date:   Thu Dec 5 20:25:52 2024 +1100

    Merge branch 'wip-1.1.10' of https://github.com/jchristel/SampleCodeRevitBatchProcessor into wip-1.1.10

commit 114f59b
Author: jchristel <[email protected]>
Date:   Thu Dec 5 20:25:48 2024 +1100

    Update FamiliesSelectionViewModel.py

    filter is now working!

commit 2766051
Author: jchristel <[email protected]>
Date:   Thu Dec 5 20:25:30 2024 +1100

    Delete FilterItem.py

commit 9573dfb
Author: jchristel <[email protected]>
Date:   Thu Dec 5 20:25:24 2024 +1100

    Update FamiliesSelectionView.xaml

    added vertical scroll bar to context menu

commit 9745961
Author: jchristel <[email protected]>
Date:   Thu Dec 5 20:24:57 2024 +1100

    Create FilterItem.py

commit e125202
Author: Jan Christel <[email protected]>
Date:   Thu Dec 5 17:00:59 2024 +1100

    used class to reduce code duplication

commit 269065a
Author: jchristel <[email protected]>
Date:   Wed Dec 4 21:41:41 2024 +1100

    FamilyReloaderSample added context filter menu

    Added a context filter menu for each property ... now I just have to filter the view...

commit 0aac23e
Author: jchristel <[email protected]>
Date:   Wed Dec 4 20:51:50 2024 +1100

    Family Reloader

    adding grouping and start of column filtering to view model

commit ed27800
Author: Jan Christel <[email protected]>
Date:   Wed Dec 4 14:03:05 2024 +1100

    food for thought

commit 3c20126
Author: Jan Christel <[email protected]>
Date:   Wed Dec 4 14:01:24 2024 +1100

    fixing up classes RoomSpatial Classes

commit e822060
Author: Jan <[email protected]>
Date:   Wed Dec 4 09:05:49 2024 +1100

    Update family_data_family.py

    added custom equal

commit deab4a2
Author: Jan Christel <[email protected]>
Date:   Tue Dec 3 14:08:34 2024 +1100

    switch on whether to use a temp dir

commit a874b32
Author: Jan Christel <[email protected]>
Date:   Mon Dec 2 13:53:13 2024 +1100

    Create RoomSpatialForViews.py

    set up new room class to help with rls view creation

commit c080925
Author: Jan Christel <[email protected]>
Date:   Mon Dec 2 13:52:52 2024 +1100

    Update RoomSpatialObject.py

    commented out not required ini line

commit 0595637
Author: Jan Christel <[email protected]>
Date:   Mon Dec 2 13:52:29 2024 +1100

    Update curve.py

    added line related :
    are_lines_parallel, are_lines_perpendicular

commit 234274d
Author: jchristel <[email protected]>
Date:   Mon Dec 2 09:11:12 2024 +1100

    bug hunting

commit f9ed4d0
Author: jchristel <[email protected]>
Date:   Mon Dec 2 08:07:18 2024 +1100

    Update FamilyFindMissingTests.cs

    added TestFindMissingFamiliesDirectHostFamilies()

commit aa8e565
Author: jchristel <[email protected]>
Date:   Mon Dec 2 08:07:04 2024 +1100

    Update family_base_data_missing_families.py

    doc string updates,
    bug fix in def find_missing_families_direct_host_families

commit 6837503
Author: Jan Christel <[email protected]>
Date:   Sun Dec 1 16:55:04 2024 +1100

    Update setup.cfg
  • Loading branch information
jchristel committed Dec 8, 2024
1 parent 7832ae1 commit a3eb071
Show file tree
Hide file tree
Showing 21 changed files with 1,170 additions and 251 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@
# import System
import sys # required to return an exit code
import os

from csv import QUOTE_MINIMAL
# import common library
import settings as settings # sets up all commonly used variables and path locations!

Expand Down Expand Up @@ -102,7 +102,12 @@ def merge_files():
"""
Appends temp report files to log files. Returns a list of files where that log file did not exist or which failed to append to data log file
:return: List of files
:param file_name: file name without path and extension
:type file_name: str
:param filter: file name ends on: filter
:type filter: str
:return: List of files as fully qualified file names which failed to append to data log file
:rtype: [str]
"""

Expand All @@ -117,7 +122,11 @@ def merge_files():
output("Need to create data file: {}".format(data_file_name))
try:
write_report_data_as_csv(
file_name=data_file_name, header=rFns.LOG_FILE_HEADER, data=[]
file_name=data_file_name,
header=rFns.LOG_FILE_HEADER,
data=[],
enforce_ascii=True,
quoting=QUOTE_MINIMAL,
)
except Exception as e:
output(
Expand All @@ -139,45 +148,67 @@ def merge_files():
file_without_ext = get_file_name_without_ext(file_match)

# append single temp file to data log file
flag_append = append_to_file(data_file_name, file_match, True)
if flag_append:
result_append = append_to_file(data_file_name, file_match, True)
if result_append.status:
output("Appended: {} to: {}".format(file_without_ext, log_file_name))
else:
output(
"Failed to append: {} to: {}".format(
file_without_ext, log_file_name
"Failed to append: {} to: {} with message: {}".format(
file_without_ext, log_file_name, result_append.message
)
)
failed_files.append(file_match)
return failed_files


def append_files(
folder_path, file_prefix, file_suffix, file_extension, out_put_file_name
def append_files_wrapper(
folder_path, file_prefix, file_suffix, file_extension, output_file_name, **kwargs
):
"""
DuHast append file wrapper...
Used to append warnings to warnings report
:param folder_path: Directory path where the files are located
:type folder_path: str
:param file_prefix: The file prefix common between files to be combined
:type file_prefix: str
:param file_suffix:The file suffix common between files to be combined
:type file_suffix: str
:param file_extension: The file extension of the files to be combined
:type file_extension: str
:param output_file_name: The name of the file to be created
:type output_file_name: str
"""

file_list = get_files_single_directory(
folder_path, file_prefix, file_suffix, file_extension
)

# check if any files were found in the directory
if len(file_list) == 0:
output(
"No files found with prefix: {} suffix: {} extension: {}".format(
file_prefix, file_suffix, file_extension
)
)
return

# build fully qualified out put file name
full_out_file_name = os.path.join(settings.OUTPUT_FOLDER, out_put_file_name)
full_out_file_name = os.path.join(settings.OUTPUT_FOLDER, output_file_name)

for file in file_list:
append_flag = append_to_file(
append_result = append_to_file(
source_file=full_out_file_name, append_file=file, ignore_first_row=True
)
output(
"...appended {} to {} with status [{}]".format(
file, full_out_file_name, append_flag
file, full_out_file_name, append_result.status
)
)

def combine_csv_files(folder_path, file_prefix, file_suffix, file_extension, out_put_file_name):
def combine_csv_files_wrapper(folder_path, file_prefix, file_suffix, file_extension, output_file_name, overwrite_existing, **kwargs):
"""
Combines csv files into a single csv file with header independent of the files being combined.
Expand All @@ -189,22 +220,57 @@ def combine_csv_files(folder_path, file_prefix, file_suffix, file_extension, out
:type file_suffix: str
:param file_extension: The file extension of the files to be combined
:type file_extension: str
:param out_put_file_name: The name of the file to be created
:type out_put_file_name: str
:param output_file_name: The name of the file to be created
:type output_file_name: str
"""

combine_files_csv_header_independent(
try:
combine_files_csv_header_independent(
folder_path=folder_path,
file_prefix=file_prefix,
file_suffix=file_suffix,
file_extension=file_extension,
output_file_name=output_file_name,
overwrite_existing=overwrite_existing,
)
except Exception as e:
output(
"Failed to combine files {} with exception: [{}]".format(file_suffix, e)
)

def combine_files_wrapper(folder_path,file_prefix,file_suffix,file_extension,output_file_name, **kwargs):
"""
Combines files into a single file with a common header.
:param folder_path: Directory path where the files are located
:type folder_path: str
:param file_prefix: The file prefix common between files to be combined
:type file_prefix: str
:param file_suffix:The file suffix common between files to be combined
:type file_suffix: str
:param file_extension: The file extension of the files to be combined
:type file_extension: str
:param output_file_name: The name of the file to be created
:type output_file_name: str
"""

result_combine = combine_files(
folder_path=folder_path,
file_prefix=file_prefix,
file_suffix=file_suffix,
file_extension=file_extension,
out_put_file_name=out_put_file_name,
overwrite_existing=True,
output_file_name=output_file_name,
)

output(
"...combined {} to {} with status [{}]".format(
file_suffix, output_file_name, result_combine.status
)
)


def combine_data_files():
"""
Combines varies report files which are created per Revit project file into a single text file
Combines varies report files which are created per Revit project file into a single text file.
"""
for file_to_combine in FILE_DATA_TO_COMBINE:
output("Combining {} report files.".format(file_to_combine[0]))
Expand All @@ -213,7 +279,7 @@ def combine_data_files():
file_prefix="",
file_suffix=file_to_combine[0],
file_extension=settings.REPORT_FILE_NAME_EXTENSION,
out_put_file_name=file_to_combine[1],
output_file_name=file_to_combine[1],
overwrite_existing=True, # make sure previous files are overwritten
)

Expand All @@ -223,32 +289,32 @@ def combine_data_files():
[
settings.REPORT_EXTENSION_SHEETS_SHORT,
settings.COMBINED_REPORT_NAME_SHEETS_SHORT,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_SHEETS,
settings.COMBINED_REPORT_NAME_SHEETS,
combine_csv_files,
combine_csv_files_wrapper,
],
[
settings.REPORT_EXTENSION_SHARED_PARAMETERS,
settings.COMBINED_REPORT_NAME_SHARED_PARAMETERS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_GRIDS,
settings.COMBINED_REPORT_NAME_GRIDS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_LEVELS,
settings.COMBINED_REPORT_NAME_LEVELS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_WORKSETS,
settings.COMBINED_REPORT_NAME_WORKSETS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_GEO_DATA,
Expand All @@ -258,37 +324,37 @@ def combine_data_files():
[
settings.REPORT_EXTENSION_FAMILIES,
settings.COMBINED_REPORT_NAME_FAMILIES,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_MARKED_VIEWS,
settings.COMBINED_REPORT_NAME_MARKED_VIEWS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_WALL_TYPES,
settings.COMBINED_REPORT_NAME_WALL_TYPES,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_VIEWS,
settings.COMBINED_REPORT_NAME_VIEWS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_CAD_LINKS,
settings.COMBINED_REPORT_NAME_CAD_LINKS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_REVIT_LINKS,
settings.COMBINED_REPORT_NAME_REVIT_LINKS,
combine_files,
combine_files_wrapper,
],
[
settings.REPORT_EXTENSION_WARNING_TYPES,
settings.COMBINED_REPORT_NAME_WARNING_TYPES,
append_files,
append_files_wrapper,
],
]

Expand All @@ -298,19 +364,22 @@ def combine_data_files():

exit_code = 0
try:
# merge revit model health data files into overall .log file
failed_files_ = merge_files()
except Exception as e:
output("Failed to merge files: [{}]".format(e))
exit_code = 1

try:
# combine data files per project file and data point into single data files per data point
output("Combining report files:")
combine_data_files()
except Exception as e:
output("Failed to combine report files: [{}]".format(e))
exit_code = 1

try:
# create view template hash table files
output("Creating view template hash table:")
combine_vt_data_result = combine_vt_reports(settings.OUTPUT_FOLDER)
output(
Expand All @@ -324,6 +393,9 @@ def combine_data_files():


try:
# convert files into parquet file format
# this required python 3.10or higher and cant be run in the same post process script as the other tasks
# TODO: move into separate script
output("Converting view template hash table files to parquet file format:")
convert_to_parquet_result = convert_vt_reports_to_parquet(settings.OUTPUT_FOLDER)
output(
Expand Down
8 changes: 4 additions & 4 deletions Samples/Flows/Post_CombineReports.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,23 +109,23 @@ def output(message = ''):
DATE_STAMP = dateStamp.get_file_date_stamp()

# combine report files based on:
fileCombine.combine_files(
combine_status_cad = fileCombine.combine_files(
ROOT_PATH, # - part report location
DATE_STAMP, # - part report prefix ( same date stamp as current)
'_CAD', # - part report file name suffix
'.txt', # - part report file extension
DATE_STAMP + '_CAD_Links_summary.txt' # - combined report file name in same location as part reports
)
# notify users
output('Writing summary Data.... finished: {}_CAD_Links_summary.txt'.format(DATE_STAMP))
output('Writing summary Data.... finished:[{}] {}_CAD_Links_summary.txt'.format(combine_status_cad.status, DATE_STAMP))

# combine report files based on:
fileCombine.combine_files(
combine_status_revit = fileCombine.combine_files(
ROOT_PATH, # - part report location
DATE_STAMP, # - part report prefix ( same date stamp as current)
'_RVT', # - part report file name suffix
'.txt', # - part report file extension
DATE_STAMP + '_RVT_Links_summary.txt' # - combined report file name in same location as part reports
)
# notify user
output('Writing summary Data.... finished: {}_RVT_Links_summary.txt'.format(DATE_STAMP))
output('Writing summary Data.... finished:[{}] {}_RVT_Links_summary.txt'.format(combine_status_revit.status, DATE_STAMP))
Original file line number Diff line number Diff line change
Expand Up @@ -126,17 +126,20 @@ def delete_temp_files(keep_files):

def combine_data_files():
"""
Combines varies report files into single text file.
Combines varies report files into single csv text file.
Files are filter based on FILE_DATA_TO_COMBINE list.
"""

for to_combine in FILE_DATA_TO_COMBINE:
output("Combining {} report files.".format(to_combine[0]))
# combine files
combine_files(
combine_result = combine_files(
settings.WORKING_DIRECTORY, "", to_combine[0], ".temp", to_combine[1]
)
output("Combined report files. [{}]".format(combine_result.status))
if not combine_result.status:
output("Combined report file: {}".format(combine_result.message))


def move_files():
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -125,21 +125,24 @@ def delete_temp_files(keep_files):

def combine_data_files():
"""
Combines varies report files into single text file.
Combines varies report files into single csv text file.
Files are filter based on FILE_DATA_TO_COMBINE list.
"""

for to_combine in settings.FILE_DATA_TO_COMBINE:
output("Combining {} report files.".format(to_combine[0]))
# combine files
combine_files(
combine_result = combine_files(
settings.WORKING_DIRECTORY,
"",
to_combine[0],
settings.TEMP_FILE_EXTENSION,
to_combine[1],
)
output("Combined report files. [{}]".format(combine_result.status))
if not combine_result.status:
output("Combined report file: {}".format(combine_result.message))


def move_files():
Expand Down
Loading

0 comments on commit a3eb071

Please sign in to comment.