Better handling of rejected lines and preparation of resubmission is …

…complete. Also improved documentation.

README.md

@@ -120,6 +120,8 @@ python zymoTransmit.py -s
 ```
 There are several codes related to SARS-CoV-2 testing available, so please look carefully for the one that most closely describes your test method and sample.
 
+When preparing this report, please avoid the use of placeholders for missing or non-applicable data; **if something has no value, either due to being non-applicable or missing, leave it blank**!  This will avoid any confusion or potential to treat a value such as "N/A" as real data.  Several potential result terms have already been programmed in to be interpreted by the program as positive, negative, or other results.  These terms can be seen in the config.py file in zymoTransmitSupport and terms can be added or removed if needed.
+
 
 #### Transmitting results
 Once a report has been filled out correctly, submitting reports requires a simple command:
@@ -144,13 +146,15 @@ python zymoTransmit.py [argument] [filename if needed]
 --testConnection	|	-t	|	Tests the connection to the health department, run as a final step of setup
 --snomed	|	-s	|	Display relevant SNOMED codes for specimen types
 --loinc	|	-l	|	Display relevant LOINC codes for testing types
+--cdph  | | Accept a CDPH-formatted CSV file (run using **CDPH.bat**)
 
 
 ## OUTPUT
-All outputs will be written to the designated output folder for the container, which will unmount upon completion of the run.
+The key output is sent over the Internet to the agency receiving the test reports.  Local outputs will be as follows:
+
+The transmission logs folder will get two files per session.  Both will contain timestamps in the name and one will be called resultText[*timestamp*].hl7, which will contain the raw HL7 data transmitted during the session.  The other will be called submissionLog[*timestamp*].txt and will contain receipts for each patientID:specimenID combination sent.  **If a submission was rejected by the gateway, the reason(s) will be found in this file.**
 
-There will be two primary output files, one report in HTML format and one in JSON format.  Examples of these can be seen [here for HTML](https://github.com/Zymo-Research/miqScoreShotgunPublic/blob/master/exampleReport.html) and [here for JSON](https://github.com/Zymo-Research/miqScoreShotgunPublic/blob/master/exampleReport.json).  The HTML report is designed to be viewed in a browser and gives an overview of the results for the sample.  The JSON report, while human-readable, is designed primarily to facilitate analysis using an automated script.  It also provides much more detailed information on the results than the HTML report.  
-In addition to the two primary files, there will be a log file that can be used in the event of a problem with analysis for additional information on the run.  Finally, there will be several files generated by the DADA2 pipeline.  If you are familiar with DADA2, you will be familiar with these outputs.
+An additional file called rejects.csv will be created at the start of each run in the program root folder if it does not already exist.  This file will contain lines for any results that were either unable to be converted to HL7 (often due to uninterpretable results) or any lines that were not successfully transmitted through the gateway (this could be due to a failure in the gateway itself or something invalid in the data).  Check the submissionLog file mentioned in the previous paragraph to determine if corrections are needed before retransmission.  This CSV file can be opened in a spreadsheet application (such as Microsoft Excel) to fix any incorrect information.  The file can then be renamed, keeping the CSV extension and run through the program as new data of the original format and a new rejects file will be started automatically.  This practice should make retransmission of failed attempts easier for the user while minimizing duplicate transmissions seen by the gateway.
 
 ## Contributing
 
@@ -166,6 +170,12 @@ Major release: Newly required parameter or other change that is not entirely bac
 Minor release: New optional parameter
 Patch release: No changes to parameters
 
+## Current Version
+
+The current major release contains all initial functionality plus some additional features designed to help the California Department of Public Health clear backlogged data using their existing form.  Many of these extended features will also help other users with their submissions.  Essential features covered include transmission of data from this program's preferred table format (either CSV or tab-delimited text), transmission of a large block of HL7 data, transmission of a folder with individual HL7 data blocks, transmission of a CDPH-formatted CSV file, and handling of results that fail to transmit for various reasons.
+
+This major release has been nicknamed Imahara's Pudding Cup after [Grant Imahara](https://en.wikipedia.org/wiki/Grant_Imahara), a popular advocate of STEAM education and performing random acts of kindness for others.
+
 ## Authors
 
 - **Michael M. Weinstein** - *Project Lead, Programming and Design* - [michael-weinstein](https://github.com/michael-weinstein)

zymoTransmit.py

@@ -109,6 +109,8 @@ def __init__(self):
         if not self.hl7Directory and not os.path.isfile(inputValue):
             raise FileNotFoundError("No such file %s" %inputValue)
         self.input = inputValue
+        if os.path.abspath(self.input) == os.path.abspath(os.path.join(contentRoot, "rejects.csv")):
+            raise RuntimeError("Please avoid running this program directly on the rejects.csv file it creates.  Resubmitting this file after correcting any issues IS recommended, but only after renaming that file to something else.")
         if convertCertificate:
             convertPFX(self.input)
             input("Press enter to quit.")
@@ -127,7 +129,6 @@ def getTestResults(testResultPath:str="results.txt", cdphCSV:bool=False):
 
 def makeHL7Codes(resultList:typing.List[zymoTransmitSupport.inputOutput.resultReader.TestResult]):
     hl7Sets = {}
-    skippedData = []
     for result in resultList:
         patientID = result.patientID
         specimenID = result.specimenID
@@ -144,11 +145,10 @@ def makeHL7Codes(resultList:typing.List[zymoTransmitSupport.inputOutput.resultRe
             currentSet.append(zymoTransmitSupport.hl7Encoder.encoders.makeNTELine(result))
         if not result.okToTransmit:
             print("Skipping preparation of %s:%s for the following reasons:" %(result.patientID, result.specimenID))
-            for reason in result.reasonsNotToTransmit:
+            for reason in result.reasonForFailedTransmission:
                 print("\t%s" %reason)
             del hl7Sets[(patientID, specimenID)]
-            skippedData.append((patientID, specimenID))
-    return hl7Sets, skippedData
+    return hl7Sets
 
 
 def makeHL7Blocks(hl7Sets:typing.Dict[typing.Tuple[str, str], typing.List[zymoTransmitSupport.hl7Encoder.generics.Hl7Line]]):
@@ -167,14 +167,14 @@ def makeHL7TextRecord(hl7Blocks:typing.Dict[typing.Tuple[str, str], str]):
     return textRecord
 
 
-def processRejects(resultList:typing.List[zymoTransmitSupport.inputOutput.resultReader.TestResult]):
+def processRejects(resultList:typing.List[zymoTransmitSupport.inputOutput.resultReader.TestResult], delimiter:str="\t"):
     file = open(os.path.join(contentRoot, "rejects.csv"), 'a', newline="")
     for result in resultList:
         csvHandle = csv.writer(file)
-        if result.okToTransmit:
+        if result.okToTransmit and result.transmittedSuccessfully:
             continue
         if type(result.rawLine) == str:
-            print(result.rawLine, file=file, end="\n")
+            csvHandle.writerow(result.rawLine.split(delimiter))
         else:
             csvHandle.writerow(result.rawLine)
     file.close()
@@ -211,17 +211,22 @@ def prepareAndSendResults(args:CheckArgs):
             hl7TextBlocks = zymoTransmitSupport.inputOutput.rawHL7.textBlocksFromRawHL7(args.input)
         else:
             resultList = getTestResults(args.input, args.cdph)
-            hl7Sets, skippedData = makeHL7Codes(resultList)
+            hl7Sets = makeHL7Codes(resultList)
             hl7TextBlocks = makeHL7Blocks(hl7Sets)
     hl7TextRecord = makeHL7TextRecord(hl7TextBlocks)
     if not args.noTransmit:
-        transmissionResults = zymoTransmitSupport.inputOutput.soapAPI.transmitBlocks(client, hl7TextBlocks)
+        transmissionResults = zymoTransmitSupport.inputOutput.soapAPI.transmitBlocks(client, hl7TextBlocks, resultList)
         resultText = zymoTransmitSupport.inputOutput.logger.writeLogFile(config.Configuration.logFolder, transmissionResults, hl7TextRecord)
         print(resultText)
+        for result in resultList:
+            if not (result.okToTransmit and result.transmittedSuccessfully):
+                skippedData.append((result.patientID, result.specimenID, result.reasonForFailedTransmission))
         if skippedData:
-            print("WARNING: Some results were skipped for reasons listed above:")
-            for patientID, specimenID in skippedData:
-                print("%s:%s was skipped" %(patientID, specimenID))
+            print("\n\nWARNING: SOME RESULTS WERE SKIPPED, FAILED TO TRANSMIT, OR WERE REJECTED BY THE GATEWAY FOR REASONS BELOW:\n")
+            for patientID, specimenID, reasons in skippedData:
+                print("%s:%s was not successfully transmitted because:" %(patientID, specimenID))
+                for reason in reasons:
+                    print("\t%s" %reason)
     else:
         print("Results not transmitted due to argument noTransmit being set to true.")
     if skippedData:
@@ -237,7 +242,6 @@ def makeDirectoriesIfNeeded():
         file = open(os.path.join(contentRoot, "rejects.csv"), 'w')
         file.write("#Header for lines that were not transmitted due to issue with interpretation\n")
         file.close()
-        print("something")
 
 
 class PlaceHolderException(Exception):

zymoTransmitSupport/hl7Encoder/encoders.py

@@ -220,7 +220,7 @@ def makeObservationValueAndAbnormalityObjects(resultString:str):
             print("Unable to classify result '%s' for patient %s specimen %s. Preferred terms are: detected, indeterminate, negative, and unsatisfactory specimen." %(resultString, result.patientID, result.specimenID))
             resultTerm = ""
             result.okToTransmit = False
-            result.reasonsNotToTransmit.append("Failed to interpret result value")
+            result.reasonForFailedTransmission.append("Failed to interpret result value. Please modify config.py to interpret the result value or modify the result value to something that is already interpreted.")
         return (observedResults.getObservationValue(resultTerm),
                 observedResults.getAbnormalityObject(resultTerm))
 

zymoTransmitSupport/inputOutput/resultReader.py

@@ -14,6 +14,7 @@ class TestResult(object):
     def __init__(self, rawLine: [str, collections.Iterable], delimiter: str = "\t"):
         self.rawLine = rawLine
         if type(self.rawLine) == str:
+            self.rawLine = self.rawLine.strip()
             self.elementArray = self.processRawLine(delimiter)
         elif isinstance(rawLine, collections.Iterable):
             self.elementArray = self.processList(self.rawLine)
@@ -59,7 +60,8 @@ def __init__(self, rawLine: [str, collections.Iterable], delimiter: str = "\t"):
         self.reportedDateTime = self.processDateAndTime(reportedDate, reportedTime)
         self.auxiliaryData = {}
         self.okToTransmit = True
-        self.reasonsNotToTransmit = []
+        self.reasonForFailedTransmission = []
+        self.transmittedSuccessfully = None
 
     def processRawLine(self, delimiter):
         rawLine = self.rawLine.strip()

zymoTransmitSupport/inputOutput/soapAPI.py

@@ -1,5 +1,7 @@
 import zeep
 from .. import config as defaultConfig
+from . import resultReader
+import typing
 
 config = defaultConfig
 
@@ -40,7 +42,17 @@ def __str__(self):
         return outputBlock
 
 
-def transmitBlocks(client:zeep.Client, hl7Blocks:dict):
+def transmitBlocks(client:zeep.Client, hl7Blocks:dict, resultList:typing.List[resultReader.TestResult]=None):
+    def makeResultKey():
+        resultKey = {}
+        for index, result in enumerate(resultList):
+            key = (result.patientID, result.specimenID)
+            resultKey[key] = index
+        return resultKey
+    if resultList is None:
+        resultKey = {}
+    else:
+        resultKey = makeResultKey()
     client.raw_response = True
     submissionResults = []
     if config.Configuration.productionReady:
@@ -62,12 +74,20 @@ def transmitBlocks(client:zeep.Client, hl7Blocks:dict):
         except Exception as err:
             submissionStatus = SubmissionStatus(patientID, specimenID, None, str(err))
             print("ERROR: attempted to submit %s:%s, but it failed to return an error. See submission log for more details." %(patientID, specimenID))
+            if resultID in resultKey:
+                resultList[resultKey[resultID]].transmittedSuccessfully = False
+                resultList[resultKey[resultID]].reasonForFailedTransmission.append("Gateway failed to respond. This is likely a gateway issue and may self-resolve if given some time.")
         else:
             if response.status == "VALID":
                 submissionStatus = SubmissionStatus(patientID, specimenID, True, getattr(response, "return"))
                 print("Successfully submitted %s:%s" %(patientID, specimenID))
+                if resultID in resultKey:
+                    resultList[resultKey[resultID]].transmittedSuccessfully = True
             else:
                 submissionStatus = SubmissionStatus(patientID, specimenID, False, getattr(response, "return"))
                 print("ERROR: attempted to submit %s:%s, but it was rejected. See submission log for more details." %(patientID, specimenID))
+                if resultID in resultKey:
+                    resultList[resultKey[resultID]].transmittedSuccessfully = False
+                    resultList[resultKey[resultID]].reasonForFailedTransmission.append("Gateway rejected transmission. This indicates a likely error in the data and requires some correction before attempting to transmit again. See the log file for specific errors.")
         submissionResults.append(submissionStatus)
     return submissionResults

zymoTransmitSupport/supportData.py

@@ -1,4 +1,4 @@
 
-softwareVersion = "0.0.1"
+softwareVersion = "1.0.0"
 
-softwareDate = "20200606"
+softwareDate = "20200719"