Merge pull request #4

optumCompatibility has been used to submit data into production.
Zymo-Research · Jan 15, 2021 · 9019a89 · 9019a89
2 parents a86eedc + 1009811
commit 9019a89
Show file tree

Hide file tree

Showing 12 changed files with 207 additions and 15 deletions.
diff --git a/README.md b/README.md
@@ -19,7 +19,7 @@ It is with these principles guiding our efforts that we offer this software pack
 At present, there is not publication planned for this software.  If anybody in the public health/epidemiology field wishes to collaborate on one (especially the Fielding School of Public Health), please contact us.
 
 ## Quick Start Guide
-This Program was written in Python 3.6.4.  It should work with other version of 3.6 and most likely later versions as well.
+This Program was written in Python 3.8.  It has also been tested with version 3.6 and most likely will work with other versions as well.
 Earlier versions may have some difficulties. Please report any compatibility problems you may have and I will see about addressing them if you really must run on a different version of Python.
 
 If you do not already have the Python language installed on your computer, you can download the web-based installer from this page on [python.org](https://www.python.org/downloads/release/python-364/) or directly from [here](https://www.python.org/ftp/python/3.6.4/python-3.6.4-amd64-webinstall.exe) on a Windows computer.  **During Python installation, be sure to enable Python in your PATH variables, as that will make this program much easier to run in the future.  Also be sure to approve installation of Tcl/Tk and PIP, as those will also contribute to making this easier to run in the future.**
@@ -81,6 +81,12 @@ If you need to edit this file in the future, it can be found at ```zymoTransmit/
 ```
 python zymoTransmit.py
 ```
+**If NOT using a PFX certificate (you have CER and KEY files available)**
+
+You should have files called impl-cie-ws (for testing) and odxi-ws (for production) with endings of .cer and .key for each.  Simply copy these credential files into the certificates folder in your Zymo Research Transmit directory.  No further attention to these files should be required.
+
+**If using a PFX certificate for the legacy CDPH gateway**
+
 Convert your certificate for use (if you do not supply a file name, a file browser window will appear if GUI functionality is available)
 ```
 python zymoTransmit.py -c [fileName.pfx]
@@ -179,9 +185,9 @@ Patch release: No changes to parameters
 
 ## Current Version
 
-The current major release contains all initial functionality plus some additional features designed to help the California Department of Public Health clear backlogged data using their existing form.  Many of these extended features will also help other users with their submissions.  Essential features covered include transmission of data from this program's preferred table format (either CSV or tab-delimited text), transmission of a large block of HL7 data, transmission of a folder with individual HL7 data blocks, transmission of a CDPH-formatted CSV file, and handling of results that fail to transmit for various reasons.
+The current version contains all functionality from the previous version with the new key feature being the ability to connect to the new Optum API for reporting results in California.
 
-This major release has been nicknamed Imahara's Pudding Cup after [Grant Imahara](https://en.wikipedia.org/wiki/Grant_Imahara), a popular advocate of STEAM education and performing random acts of kindness for others.
+This major release has been nicknamed Geoff Peterson's Metal Mohawk in remembrance of [Grant Imahara](https://en.wikipedia.org/wiki/Grant_Imahara), a popular advocate of STEAM education and performing random acts of kindness for others.
 
 ## Authors
 
@@ -202,6 +208,7 @@ We would like to thank the following, without whom this would not have happened:
 * The Python Foundation
 * The staff at Zymo Research
 * The UC System
+  - Special thanks to UCLA's departments of Molecular, Cell, and Developmental Biology as well as Computational Medicine
 * Pangea Laboratory
 * The California Department of Public Health
 * Our customers

diff --git a/configUpgrade.txt b/configUpgrade.txt
@@ -0,0 +1,11 @@
+## Copy the text below into your config file if you are upgrading from version 1.x of this program.
+## The location is not important, but adding it to the end of the connection section would be ideal location
+
+Using Optum: FALSE
+Local WSDL Folder: wsdl
+Optum Testing WSDL File: ca_impl_wsdl.xml
+Optum Production WSDL File: ca_prod_wsdl.xml
+Optum Testing Certificate: impl-cie-ws.cer
+Optum Testing Key: impl-cie-ws.key
+Optum Production Certificate: odxi-ws.cer
+Optum Production Key: odxi-ws.key
diff --git a/raceEthnicityCodes.txt b/raceEthnicityCodes.txt
@@ -9,4 +9,4 @@
 PHC1369	not obtainable			
 PHC1367	refused			
 ASKU	asked but unknown			
-UNK	unknown	u		
+U	unknown	unk
diff --git a/requirements.txt b/requirements.txt
@@ -1,4 +1,5 @@
 pyOpenSSL==19.1.0
 zipcodes==1.1.0
-zeep==3.4.0
-pytk==0.0.2.0
+zeep==4.0.0
+pytk==0.0.2.0
+pathlib==1.0.1
diff --git a/templateSubmission.csv b/templateSubmission.csv
@@ -1,2 +1,2 @@
 #PatientID,Patient Last Name,Patient First Name,Patient Middle Name,Patient Date of Birth,Patient Sex,Patient Street Address,Patient City,Patient State,Patient Zip,Patient Phone,Provider Last Name,Provider First Name,Provider Middle Name,Provider Street,Provider City,Provider State,Provider Zip,Provider Phone,Specimen ID,Collection Date,Collection Time,Received Date,Received Time,Specimen SNOMED Code,Test LOINC Code,Analysis Date,Analysis Time,Result,Reported Date,Reported Time,Notes,Race,Ethnicity,Accession ID,Test Ordered Date,Test Ordered Time,Test Equipment Code,Test Equipment Description,Test Equipment ID Number,Provider NPI
-1K94U,Weinstein,Andrew,P,8/15/2017,M,17062 Murphy Ave. #1,Irvine,CA,92614,888-882-9682,Rain,Leo,,13842 Tustin East Drive,Tustin,CA,92780,(818) 344-3434,1234567,6/8/2020,0:00,6/8/2020,13:00,258529004,94306-8,6/9/2020,11:00,Negative,6/10/2020,14:00,"Phone is pretty flexible for formatting, just be reasonable.  Email address is also an acceptable entry there, if needed. Please stick to mm/dd/yyyy formatting for dates.  Having a # at the start of a line will cause that line to be ignored as a comment or header.",unknown,unk,1A2B3C4D5E,6/8/2020,0:00,qPCR,Zymo Research Quick SARS-CoV-2 rRT-PCR Kit,R3011,
+#1K94U,Weinstein,Andrew,P,8/15/2017,M,17062 Murphy Ave. #1,Irvine,CA,92614,888-882-9682,Rain,Leo,,13842 Tustin East Drive,Tustin,CA,92780,(818) 344-3434,1234567,6/8/2020,0:00,6/8/2020,13:00,258529004,94306-8,6/9/2020,11:00,Negative,6/10/2020,14:00,"Phone is pretty flexible for formatting, just be reasonable.  Email address is also an acceptable entry there, if needed. Please stick to mm/dd/yyyy formatting for dates.  Having a # at the start of a line will cause that line to be ignored as a comment or header.",unknown,unk,1A2B3C4D5E,6/8/2020,0:00,qPCR,Zymo Research Quick SARS-CoV-2 rRT-PCR Kit,R3011,
diff --git a/wsdl/ca_impl_wsdl.xml b/wsdl/ca_impl_wsdl.xml
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<definitions xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:s="http://www.w3.org/2001/XMLSchema" xmlns:s0="urn:cdc:iisb:2011" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" targetNamespace="urn:cdc:iisb:2011">
+    <types>
+        <s:schema targetNamespace="urn:cdc:iisb:2011">
+            <s:element name="submitMessage">
+                <s:complexType>
+                    <s:sequence>
+                        <s:element minOccurs="0" name="userid" type="s:string"/>
+                        <s:element minOccurs="0" name="password" type="s:string"/>
+                        <s:element minOccurs="0" name="dataownerid" type="s:string"/>
+                        <s:element minOccurs="0" name="cdphprogramid" type="s:string"/>
+                        <s:element minOccurs="0" name="cdphprogramenvironment" type="s:string"/>
+                        <s:element minOccurs="0" name="action" type="s:string"/>
+                        <s:element minOccurs="0" name="messagecontent" type="s:string"/>
+                    </s:sequence>
+                </s:complexType>
+            </s:element>
+            <s:element name="submitMessageResponse">
+                <s:complexType>
+                    <s:sequence>
+                        <s:element minOccurs="0" name="status" type="s:string"/>
+                        <s:element minOccurs="0" name="return" type="s:string"/>
+                    </s:sequence>
+                </s:complexType>
+            </s:element>
+        </s:schema>
+    </types>
+    <message name="submitMessageSoapIn">
+        <part name="parameters" element="s0:submitMessage"/>
+    </message>
+    <message name="submitMessageSoapOut">
+        <part name="parameters" element="s0:submitMessageResponse"/>
+    </message>
+    <portType name="HL7SOAPEndPointSvcSoap">
+        <operation name="submitMessage">
+            <input message="s0:submitMessageSoapIn"/>
+            <output message="s0:submitMessageSoapOut"/>
+        </operation>
+    </portType>
+    <binding name="HL7SOAPEndPointSvcSoap" type="s0:HL7SOAPEndPointSvcSoap">
+        <soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"/>
+        <operation name="submitMessage">
+            <soap:operation soapAction="urn:cdc:iisb:2011/OCIELib.Covid.CA.Service.HL7SOAPEndPointSvc.submitMessage" style="document"/>
+            <input>
+                <soap:body use="literal"/>
+            </input>
+            <output>
+                <soap:body use="literal"/>
+            </output>
+        </operation>
+    </binding>
+    <service name="HL7SOAPEndPointSvc">
+        <port name="HL7SOAPEndPointSvcSoap" binding="s0:HL7SOAPEndPointSvcSoap">
+            <soap:address location="https://gateway-tls.linkhealth.com/capreprod"/>
+        </port>
+    </service>
+</definitions>
diff --git a/wsdl/ca_prod_wsdl.xml b/wsdl/ca_prod_wsdl.xml
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<definitions xmlns="http://schemas.xmlsoap.org/wsdl/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/" xmlns:s="http://www.w3.org/2001/XMLSchema" xmlns:s0="urn:cdc:iisb:2011" xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" targetNamespace="urn:cdc:iisb:2011">
+    <types>
+        <s:schema targetNamespace="urn:cdc:iisb:2011">
+            <s:element name="submitMessage">
+                <s:complexType>
+                    <s:sequence>
+                        <s:element minOccurs="0" name="userid" type="s:string"/>
+                        <s:element minOccurs="0" name="password" type="s:string"/>
+                        <s:element minOccurs="0" name="dataownerid" type="s:string"/>
+                        <s:element minOccurs="0" name="cdphprogramid" type="s:string"/>
+                        <s:element minOccurs="0" name="cdphprogramenvironment" type="s:string"/>
+                        <s:element minOccurs="0" name="action" type="s:string"/>
+                        <s:element minOccurs="0" name="messagecontent" type="s:string"/>
+                    </s:sequence>
+                </s:complexType>
+            </s:element>
+            <s:element name="submitMessageResponse">
+                <s:complexType>
+                    <s:sequence>
+                        <s:element minOccurs="0" name="status" type="s:string"/>
+                        <s:element minOccurs="0" name="return" type="s:string"/>
+                    </s:sequence>
+                </s:complexType>
+            </s:element>
+        </s:schema>
+    </types>
+    <message name="submitMessageSoapIn">
+        <part name="parameters" element="s0:submitMessage"/>
+    </message>
+    <message name="submitMessageSoapOut">
+        <part name="parameters" element="s0:submitMessageResponse"/>
+    </message>
+    <portType name="HL7SOAPEndPointSvcSoap">
+        <operation name="submitMessage">
+            <input message="s0:submitMessageSoapIn"/>
+            <output message="s0:submitMessageSoapOut"/>
+        </operation>
+    </portType>
+    <binding name="HL7SOAPEndPointSvcSoap" type="s0:HL7SOAPEndPointSvcSoap">
+        <soap:binding transport="http://schemas.xmlsoap.org/soap/http" style="document"/>
+        <operation name="submitMessage">
+            <soap:operation soapAction="urn:cdc:iisb:2011/OCIELib.Covid.CA.Service.HL7SOAPEndPointSvc.submitMessage" style="document"/>
+            <input>
+                <soap:body use="literal"/>
+            </input>
+            <output>
+                <soap:body use="literal"/>
+            </output>
+        </operation>
+    </binding>
+    <service name="HL7SOAPEndPointSvc">
+        <port name="HL7SOAPEndPointSvcSoap" binding="s0:HL7SOAPEndPointSvcSoap">
+            <soap:address location="https://gateway-tls.linkhealth.com/caprod"/>
+        </port>
+    </service>
+</definitions>
diff --git a/zymoTransmit.py b/zymoTransmit.py
@@ -3,6 +3,7 @@
     import sys
     import traceback
     import csv
+    import pathlib
 
     contentRoot = os.path.split(os.path.abspath(__file__))[0]
 
@@ -198,6 +199,35 @@ def processRejects(resultList:typing.List[zymoTransmitSupport.inputOutput.result
     file.close()
 
 
+def selectCertificatePath():
+    if config.Connection.usingOptum:
+        if config.Configuration.productionReady:
+            certFile = config.Connection.optumProductionCertificate
+            keyFile = config.Connection.optumProductionKey
+        else:
+            certFile = config.Connection.optumTestingCertificate
+            keyFile = config.Connection.optumTestingKey
+        certPath = os.path.join(contentRoot, config.Connection.certificateFolder, certFile)
+        keyPath = os.path.join(contentRoot, config.Connection.certificateFolder, keyFile)
+        certificateFilePath = (certPath, keyPath)
+    else:
+        certificateFilePath = os.path.join(contentRoot, config.Connection.certificateFolder, config.Connection.certificateFileName)
+    return certificateFilePath
+
+
+def selectURLForWSDL():
+    if config.Connection.usingOptum:
+        if config.Configuration.productionReady:
+            relativePath = os.path.join(config.Connection.localWSDLFolder, config.Connection.optumProductionWSDL)
+        else:
+            relativePath = os.path.join(config.Connection.localWSDLFolder, config.Connection.optumTestingWSDL)
+        absolutePath = os.path.abspath(relativePath)
+        wsdlURL = pathlib.Path(absolutePath).as_uri()
+    else:
+        wsdlURL = config.Connection.wsdlURL
+    return wsdlURL
+
+
 def prepareAndSendResults(args:CheckArgs):
     if not args.hl7Directory:
         if os.path.abspath(args.input) == os.path.join(contentRoot, "config.txt"):
@@ -216,9 +246,10 @@ def prepareAndSendResults(args:CheckArgs):
             quit()
     skippedData = []
     resultList = []
-    certificateFilePath = os.path.join(contentRoot, config.Connection.certificateFolder, config.Connection.certificateFileName)
+    certificateFilePath = selectCertificatePath()
+    wsdlURL = selectURLForWSDL()
     client, session = zymoTransmitSupport.inputOutput.connection.getSOAPClient(
-        config.Connection.wsdlURL, certificateFilePath, dumpClientInfo=False)
+        wsdlURL, certificateFilePath, dumpClientInfo=False, testOnly=False)
     if args.hl7Directory:
         if not os.path.isdir(args.input):
             raise NotADirectoryError("%s was given as a directory for raw HL7 block files, but it is not a directory.")

diff --git a/zymoTransmitSupport/config.py b/zymoTransmitSupport/config.py
@@ -97,6 +97,14 @@ class Connection:
     certificateFileName = configDict.setdefault("certificate file name", "")
     userName = configDict.setdefault("gateway user", "")
     password = configDict.setdefault("gateway password", "")
+    usingOptum = configDict.setdefault("using optum", False)
+    localWSDLFolder = configDict.setdefault("local wsdl folder", "")
+    optumTestingWSDL = configDict.setdefault("optum testing wsdl file", "")
+    optumProductionWSDL = configDict.setdefault("optum production wsdl file", "")
+    optumTestingCertificate = configDict.setdefault("optum testing certificate", "")
+    optumTestingKey = configDict.setdefault("optum testing key", "")
+    optumProductionCertificate = configDict.setdefault("optum production certificate", "")
+    optumProductionKey = configDict.setdefault("optum production key", "")
 
 
 class Configuration:

diff --git a/zymoTransmitSupport/configClean.txt b/zymoTransmitSupport/configClean.txt
@@ -64,6 +64,14 @@ WSDL URL: https://hiegateway.cdph.ca.gov/submit/CDPH_transfer.wsdl
 Submission URL: https://hiegateway.cdph.ca.gov/submit/services/CDPH_transfer.CDPH_transferHttpsSoap12Endpoint
 Certificate Folder: certificates
 Certificate File Name: certificate.pem
+Using Optum: FALSE
+Local WSDL Folder: wsdl
+Optum Testing WSDL File: ca_impl_wsdl.xml
+Optum Production WSDL File: ca_prod_wsdl.xml
+Optum Testing Certificate: impl-cie-ws.cer
+Optum Testing Key: impl-cie-ws.key
+Optum Production Certificate: odxi-ws.cer
+Optum Production Key: odxi-ws.key
 
 ## Message Header Values
 Receiver Name: CDPH CA REDIE

diff --git a/zymoTransmitSupport/inputOutput/connection.py b/zymoTransmitSupport/inputOutput/connection.py
@@ -26,16 +26,28 @@ def getSOAPClient(wsdlURL:str, clientCertificatePath:str=None, dumpClientInfo:bo
     if strongSSLCipherSuiteEnforcement:
         session.mount("https://", adapter=TLSEnforcer())
     if clientCertificatePath:
-        if not os.path.isfile(clientCertificatePath):
-            raise FileNotFoundError("Unable to find client certificate path at %s" %clientCertificatePath)
+        if type(clientCertificatePath) == str:
+            if not os.path.isfile(clientCertificatePath):
+                raise FileNotFoundError("Unable to find client certificate path at %s" %clientCertificatePath)
+        else:
+            if type(clientCertificatePath) in [tuple, list]:
+                if not len(clientCertificatePath) == 2:
+                    raise ValueError("List/tuple certificate paths should have exactly two elements with the first being the certificate and second being key.")
+                certPath, keyPath = clientCertificatePath
+                if not os.path.isfile(certPath) or not os.path.isfile(keyPath):
+                    raise FileNotFoundError("Unable to find client certificate and/or key path.\nCert: %s\nKey: %s" %(certPath, keyPath))
         session.cert = clientCertificatePath
     transport = zeep.transports.Transport(session=session, timeout=30)
-    client = zeep.Client(wsdlURL, transport=transport)
+    try:
+        client = zeep.Client(wsdlURL, transport=transport)
+    except OSError:
+        wsdlURL = wsdlURL.replace("file:///", "file://")  # Seems like a weird quirk on windows where it wants to add a root / to the URI.  Might be a bug somewhere I need to report.  Actually, fixed in the latest zeep major release.
+        client = zeep.Client(wsdlURL, transport=transport)
     #testUpload = client.service.connectivityTest("ping")
     if testOnly or dumpClientInfo:
         print("Server returned:")
         client.wsdl.dump()
         if testOnly:
             input("Press ENTER key to quit.")
             quit()
-    return client, session
+    return client, session
diff --git a/zymoTransmitSupport/supportData.py b/zymoTransmitSupport/supportData.py
@@ -1,4 +1,4 @@
 
-softwareVersion = "1.4.0"
+softwareVersion = "2.0.0"
 
-softwareDate = "20201201"
+softwareDate = "20201228"