Implemented Manager XML for better usability and security.

CHANGELOG.md

@@ -1,3 +1,11 @@
-## 0.1.0
+## 1.0
 
 First version
+
+## 1.1
+
+- Added Manager XML-file for better usability when adding inputs
+- Removed protocol parameter
+- Changed hostname parameter to include protocol
+- Changed API key to be a password-entry
+- Added license info for Splunklib

README.md

@@ -6,6 +6,16 @@ The Splunk TA for Octopus Deploy is a technology add-on for retrieving data from
 
 You must have an installed and configured Octopus application to complete the setup process of this app.
 
+### Used libraries
+
+Splunklib which originates from the Splunk Software Development Kit for Python. The Splunk Software Development Kit for Python is licensed under the [Apache
+License 2.0](https://www.apache.org/licenses/LICENSE-2.0.html).
+
+For compatibility with Python 2.#, The Splunk Software Development Kit
+for Python ships with ordereddict.py from the ordereddict package on
+[PyPI](http://pypi.python.org/pypi/ordereddict/1.1), which is licensed
+under the MIT license (see the top of bin/splunklib/ordereddict.py).
+
 ### Install Prerequisites
 
 Too make sure everything works correctly make sure the following is available and working:
@@ -32,14 +42,15 @@ The TA will not deliver predefined inputs for now so you need to configure each
 
 When you press "New" the Input Wizard will start and you will see one screen with multiple fields.
 
-1. name > The name for Input in the Splunk UI (e.g. Octopus Users)
+1. Name > The name for Input in the Splunk UI (e.g. Octopus Users)
 2. Endpoint > The Octopus endpoint to access (e.g. users)
-3. Protocol > The protocol on which Octopus Deploy is listening (HTTP or HTTPS)
-4. Hostname > The hostname and port on which Octopus Deploy is listening (e.g. octopus.example.com:8080)
-5. API key > The API key needed for authentication with the API. See [Octopus Wiki](https://github.com/OctopusDeploy/OctopusDeploy-Api/wiki).
-6. Use check pointing > If you do not want to index all the data inside each time the Input is running enable check pointing. The Input will keep track of the last indexed record and will only index new records.
+2. Hostname > The protocol, hostname and port on which Octopus Deploy is listening (e.g. http://octopus.example.com:8080)
+5. Password > The API key needed for authentication with the API. See [Octopus Wiki](https://github.com/OctopusDeploy/OctopusDeploy-Api/wiki). 
+6. Confirm password > The API key needed for authentication with the API.
+7. Use check pointing > If you do not want to index all the data inside each time the Input is running enable check pointing. The Input will keep track of the last indexed record and will only index new records.
+8. Interval > Configure the interval on which the Input needs to run. 
 
-Check "More Settings" to configure Interval, Sourcetype, Host and Index. By default the TA will use the `octopus` Index and an Interval of 300 seconds if you add an Input.
+Check "More Settings" to configure Sourcetype, Host and Index.
 
 The following list show the most common endpoints for Octopus Deploy API and if check pointing should be applied:
 * machines (use check pointing: NO)
@@ -71,7 +82,7 @@ This app is developed in the open at [GitHub](https://github.com/cmeerbeek/TA-oc
 
 ## ToDo
 
-* Add configuration page so you only need to provide the authentication information and data starts flowing.
+* Rebuild Add-on using Splunk Add-on Builder so the TA will confirm to Splunk best practices and uses a nice UI for configuring inputs
 
 ## Copyright
 

bin/TA-octopus_deploy.py

@@ -114,10 +114,10 @@ def load_checkpoint(checkpoint, checkpoint_dir):
 ###############################
 
 
-def getEntries(endpoint, protocol, hostname, api_key, use_checkpoint, checkpoint):
+def getEntries(endpoint, hostname, api_key, use_checkpoint, checkpoint):
     logger = setup_logging()
     logger.info("getEntries: " + time.strftime("%d-%m-%Y %H:%M:%S"))
-    octopus_url = "%s://%s/api/%s" % (protocol, hostname, endpoint)
+    octopus_url = "%s/api/%s" % (hostname, endpoint)
     if int(use_checkpoint) == 1:
         checkpoint_dir = os.path.join(
             _SPLUNK_HOME, 'var', 'lib', 'splunk', 'modinputs', 'TA-octopus_deploy')
@@ -130,6 +130,7 @@ def getEntries(endpoint, protocol, hostname, api_key, use_checkpoint, checkpoint
             headers={
                 "X-Octopus-ApiKey": api_key,
             },
+            verify=False,
         )
         response.raise_for_status()
 
@@ -163,7 +164,7 @@ def getEntries(endpoint, protocol, hostname, api_key, use_checkpoint, checkpoint
         # Try to get next page if available, else write most recent deployment
         # id and exit
         try:
-            octopus_url = protocol + "://" + hostname + \
+            octopus_url = hostname + \
                 json_response['Links']['Page.Next']
         except Exception:
             break
@@ -189,17 +190,10 @@ def get_scheme(self):
         endpoint_argument.require_on_create = True
         scheme.add_argument(endpoint_argument)
 
-        protocol_argument = Argument("protocol")
-        protocol_argument.title = "Protocol"
-        protocol_argument.data_type = Argument.data_type_string
-        protocol_argument.description = "Protocol for accessing the Octopus Deploy API (http or https)"
-        protocol_argument.require_on_create = True
-        scheme.add_argument(protocol_argument)
-
         hostname_argument = Argument("hostname")
         hostname_argument.title = "Hostname"
         hostname_argument.data_type = Argument.data_type_string
-        hostname_argument.description = "Hostname of Octopus Deploy environment (hostname:port)"
+        hostname_argument.description = "Hostname of Octopus Deploy environment (http(s)://hostname:port)"
         hostname_argument.require_on_create = True
         scheme.add_argument(hostname_argument)
 
@@ -226,10 +220,9 @@ def validate_input(self, validation_definition):
         # Get the values of the parameters, and construct a URL for the Octopus
         # Deploy API
         endpoint = validation_definition.parameters["endpoint"]
-        protocol = validation_definition.parameters["protocol"]
         hostname = validation_definition.parameters["hostname"]
         api_key = validation_definition.parameters["api_key"]
-        octopus_url = "%s://%s/api/%s" % (protocol, hostname, endpoint)
+        octopus_url = "%s/api/%s" % (hostname, endpoint)
         logger.info("URL: " + octopus_url)
 
         # Read the response from the Octopus Deploy API, then parse the JSON data into an object
@@ -240,6 +233,7 @@ def validate_input(self, validation_definition):
                 headers={
                     "X-Octopus-ApiKey": api_key,
                 },
+                verify=False,
             )
             response.raise_for_status()
         except requests.exceptions.HTTPError as err:
@@ -264,13 +258,12 @@ def stream_events(self, inputs, ew):
 
         for input_name, input_item in inputs.inputs.iteritems():
             endpoint = input_item['endpoint']
-            protocol = input_item['protocol']
             hostname = input_item['hostname']
             api_key = input_item['api_key']
             use_checkpoint = input_item['use_checkpoint']
             checkpoint = md5.new(input_name).hexdigest()
 
-            data = getEntries(endpoint, protocol, hostname,
+            data = getEntries(endpoint, hostname,
                               api_key, use_checkpoint, checkpoint)
 
             for d in data:

default/app.conf

@@ -11,4 +11,4 @@ label = TA Octopus Deploy
 [launcher]
 author = cmeerbeek
 description = Stream data from the Octopus Deploy API
-version = 1.0
+version = 1.1

default/data/ui/manager/octopus-setup.xml

@@ -0,0 +1,209 @@
+<endpoint name="data/inputs/TA-octopus_deploy">
+  <header>Connect to the Octopus Deploy API and stream events to Splunk.</header>
+
+  <breadcrumb>
+    <parent hidecurrent="False">datainputstats</parent>
+    <name>Octopus Deploy API</name>
+  </breadcrumb>
+
+  <elements>
+    <element name="inputFields" type="fieldset">
+      <view name="list"/>
+      <view name="edit"/>
+      <view name="create"/>
+      <elements>
+
+        <element name="name" label="Name">
+          <view name="list"/>
+          <view name="create"/>          
+        </element>
+
+
+        <element name="endpoint" type="textfield" label="Endpoint">
+          <view name="list"/>
+          <view name="edit"/>
+          <view name="create"/>
+          <key name="exampleText">Octopus Deploy API endpoint (e.g. environments,events,task,releases)</key>
+        </element>
+
+        <element name="hostname" type="textfield" label="Hostname">
+          <view name="list"/>
+          <view name="edit"/>
+          <view name="create"/>
+          <key name="exampleText">URI of Octopus Deploy environment (http(s)://hostname:port)</key>
+        </element>
+
+        <element name="api_key" type="password" label="Password">
+            <view name="edit"/>
+            <view name="create"/>
+            <key name="exampleText">Use this field, and the confirm field, for the Octopus Deploy API key (https://octopus.com/docs/how-to/how-to-create-an-api-key)</key>
+        </element>
+
+        <element name="use_checkpoint" type="checkbox" label="Use check pointing">
+            <view name="edit"/>
+            <view name="create"/>
+            <key name="exampleText">Decide if check pointing is needed for this input</key>
+        </element>
+
+        <element name="interval" type="textfield" label="Interval">
+            <view name="list"/>
+            <view name="edit"/>
+            <view name="create"/>
+            <key name="exampleText">The interval defining how often to perform the check; can include time units (e.g. 15m for 15 minutes, 8h for 8 hours)</key>
+        </element>
+
+      </elements>
+    </element>
+
+    <!-- The sourcetype selection content is typically in the page by default -->
+    <element name="spl-ctrl_EnableAdvanced" type="checkbox" label="More settings" class="spl-mgr-advanced-switch">
+      <view name="edit"/>
+      <view name="create"/>
+      <onChange>
+        <key name="_action">showonly</key>
+        <key name="0">NONE</key>
+        <key name="1">ALL</key>
+        <group_set>
+          <group name="advanced"/>
+        </group_set>
+      </onChange>
+    </element>
+
+    <element name="sourcetype_selection_fix" type="sourcetype_selection_fix">
+      <view name="create"/>
+      <view name="edit"/>
+    </element>
+
+    <element name="password_copy_fix" type="password_copy_fix">
+      <view name="edit"/>
+    </element>
+
+    <element name="advanced" type="fieldset" class="spl-mgr-advanced-options">
+      <view name="edit"/>
+      <view name="create"/>
+
+      <elements>        
+        <element name="sourcetypeFields" type="fieldset">
+          <view name="list"/>
+          <view name="edit"/>
+          <view name="create"/>
+          <elements>
+
+            <element name="spl-ctrl_sourcetypeSelect" type="select" label="Set the source type">
+              <onChange>
+                <key name="auto">NONE</key>
+                <key name="_action">showonly</key>
+                <group_set>
+                  <group name="sourcetype"/>
+                  <group name="spl-ctrl_from_list"/>
+                </group_set>
+                <key name="sourcetype">sourcetype</key>
+                <key name="spl-ctrl_from_list">spl-ctrl_from_list</key>
+              </onChange>
+              <options>
+                <opt value="auto" label="Automatic"/>
+                <opt value="sourcetype" label="Manual"/>
+                <opt value="spl-ctrl_from_list" label="From list"/>
+              </options>
+              <view name="edit"/>
+              <view name="create"/>
+              <key name="exampleText">When this is set to automatic, Splunk classifies and assigns the sourcetype automatically, and gives unknown sourcetypes placeholder names.</key>
+              <key name="processValueEdit">[[ e for e in ['sourcetype'] if form_defaults.get(e) ][0]]</key>
+              <key name="processValueAdd">[[ e for e in ['sourcetype'] if form_defaults.get(e) ][0]]</key>
+            </element>
+
+            <element name="sourcetype" type="textfield" label="Source type">
+              <view name="list"/>
+              <view name="edit"/>
+              <view name="create"/>
+              <key name="processValueList">_('Automatic') if (value==None or value=='') else value</key>
+              <key name="submitValueAdd">value if 
+                form_data.get('spl-ctrl_sourcetypeSelect')=='sourcetype' 
+                else (form_data.get('spl-ctrl_from_list') 
+                if form_data.get('spl-ctrl_sourcetypeSelect')=='spl-ctrl_from_list' 
+                else '')</key>
+              <key name="submitValueEdit">value if 
+                form_data.get('spl-ctrl_sourcetypeSelect')=='sourcetype' 
+                else (form_data.get('spl-ctrl_from_list') 
+                if form_data.get('spl-ctrl_sourcetypeSelect')=='spl-ctrl_from_list' 
+                else '')</key>
+              <key name="labelList">Source type</key>
+            </element>
+
+            <element name="spl-ctrl_from_list" type="select" label="Select source type from list">
+              <view name="edit"/>
+              <view name="create"/>
+              <key name="exampleText">Splunk classifies all common data types automatically, 
+                  but if you're looking for something specific, you can find more source types 
+                  in the <![CDATA[<a href="../../../apps/remote">Splunkbase apps browser</a>]]> 
+                  or online at <![CDATA[<a href="http://www.splunkbase.com/" 
+                  target="_blank">www.splunkbase.com</a>]]>.</key>
+              <key name="requiredIfVisible" />
+              <key name="dynamicOptions" type="dict">
+                <key name="prefixOptions" type="list">
+                  <item type="list">
+                    <item></item>
+                    <item>Choose...</item>
+                  </item>
+                </key>
+                <key name="keyName">title</key>
+                <key name="keyValue">title</key>
+                <key name="splunkSource">/saved/sourcetypes</key>
+                <key name="splunkSourceParams" type="dict">
+                  <key name="count">-1</key>
+                  <key name="search">'pulldown_type=true'</key>
+                </key>
+              </key>
+
+            </element>
+          </elements>
+          <key name="legend">Source type</key>
+          <key name="helpText">Tell Splunk what kind of data this is so you can group it with 
+            other data of the same type when you search. Splunk does this 
+            automatically, but you can specify what you want if Splunk gets it 
+            wrong.</key>
+
+        </element>
+
+        <element name="hostFields" type="fieldset">
+          <key name="legend">Host</key>
+          <view name="list"/>
+          <view name="edit"/>
+          <view name="create"/>
+          <elements>
+            <element name="host" type="textfield" label="Host field value">
+              <view name="edit"/>
+              <view name="create"/>
+            </element>
+          </elements>
+        </element>  
+
+        <element name="indexField" type="fieldset">
+          <key name="legend">Index</key>
+          <key name="helpText">Set the destination index for this source.</key>
+          <view name="list"/>
+          <view name="edit"/>
+          <view name="create"/>
+          <elements>
+            <element name="index" type="select" label="Index">
+              <view name="list"/>
+              <view name="edit"/>
+              <view name="create"/>
+              <key name="dynamicOptions" type="dict">
+                <key name="keyName">title</key>
+                <key name="keyValue">title</key>
+                <key name="splunkSource">/data/indexes</key>
+                <key name="splunkSourceParams" type="dict">
+                  <key name="search">'isInternal=false disabled=false'</key>
+                  <key name="count">-1</key>
+                </key>
+              </key>
+            </element>
+          </elements>
+        </element>
+
+      </elements>
+    </element>
+
+  </elements>
+</endpoint>

metadata/default.meta

@@ -0,0 +1,2 @@
+[]
+export = system