Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]Tick aggregate functions #386

Open
wants to merge 39 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
035f86f
added functionality to edit status card
holayeah Dec 19, 2016
25ccbca
fixing issues
holayeah Dec 19, 2016
ec13599
fixing issues
holayeah Dec 19, 2016
20b646c
enhanced the ui
holayeah Dec 19, 2016
795763f
switched to useing modal
holayeah Dec 19, 2016
ef7c084
aligned the data analysis graph update button
holayeah Dec 19, 2016
2618320
removed test that import csv from victron api
holayeah Dec 19, 2016
05a3b31
added permission to edit status card
holayeah Dec 20, 2016
605b733
added Tick script models
holayeah Dec 21, 2016
5c0ddf2
added tick script to the admin
holayeah Dec 21, 2016
394fe5c
Merge branch 'master' into tick_aggregate_functions
holayeah Dec 21, 2016
f07218e
quick rename for field type to script type
holayeah Dec 22, 2016
bd67867
Merge branch 'tick_aggregate_functions' of https://github.com/GreatLa…
holayeah Dec 22, 2016
5a26566
added a report tick template
holayeah Jan 3, 2017
d59ca87
enhanced the report aggregate template
holayeah Jan 3, 2017
d42f069
added a function to add report tasks to kapacitor given a report inst…
holayeah Jan 5, 2017
2659f76
fixing issues
holayeah Jan 5, 2017
4b0f160
restsarting tests
holayeah Jan 9, 2017
b5e9100
some debugging messages
holayeah Jan 9, 2017
0a0376e
fixing
holayeah Jan 9, 2017
38aa66c
testing edit reports
holayeah Jan 9, 2017
afe79e4
DEBUGGING: changed travis to make tests fast for this bug
holayeah Jan 10, 2017
c8bd0a9
debugging
holayeah Jan 10, 2017
34a8f3f
fixing
holayeah Jan 10, 2017
b6cdc7a
kapacitor tasks created on add report
holayeah Jan 10, 2017
7276efa
fixing
holayeah Jan 10, 2017
4bda7f5
Merge branch 'master' into tick_aggregate_functions
holayeah Jan 12, 2017
a174743
functionality to delete the tasks
holayeah Jan 23, 2017
dc3444d
merging master
holayeah Jan 26, 2017
a018240
fixed: when adding tasks corresponding kapacitor jobs are added
holayeah Jan 26, 2017
e1c51ed
removed admin backup files
holayeah Jan 26, 2017
15724e2
added pattern to list tasks, and changed reports jobs to not match th…
holayeah Jan 26, 2017
6711bea
added functionality to delete tasks when deleting report Job
holayeah Jan 26, 2017
1f9fa96
Merge branch 'tick_aggregate_functions' of https://github.com/GreatLa…
holayeah Jan 26, 2017
8a8d0a2
fixing
holayeah Jan 27, 2017
951da76
added tests for deleting reports jobs
holayeah Jan 27, 2017
1023350
added functions to enable and disable tasks
holayeah Jan 27, 2017
8250df9
added test for set status kapacitor attribute
holayeah Jan 27, 2017
475c388
added functionality to edit report task
holayeah Jan 27, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions seshdash/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,3 +95,7 @@ class StatusRule(GuardedModelAdmin):
@admin.register(Report)
class ReportDisplay(GuardedModelAdmin):
pass

@admin.register(Tick_Script)
class TickScript(GuardedModelAdmin):
pass
29 changes: 29 additions & 0 deletions seshdash/migrations/0004_tick_script.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
# -*- coding: utf-8 -*-
# Generated by Django 1.9.1 on 2016-12-21 15:25
from __future__ import unicode_literals

from django.db import migrations, models
import django.db.models.deletion


class Migration(migrations.Migration):

dependencies = [
('seshdash', '0003_auto_20161212_1209'),
]

operations = [
migrations.CreateModel(
name='Tick_Script',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('input_field_name', models.CharField(max_length=40)),
('output_field_name', models.CharField(max_length=40)),
('script', models.TextField()),
('function', models.CharField(max_length=20)),
('interval', models.CharField(max_length=10)),
('type', models.CharField(choices=[(b'stream', b'Stream'), (b'batch', b'Batch')], max_length=10)),
('site', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='seshdash.Sesh_Site')),
],
),
]
20 changes: 20 additions & 0 deletions seshdash/migrations/0005_auto_20161222_1026.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# -*- coding: utf-8 -*-
# Generated by Django 1.9.1 on 2016-12-22 08:26
from __future__ import unicode_literals

from django.db import migrations


class Migration(migrations.Migration):

dependencies = [
('seshdash', '0004_tick_script'),
]

operations = [
migrations.RenameField(
model_name='tick_script',
old_name='type',
new_name='script_type',
),
]
21 changes: 21 additions & 0 deletions seshdash/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -811,3 +811,24 @@ class Data_Process_Rule(models.Model):
duration = models.CharField(max_length=40, default="24h", choices=DURATION)
interval = models.CharField(max_length=10, default="5m", choices=TIME_BUCKETS)
output_field = models.ForeignKey(Trend_Data_Point)


class Tick_Script(models.Model):
"""
This is a model to hold data about the
tick scripts
"""
SCRIPT_TYPE_CHOICES = (
('stream', 'Stream'),
('batch', 'Batch'),
)

site = models.ForeignKey(Sesh_Site)
input_field_name = models.CharField(max_length=40) # The influx measurement to operate on
output_field_name = models.CharField(max_length=40) # The influx measurement to output results of the operation
script = models.TextField()
function = models.CharField(max_length=20) # Influx function operation to use
interval = models.CharField(max_length=10) # The interval the task script runs on e.g 1min, 1h, 1d, 1w
script_type = models.CharField(max_length=10, choices=SCRIPT_TYPE_CHOICES)


27 changes: 27 additions & 0 deletions seshdash/utils/reporting.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
from seshdash.models import Sesh_Site,Site_Weather_Data,BoM_Data_Point, Alert_Rule, Sesh_Alert,Daily_Data_Point, Sesh_User
from seshdash.utils.send_mail import send_mail
from seshdash.utils.model_tools import get_measurement_unit
from seshdash.data.db.kapacitor import Kapacitor

from django.forms.models import model_to_dict
from django.utils import timezone
from django.db.models import Avg, Sum
from django.apps import apps
from django.core.exceptions import FieldError
from django.conf import settings
from django.template.loader import get_template

from guardian.shortcuts import get_users_with_perms
from datetime import datetime
Expand Down Expand Up @@ -213,3 +216,27 @@ def get_table_report_dict(report_table_name, operations):
report_table_attributes.append(report_attribute_dict)

return report_table_attributes


"""
Kapacitor report utils
"""
def add_report_kap_task():
"""
This function adds a report task to kapacitor
"""
data = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is data hardcoded?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was for testing ,Now the function accepts a report instance and pulls the data from the report instance.

'database': settings.INFLUX_DB,
'operator': 'mean',
'field': 'battery_voltage',
'output_field': 'mean_battery_voltage',
'duration': '5m',
'site_id': '1'
}

kap = Kapacitor()
t = get_template('seshdash/kapacitor_tasks/aggregate_report.tick')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is using the Django template for the tick script. The user should also be able to give his/her own custom tick script. using the tick script model object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we do if the function is given a tick script instance use that as default we have the template for reports

print "About to send this template: %s" % (t.render(data))
response = kap.create_task('testing', t.render(data))

return response
24 changes: 24 additions & 0 deletions templates/seshdash/kapacitor_tasks/aggregate_report.tick
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// Example report tick script
// A user wants to get average(mean) for a field battery_voltage in a db called sesh
// The result will be outputed in a field called mean_battery_voltage in the sesh db


var database = '{{ database }}'

var duration = '{{ duration }}'

var output_field = '{{ output_field }}'

var points = batch
|query ('''
SELECT {{operator}}(value)
FROM "{{database}}"."autogen"."{{ field }}"
WHERE site_id={{site_id}}
''')
.period(duration)
.every(duration)

points
|influxDBOut()
.database(database)
.measurement(output_field)