ZigBee object persistence across multiple HTTP request

[jeff1evesque]

We need to store the ZigBee class instance in some cache, or the reference of the class in a database. This useful for cases where a user (via web-interface, or http request) connects to an xbee device, and there exists a limit for the number of connections allowed.

If such is the case, we need to research if python can store the ZigBee instance, so that, if the number of unique connections reaches a limit, to not issue more ZigBee connections (to the desired device). This will be done (if possible) by counting the number of active ZigBee processes, either from a database or system cache.

[jeff1evesque]

IRC #python (12/06/14 ~ 8:00pm EST):

jeffreylevesque: i have a user that accesses a python program via the web. That python program creates an object related to the users session. He may goto another page, and execute other commands. The other commands pertain to the object that was first created. I need that object cached, or stored in the database somehow. What's the best way?

nedbat: jeffreylevesque, store persistent state in a database. put db ids into the session

jeffreylevesque: what are persistent states?
jeffreylevesque well, the user creates a zigbee thread upon pressing a button. Later zigbee related commands are predicated to the original zigbee object / thread that was created.
jeffreylevesque how do i store the zigbee object, either in db or cache? I'd like other commands (html button) to be associated with the zigbee thread they originally created

nedbat: jeffreylevesque, by persistent state, I mean anything you want to remember from one request to the next.
nedbat: jeffreylevesque, in your example, the "zigbee thread that was created" is persistent state associated with the user.

[jeff1evesque]

We may implement a python global dictionary, and store the object within it. This will allow us to avoid the use of a database, and later clearing the database cached values upon system reboot.

IRC #python (12/07/14 ~ 4:30pm EST):

jeffreylevesque: offby1, i created an instance of a class. Now, I'd like that instance to be cached on the system environment. That cache has a variable named associated with it. I want to store that variable to my database.

offby1: jeffreylevesque, sure -- stick it into a global dict, e.g.,

jeffreylevesque: offby1, that will exists for as long as the machine is not turned off, or variable being cleaned up?

offby1: jeffreylevesque, sure. If the dict is a global. It's just a variable like any other.

jeffreylevesque: offby1, I probably can get away without using a database for this. Just create a dictionary of users, and the XBee object. This way, when the machine gets rebooted for whatever reason, I don't need to clear the database for the cached values - since they wouldn't exist anyways in the python global variables.

offby1: jeffreylevesque, agreed: if you explicitly want to start with a clean slate every time you start the app, then you do NOT want to use a database.
offby1: life will be a lot simpler.

[washort]

No such thing as "object persistence". Objects exist inside a process and can't live longer than the process does.

[jeff1evesque]

I've determined that an http request that invokes the Zigbee object, creates a new process (pid), called python. I've rebooted the server, executed the same http request (via web-interface / ajax), which creates the same process called python. This process never terminates, since the method that invokes the Zigbee object (via http request), simply contains one line of code:

def start(self):
  self.xbee = ZigBee(self.ser,callback=self.message_received,escaped=True)

Therefore, the new process, is most likely associated with Zigbee instance, and lives indefinitely, until the machine is rebooted.

Note: i used the command ps -e to review all processes before invoking the Zigbee object (via web-interface / ajax), and after the invocation to determine new processes.

[jeff1evesque]

If the python method which originally contained the one line invoking the Zigbee object:

def start(self):
  self.xbee = ZigBee(self.ser,callback=self.message_received,escaped=True)

is replaced with dummy code:

def start(self):
  print 'hello, there'

no new process is created, when an http request (via web-interface / ajax) is made to the replaced method. The replaced dummy code method has a similar result as the following:

def start(self):
  self.xbee = ZigBee(self.ser,callback=self.message_received,escaped=True)
  self.xbee.halt()
  self.ser.close()

No, new process is created - since, any threads related the ZigBee instance is cleaned up, then the serial port associated with the ZigBee instance is closed.

Note: i used the command ps -e to review all processes before invoking the Zigbee object (via web-interface / ajax), and after the invocation to determine new processes.

[jeff1evesque]

If we change the original method responsible for invoking the ZigBee object:

def start(self):
  self.xbee = ZigBee(self.ser,callback=self.message_received,escaped=True)

to the following:

def start(self):
  self.xbee = ZigBee(self.ser,callback=self.message_received,escaped=True)
  import os
  print os.getpid()
  self.xbee.halt()
  self.ser.close()

we get the process id, or pid of the ZigBee instance, followed by the closing of any related threads, with the associated serial port being closed:

"<ZigBee(Thread-1, started -1233783696)>\",\"3151\"

[jeff1evesque]

IRC #arduino (12/09/14 ~ 12:15pm EST):

kline: ok, so you have apache

jeffreylevesque: yes sir

kline: how is apache talking to python?

jeffreylevesque: well, php is my web app
jeffreylevesque: php shell scripts into python

kline: ok, so each request will open a new python

jeffreylevesque: yes sir

kline: which will try to grab the serial port
kline: so suddenly 10 scripts are fighting over 1 serial port

yeryry: only if they remain active

jeffreylevesque: yes sir. Unless i change the code to look for a cached ZigBee object
jeffreylevesque: instead of trying to create a new ZigBee instance

yeryry: if they send something then quit, it shouldn't be a problem, unless you send requests too quickly
kline: yeryry: thats a very very poor concurrency pattern
kline: and i promise it will fall over at the worst time
kline (aka when people are taking an interest in your project)

yeryry: hmm... I hadn't seen anything that suggested more than one person would be trying to access this thing at a time... If that is the case, then yeah that wouldn't work

kline: jeffreylevesque: what i would do i make a tiny, persistent service that allows PHP to access the zigbee. the flavour of the month for such services is a RESTful http api, so youd have a tiny, peristent web app that waits for posts/gets, queues and handles them in turn, and returns json/xml/whatever suits and is parseable back to the php app

kline: http://i.imgur.com/IVYFTPf.png

kline: does this look more reasonable to you?
kline: the API service is a persistent program that doesnt need to worry about opening and closing the serial port/zigbee each time

yeryry: have one python thing always running, handling the zigbee stuff... have either PHP or another python cgi-type thing send messages it via UDP or named pipe or something?

kline: yeryry: do you agree with the overall idea of my linked image?
kline: so the flask program never closes
kline: it gets the zigbee when it first starts and sequentially handles AJAX requests, so no processes ever have to fight over the serial port

yeryry: kline: I've no idea what flask is..

kline: yeryry: web application framework for python

yeryry: Could just forget about the PHP and do it with python WSGI... Then you could have a single python script that remained running

kline: yeryry: flask supports WSGI
kline: for you, just replace flask with "WSGI framework"
kline aaah, anyway, i have to head
kline: best of luck, and no matter how you choose to do it, the #1 tip here is to decouple the zigbee from the front end webserver

jeffreylevesque: kline one last question
jeffreylevesque: so flask replaces php?

kline: implement access to it as some sort of persistent service that the web server can access

jeffreylevesque: kline: so either use flask, or make php persistent

kline: it isnt a drop in replacement, but in this case it makes the php ajax/zigbee script redundant
kline yes

[jeff1evesque]

@washort I've been a little shy about going the route of Flask. I've attempted to study it within my Machine Learning repository couple months ago. Being a little overwhelmed, I decided to use HTTP (post) requests, AJAX, and JSON to php.

[jeff1evesque]

Generally, only one connection can be made on a given serial port. Therefore, we will need to create a persistent state for a chosen python script responsible for creating the ZigBee process (i.e. ZigBee object instance) on one defined serial port.

IRC #pocoo (12/09/14 ~ 4:00pm EST):

jeffreylevesque: can you have more than one connection on the same port?
jeffreylevesque: serial port

fmerges: no
fmerges: you should not do that

IRC #arduino (12/09/14 ~ 4:00pm EST):

jeffreylevesque: can you connect more than one connection on a given serial port?

old-papa-work: only one connection per port.
old-papa-work Rx to Tx and Tx to Rx..
old-papa-work match baud, bit, parity and stop if needed
old-papa-work if you need more than 1 serial port, Teensy has 3 hardware

IRC #raspberrypi (12/09/14 ~ 4:00pm EST):

jeffreylevesque: can you connect more than one connection on a given serial port?

Tachyon: yes and no
Tachyon: you shouldn't, but you can
Tachyon: but obviously will encounter strangeness
Tachyon: it's save if you just want to monitor with one etc.

IRC #python (12/09/14 ~ 4:00pm EST):

jeffreylevesque: can you connect more than one connection on a given serial port?

supa_: jeffreylevesque, You're trying to read and write to one serial port from two different things? That's not the best.

jeffreylevesque: supa_ so it's possible, and not recommended, or not possible?

MrElendig: jeffreylevesque: multiplexers do exist, commonly used for eg can/nmea/seatalk/similar

supa_: jeffreylevesque: Are you in Windows? Definitely not possible in Windows.