Ib integration

[ryankennedyio]

Starting this as a long-running branch. Expect it to be merged when at least the IBService is capable of running an infinite loop without blocking the QSTrader trading_session.

More discussions in Slack for reasoning & alternate approaches, but for the record:

IBService abstracts away communication and message handling of the Python/IB API. Mainly because it has this annoying blocking infinite loop, which the Swigibpy version also had, but in a separate process.

Any component in QSTrader that wishes to interact with IB needs to go through the IBService. Call the Python API client methods on the IBService. IBService implements the EWrapper overrides that allow the IB responses to come back. All data that comes back from IB should be stored in Queues (or some other data structure) which are members of the IBService. Each component should look for the data it requested inside the appropriate queue (i.e. historic bars, market bars, portfolio updates, etc).

A nice bonus with this approach is that mocking IB for the purposes of testing any IBComponent (IBPriceHandler, IBExecutionHandler, etc) is simply a case of providing stub EClient methods, and returning some dummy data in the Queues that would normally be filled by the IB Response.

[ryankennedyio]

A major TODO here is that the IBService should be launched in a new thread or process, allowing it to run the infinite loop of getting/sending messages to IB. There should be some discussions around race condition safety, but I think since the Python Queue is a thread-safe multi-producer, multi-consumer queue, there should be no issues. Also, this is only one-producer, multi-consumer.

[ryankennedyio]

I've also just thought -- if multiple QSTrader components are going to be getting the same data directly from the callback data queues, neither will get all the data (get() removes from a queue).

Can't think of any clean fixes there yet.

[bzillins]

When previously trying to write a service to store historical options data I found it useful to pass a custom object with a unique ID and the contract object.
When you submit a type request for futures or options on the root symbol IB will return a stream of contract objects. If the functions to subscribe and request can accept the connection objects or a data structure containing the objects it saves you from having to pull out the relevant information required to recreate the objects later.

[timhubber]

Hi, great work on the project so far.

I noticed the issue with the infinite loop.

The ibapi client loop is contained in a method named 'run'. I just inherit the threading.Thread class and let the api client 'run' override the thread 'run' method. The infinite loop then runs in it's own thread.

class IBService(Wrapper, Client, threading.Thread):
    def __init__(self):
        Wrapper.__init__(self)
        Client.__init__(self, wrapper=self)
        threading.Thread.__init__(self, name='IBService')

[ryankennedyio]

Hey @timhubber that's great. I haven't had a chance to work further since the last commit, but have a block of time today where I can hopefully make a good chunk of progress. That looks like it should do the trick.

[ryankennedyio]

So, the IB Service seems to work OK, thanks again @timhubber.

IB Price Handler seems fine doing a few manual tests. Need to just pad out the test cases on it, ensure that it works with live data, then we can pull this branch in.

I have added a couple of TODOs if anyone is feeling generous :)
Otherwise will spend an hour or two on it tomorrow & again over the weekend.

[ryankennedyio]

Tests done and working, but failing because ibapi isn't available to travis. Otherwise, historic data ready to go.

Just checking in with IB to see if the API can be hosted on PyPi, otherwise we'll have to get a bit fancy with Travis (which would hopefully work).

[mhallsmoore]

Hmmm...that is going to be a tricky one. However, I reckon this must be rather common (not having access to an underlying package for builds), so I'll look into how we can get it in.

If IB don't come back to you, we can expedite creating an ibapi package in PyPi.

Great job though - this is really fantastic. If you want me to merge it in, I'll do so today.

[femtotrader]

A workaround may be to do something like:

pip install git+https://....

in .travis.yml

[ryankennedyio]

@femtotrader from memory the IB API is actually part of a bundled ZIP, which is more annoying.

I'll have to look into if travis can support custom shell commands to install dependancies.

[femtotrader]

Maybe you might contact Shail Mangla at opensource at interactivebrokers.com

[ryankennedyio]

Yep @femtotrader I've opened an issue on the private github there, unfortunately PyPi licensing isn't compatible with what IB need.

We'll have to do a custom install from the zip download... hopefully, I think it should be possible using Travis' before_install feature.

[ryankennedyio]

Live data done. A few code smells, so I've liberally decorated it with TODOs -- wouldn't mind a fresh set of eyes to look over them.

If I can find an easy solution for travis this PR should be ready to merge today.

[ryankennedyio]

Have to drop Python 2.7 as not supported by IB API.
Also seemed to install the ibapi module, but tests still failed with no such module found.