tv_grab_nl.py

#!/usr/bin/env python2
# -*- coding: utf-8 -*-

# Python 3 compatibility
from __future__ import unicode_literals
# from __future__ import print_function

description_text = """
    SYNOPSIS

    tv_grab_nl_py is a python script that trawls tvgids.nl for TV
    programming information and outputs it in XMLTV-formatted output (see
    http://membled.com/work/apps/xmltv). Users of MythTV
    (http://www.mythtv.org) will appreciate the output generated by this
    grabber, because it fills the category fields, i.e. colors in the EPG,
    and has logos for most channels automagically available. Check the
    website below for screenshots.  The newest version of this script can be
    found here:

         https://github.com/tvgrabbers/tvgrabnlpy/

    USAGE

    Check the web site above and/or run script with --help and start from there

    REQUIREMENTS

    * Python 2.6 or 2.7
    * Connection with the Internet

    QUESTIONS

    Questions (and patches) are welcome at:
    http://www.pwdebruin.net/mailman/listinfo/tv_grab_nl_py_pwdebruin.net
    https://github.com/tvgrabbers/tvgrabnlpy/issues
    https://groups.google.com/forum/#!forum/tvgrabnlpy

    UPGRADE NOTES

    If you were using tv_grab_nl from the XMLTV bundle then enable the
    compat flag or use the --compat command-line option.  Otherwise, the
    xmltvid's are wrong and you will not see any new data in MythTV.

    HISTORY

    tv_grab_nl_py used to be called tv_grab_nl_pdb, created by Paul de Bruin
    and first released on 2003/07/09. At the same time the code base switched
    from using CVS to SVN at Google Code, and as a result the version numbering
    scheme has changed. The lastest official release of tv_grab_nl_pdb is 0.48.
    The first official release of tv_grab_nl_py is 6. In 2012, The codebase
    moved to Git, and the version number was changed once more. The latest
    subversion release of tv_grab_nl_py is r109. The first Git release of
    tv_grab_nl_py is 2012-03-11 12:03.

    As of december 2014/ januari 2015 Version 2.0.0:
      Upgrading argument processing from getopt to argparse.
      Also adding some options and adding to help text.
      Fixing a small bug preventing multiple word details like 'jaar van
        premiere' from being proccessed.
      Adding genre/subgenre translation table and file (tv_grab_nl_py.set).
        Automatically adding new genre/subgenre combinations on every scan.
        Still looking into the way MythTV handles this.
        This contains also other translation tables which mostly get updated on
        every scan and gets created with defaults if not existing.
      Adding titlesplit exception list to tv_grab_nl_py.set. Especially for
        spin-off series like 'NCIS: Los Angeles'.
      Adding optional default options file and creation.  (tv_grab_nl_py.opt)
      Adding optional proccessing of HD attribute.
      Adding session log function (to the configname with .log added)
        the last log is saved to .old (like with .conf, .opt and .set files)
      Adding rtl.nl lookup for the 7 RTL channels. This adds season/episode info
        and lookup further than 4 days in the future, defaulting to 14 days.
        Genre info is missing. Timing and description from rtl.nl is used over
        tvgids.nl
      Adding  teveblad.be lookup, mainly for belgium channels. This adds
        season/episode info and lookup up to 7 days. Dutch channels only
        have prime-time info and the commercial channels are missing.
        Genre info is basic. Timing for the Belgium channels is used over
        tvgids.nl
      Adding tvgids.tv lookup. This adds lookup up to 14 days with decent genre
        info.
      Merged tv_grab_nl_py.opt into tv_grab_nl_py.conf and added several
        translation tables to tv_grab_nl_py.set.
      Moving html proccessing from pure regex filtering to ElementTree
      Reorganised code to be more generic to make adding new sources easer
        and as preparation for a configuration module. Also put the different
        sources in parallel threads.
      Working on more intelligent description proccessing.
      Working in ever more intelligent source merging.
      Working on a configuration module.
      Possibly adding ttvdb.com and tmdb3.com lookup for missing descriptions
        and season/episode info
      Possibly adding more optional (foreign) sources (atlas?)

    CONTRIBUTORS

    Main author: Paul de Bruin (paul at pwdebruin dot net)
    Current maintainer: Freek Dijkstra (software at macfreek dot nl)
    Currently 'december 2014' the latest version of '2012-03-27' adapted by:
    Hika van den Hoven hikavdh at gmail dot com, but also active on the
    mythtv list: mythtv-users at mythtv dot org

    Michel van der Laan made available his extensive collection of
    high-quality logos that is used by this script.

    Several other people have provided feedback and patches:
    Huub Bouma, Michael Heus, Udo van den Heuvel, Han Holl, Hugo van der Kooij,
    Roy van der Kuil, Ian Mcdonald, Dennis van Onselen, Remco Rotteveel, Paul
    Sijben, Willem Vermin, Michel Veerman, Sietse Visser, Mark Wormgoor.

    LICENSE

    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <http://www.gnu.org/licenses/>.
"""

# Modules we need
import re, sys, codecs, locale, argparse
import time, datetime, random, io, json, shutil
import os, os.path, pickle, smtplib, httplib
import traceback, socket, sqlite3, difflib
try:
    import urllib.request as urllib
except ImportError:
    import urllib2 as urllib
try:
    from html.entities import name2codepoint
except ImportError:
    from htmlentitydefs import name2codepoint
from copy import deepcopy
from threading import Thread, Lock, Event, active_count, Semaphore
from threading import enumerate as enumthreads
from xml.sax import saxutils
from xml.etree import cElementTree as ET
from Queue import Queue, Empty
from email.mime.text import MIMEText
try:
    unichr(42)
except NameError:
    unichr = chr    # Python 3

# check Python version
if sys.version_info[:3] < (2,7,9):
    sys.stderr.write("tv_grab_nl_py requires Pyton 2.7.9 or higher\n")
    sys.exit(2)

if sys.version_info[:2] >= (3,0):
    sys.stderr.write("tv_grab_nl_py does not yet support Pyton 3 or higher.\nExpect errors while we proceed\n")

locale.setlocale(locale.LC_ALL, '')
# XXX: fix to prevent crashes in Snow Leopard [Robert Klep]
if sys.platform == 'darwin' and sys.version_info[:3] == (2, 6, 1):
    try:
        urllib.urlopen('http://localhost.localdomain')
    except:
        pass

class AmsterdamTimeZone(datetime.tzinfo):
    """Timezone information for Amsterdam"""
    def __init__(self):
        # calculate for the current year:
        year = datetime.date.today().year
        d = datetime.datetime(year, 4, 1, 2, 0)  # Starts last Sunday in March 02:00:00
        self.dston = d - datetime.timedelta(days=d.weekday() + 1)
        d = datetime.datetime(year, 11, 1, 2, 0) # Ends last Sunday in October 02:00:00
        self.dstoff = d - datetime.timedelta(days=d.weekday() + 1)

    def tzname(self, dt):
        return unicode('CET_CEST')

    def utcoffset(self, dt):
        return datetime.timedelta(hours=1) + self.dst(dt)

    def dst(self, dt):

        if self.dston <=  dt.replace(tzinfo=None) < self.dstoff:
            return datetime.timedelta(hours=1)

        else:
            return datetime.timedelta(0)
# end AmsterdamTimeZone

class UTCTimeZone(datetime.tzinfo):
    """UTC Timezone"""
    def tzname(self, dt):
        return unicode('UTC')

    def utcoffset(self, dt):
        return datetime.timedelta(0)

    def dst(self, dt):
        return datetime.timedelta(0)

# end UTCTimeZone

CET_CEST = AmsterdamTimeZone()
UTC  = UTCTimeZone()

config = None

class Logging(Thread):
    """The tread that manages all logging.
    The function below puts them in a queue that is sampled.
    So logging can start after the queue is opend when this class is called below"""
    def __init__(self):
        Thread.__init__(self, name = 'logging')
        self.quit = False
        self.log_level = 175
        self.quiet = False
        self.graphic_frontend = False
        self.log_queue = Queue()
        self.log_output = None
        self.log_string = []
        self.all_at_details = False
        try:
            codecs.lookup(locale.getpreferredencoding())
            self.local_encoding = locale.getpreferredencoding()

        except LookupError:
            if os.name == 'nt':
                self.local_encoding = 'windows-1252'

            else:
                self.local_encoding = 'utf-8'

    def run(self):
        self.log_output = config.log_output
        lastcheck = datetime.datetime.now()
        checkinterfall = 60
        while True:
            try:
                if self.quit and self.log_queue.empty():
                    if config.opt_dict['mail_log']:
                        config.send_mail(self.log_string, config.opt_dict['mail_log_address'])

                    return(0)

                if (datetime.datetime.now() - lastcheck).total_seconds() > checkinterfall:
                    lastcheck = datetime.datetime.now()
                    self.check_thread_sanity()

                try:
                    message = self.log_queue.get(True, 5)

                except Empty:
                    continue

                if message == None:
                    continue

                if isinstance(message, (str, unicode)):
                    if message == 'Closing down\n':
                        self.quit=True

                    self.writelog(message)
                    continue

                elif isinstance(message, (list ,tuple)):
                    llevel = message[1] if len(message) > 1 else 1
                    ltarget = message[2] if len(message) > 2 else 3
                    if message[0] == None:
                        continue

                    if message[0] == 'Closing down\n':
                        self.quit = True

                    if isinstance(message[0], (str, unicode)):
                        self.writelog(message[0], llevel, ltarget)
                        continue

                    elif isinstance(message[0], (list, tuple)):
                        for m in message[0]:
                            if isinstance(m, (str, unicode)):
                                self.writelog(m, llevel, ltarget)

                        continue

                self.writelog('Unrecognized log-message: %s of type %s\n' % (message, type(message)))

            except:
                traceback.print_exc()
                pass

    def check_thread_sanity(self):
        idle_timeout = 180
        chan_count = {}
        for t in range(-1, 5):
            chan_count[t] = 0

        for t in enumthreads():
            if t.name[:8] == 'channel-':
                state = t.state
                chan_count[-1] += 1
                chan_count[state] += 1
                if state in (0, 3):
                    continue

                elif state == 1:
                    # Waiting for a basepage
                    s = t.source
                    if not isinstance(s, int):
                        continue

                    sc = xml_output.channelsource[s]
                    if sc.is_alive() and sc.state < 2:
                        continue

                    t.source_data[s].set()

                elif state == 2:
                    # Waiting for a child channel
                    s = t.source
                    if not s in config.channels.keys():
                        continue

                    sc = config.channels[s]
                    if sc.is_alive() and sc.state < 3:
                        continue

                    sc.child_data.set()

        if not self.all_at_details and chan_count[-1] == chan_count[4]:
            # All channels are at least waiting for details
            self.all_at_details = True

        if self.all_at_details:
            for t in enumthreads():
                if t.name[:7] == 'source-':
                    waittime = None
                    if isinstance(t.lastrequest, datetime.datetime):
                        waittime = (datetime.datetime.now() - t.lastrequest).total_seconds()

                    if waittime == None or waittime < idle_timeout:
                        break

            else:
                # All sources are waiting more then idle_timeout
                # So we tell all channels nothing more is coming
                for t in enumthreads():
                    if t.name[:8] == 'channel-':
                        t.detail_data.set()

    def writelog(self, message, log_level = 1, log_target = 3):
        def now():
             return datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S %Z') + ': '

        try:
            if message == None:
                return

            # If config is not yet available
            if (config == None) and (log_target & 1):
                sys.stderr.write(('Error writing to log. Not (yet) available?\n').encode(self.local_encoding, 'replace'))
                sys.stderr.write(message.encode(self.local_encoding, 'replace'))
                return

            # Log to the Frontend. To set-up later.
            if self.graphic_frontend:
                pass

            # Log to the screen
            elif log_level == 0 or ((not self.quiet) and (log_level & self.log_level) and (log_target & 1)):
                sys.stderr.write(message.encode(self.local_encoding, 'replace'))

            # Log to the log-file
            if (log_level == 0 or ((log_level & self.log_level) and (log_target & 2))) and self.log_output != None:
                if '\n' in message:
                    message = re.split('\n', message)

                    for i in range(len(message)):
                        if message[i] != '':
                            self.log_output.write(now() + message[i] + '\n')
                            if config.opt_dict['mail_log']:
                                self.log_string.append(now() + message[i] + '\n')

                else:
                    self.log_output.write(now() + message + '\n')
                    if config.opt_dict['mail_log']:
                        self.log_string.append(now() + message + '\n')

                self.log_output.flush()

        except:
            sys.stderr.write((now() + 'An error ocured while logging!\n').encode(self.local_encoding, 'replace'))
            traceback.print_exc()

# end Logging
logging = Logging()

def log(message, log_level = 1, log_target = 3):
    # If logging not (jet) available, make sure important messages go to the screen
    if (logging.log_output == None) and (log_level < 2) and (log_target & 1):
        if isinstance(message, (str, unicode)):
            sys.stderr.write(message.encode(logging.local_encoding, 'replace'))

        elif isinstance(message, (list ,tuple)):
            for m in message:
                sys.stderr.write(m.encode(logging.local_encoding, 'replace'))

        if log_target & 2:
            logging.log_queue.put([message, log_level, 2])

    else:
        logging.log_queue.put([message, log_level, log_target])

# end log()

class Configure:
    """This class holds all configuration details and manages file IO"""

    def __init__(self):
        """
        DEFAULT OPTIONS - Edit if you know what you are doing
        """
        # Version info as returned by the version function
        self.name ='tv_grab_nl_py'
        self.major = 2
        self.minor = 2
        self.patch = 21
        self.patchdate = u'20170317'
        self.alfa = False
        self.beta = False

        self.cache_return = Queue()
        self.channels = {}
        self.chan_count = 0
        self.opt_dict = {}
        # This must alway stay off. It can be turned on by a graphic frontend
        # When this runs as a module
        self.opt_dict['graphic_frontend'] = False

        # Used for creating extra debugging output to beter the code
        self.write_info_files = False

        # This handles what goes to the log and screen
        # 0 Nothing (use quiet mode to turns of screen output, but keep a log)
        # 1 include Errors and Warnings
        # 2 include page fetches
        # 4 include summaries
        # 8 include detail fetches and ttvdb lookups to the screen
        # 16 include detail fetches and ttvdb lookups to the log
        # 32 include matchlogging (see below)
        # 64 Title renames
        # 128 ttvdb.com lookup failures
        self.opt_dict['log_level'] = 175
        # The log filehandler, gets set later
        self.log_output = None

        # What match results go to the log/screen (needs code 32 above)
        # 0 = Log Nothing (just the overview)
        # 1 = log not matched programs
        # 2 = log left over programs
        # 4 = Log matches
        # 8 = Added from Groupslots
        self.opt_dict['match_log_level'] = 11

        # A selection of user agents we will impersonate, in an attempt to be less
        # conspicuous to the tvgids.nl police.
        self.user_agents = [ 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9',
               'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.19) Gecko/20081216 Ubuntu/8.04 (hardy) Firefox/2.0.0.19',
               'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
               'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.75.14 (KHTML, like Gecko) Version/7.0.3 Safari/7046A194A',
               'Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 7.0; InfoPath.3; .NET CLR 3.1.40767; Trident/6.0; en-IN)',
               'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0',
               'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.1 Safari/537.36',
               'Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0']

        # default encoding iso-8859-1 is general and iso-8859-15 is with euro support
        self.httpencoding = 'iso-8859-15'
        #~ self.httpencoding = 'utf-8'
        self.file_encoding = 'utf-8'

        # seed the random generator
        random.seed(time.time())

        # default configuration file locations
        self.hpath = ''
        if 'HOME' in os.environ:
            self.hpath = os.environ['HOME']
        elif 'HOMEPATH' in os.environ:
            self.hpath = os.environ['HOMEPATH']
        # extra test for windows users
        if os.name == 'nt' and 'USERPROFILE' in os.environ:
            self.hpath = os.environ['USERPROFILE']

        self.xmltv_dir = u'%s/.xmltv' % self.hpath
        self.config_file = u'%s/tv_grab_nl_py.conf' % self.xmltv_dir
        self.log_file = u'%s/tv_grab_nl_py.log' % self.xmltv_dir
        self.settings_file = u'%s/tv_grab_nl_py.set' % self.xmltv_dir

        # cache the detail information.
        self.program_cache_file = u'%s/program_cache' % self.xmltv_dir
        self.clean_cache = True
        self.clear_cache = False

        # where the output goes. None means to the screen (stdout)
        self.opt_dict['output_file'] = None

        # some variables for mailing the log
        self.opt_dict['mail_log'] = False
        self.opt_dict['mailserver'] = 'localhost'
        self.opt_dict['mailport'] = 25
        self.opt_dict['mail_log_address'] = 'postmaster'
        self.opt_dict['mail_info_address'] = None

        # how many seconds to wait before we timeout on a
        # url fetch, 10 seconds seems reasonable
        # and the maximum of simultaneous fetches that can occure
        self.opt_dict['global_timeout'] = 10
        self.opt_dict['max_simultaneous_fetches'] = 5

        # Wait a random number of seconds between each page fetch.
        # We want to be nice and not hammer tvgids.nl (these are the
        # friendly people that provide our data...).
        # Also, it appears tvgids.nl throttles its output.
        # So there, there is not point in lowering these numbers, if you
        # are in a hurry, use the (default) fast mode.
        self.nice_time = [1, 2]

        # Experimental strategy for clumping overlapping programming, all programs that overlap more
        # than max_overlap minutes, but less than the length of the shortest program are clumped
        # together. Highly experimental and disabled for now.
        self.do_clump = False

        # First fill the dict with some defaults
        # no output
        self.opt_dict['quiet'] = False

        # Fetch data in fast mode, i.e. do NOT grab all the detail information,
        # fast means fast, because as it then does not have to fetch a web page for each program
        self.opt_dict['fast'] = False

        # The day to start grabbing 0 means now
        self.opt_dict['offset'] = 0

        # the total number of days to fetch
        # the first four come from tvgids.nl the rest from tvgids.tv
        self.opt_dict['days'] = 14

        # None means all in slow-mode and none in fast-mode
        # Setting it to a value sets 'fast' always to False (i.e. to slow-mode)
        self.opt_dict['slowdays'] = None

        self.opt_dict['disable_source'] = []
        self.opt_dict['disable_detail_source'] = []
        self.opt_dict['disable_ttvdb'] = False
        self.ttvdb_disabled_groups = (6, 8, 11, 12, 13, 17)
        # enable this option if you were using tv_grab_nl, it adjusts the generated
        # xmltvid's so that everything works.
        self.opt_dict['compat'] = False
        self.opt_dict['legacy_xmltvids'] = False

        # Maximum length in minutes of gaps/overlaps between programs to correct
        self.opt_dict['max_overlap'] = 10

        # Strategy to use for correcting overlapping prgramming:
        # 'average' = use average of stop and start of next program
        # 'stop'    = keep stop time of current program and adjust start time of next program accordingly
        # 'start'   = keep start time of next program and adjust stop of current program accordingly
        # 'none'    = do not use any strategy and see what happens
        self.opt_dict['overlap_strategy'] = 'average'

        # insert url of channel logo into the xml data, this will be picked up by mythfilldatabase
        self.opt_dict['logos'] = True

        # Maximum number of characters to use for program description.
        # Different values may work better in different versions of MythTV.
        self.opt_dict['desc_length'] = 475

        # enable this option if you do not want the tvgids categories being translated into
        # MythTV-categories (genres)
        self.opt_dict['cattrans'] = True

        # mark programs with the HD 1080i tag in the output
        # leave off if you only record analog SD
        self.opt_dict['mark_hd'] = False

        # don't convert all the program date/times to UTC (GMT) timezone.
        # by default the current timezone is Europe/Amsterdam. This works fine
        # if you are located in the Amsterdam timezone, but not if you live abroad
        # in another timezone. If you want to use the UTC timestamp in combination
        # with mythtv, be sure to set the timezone in mythtv to 'auto'
        # (TimeOffset in Settings table)
        self.opt_dict['use_utc'] = False

        # Whether to use the split double episodes regularily seen on teveblad.be
        self.opt_dict['use_split_episodes'] = True

        # After configure, place the active channels in a separate group on top of the list
        self.opt_dict['group_active_channels'] = False

        # Whether to always update Channel- names, groups and prime_sources settings from the sourcematching.json data
        self.opt_dict['always_use_json'] = True

        self.weekdagen = ('zondag', 'maandag', 'dinsdag', 'woensdag', 'donderdag', 'vrijdag', 'zaterdag')
        self.monthnames = ('dummy', 'januari', 'februari', 'maart', 'april', 'mei', 'juni',
                                    'juli', 'augustus', 'september', 'oktober', 'november', 'december')
        # The values for the Kijkwijzer
        # Possible styles are
        # long, short, single and none
        self.opt_dict['kijkwijzerstijl'] = 'short'

        self.kijkwijzer = {'1': {'code': 'AL','text': 'Voor alle leeftijden',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/al_transp.png'},
                        '2': {'code': '6+','text': 'Afgeraden voor kinderen jonger dan 6 jaar',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/6_transp.png'},
                        '9': {'code': '9+','text': 'Afgeraden voor kinderen jonger dan 9 jaar',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/9_transp.png'},
                        '3': {'code': '12+','text': 'Afgeraden voor kinderen jonger dan 12 jaar',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/12_transp.png'},
                        '4': {'code': '16+','text': 'Niet voor personen tot 16 jaar',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/16_transp.png'},
                        'g': {'code': 'Geweld','text': 'Geweld',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/geweld_transp.png'},
                        'a': {'code': 'Angst','text': 'Angst',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/angst_transp.png'},
                        's': {'code': 'Seks','text': 'Seks',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/seks_transp.png'},
                        't': {'code': 'Grof','text': 'Grof taalgebruik',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/grof_transp.png'},
                        'h': {'code': 'Drugs','text': 'drugs- en/of alcoholmisbruik',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/drugs_transp.png'},
                        'd': {'code': 'Discriminatie','text': 'Discriminatie',
                        'icon':'http://tvgidsassets.nl/img/kijkwijzer/discriminatie_transp.png'}}

        self.tvkijkwijzer = {'AL':'1',
                                      '6':'2',
                                      '9':'9',
                                      '12':'3',
                                      '16':'4',
                                      'angst':'a',
                                      'geweld':'g',
                                      'seks':'s',
                                      'taal':'t'}

        self.vrtkijkwijzer = {'6+':'2',
                                      '9+':'9',
                                      '12+':'3',
                                      '16+':'4'}

        # Create a role translation dictionary for the xmltv credits part
        # The keys are the roles used by tvgids.nl (lowercase please)
        self.roletrans = {"regisseur"                         : "director",
                             "regie"                                        : "director",
                             "met"                                            : "actor",
                             "acteurs"                                    : "actor",
                             "acteursnamen_rolverdeling": "actor",
                             "gastacteur"                              : "guest",
                             "scenario"                                  : "writer",
                             "scenario schrijver"              : "writer",
                             "componist"                                : "composer",
                             "presentatie"                            : "presenter",
                             "presentator"                            : "presenter",
                             "verslaggever"                          : "reporter",
                             "commentaar"                              : "commentator",
                             "adapter"                                    : "adapter",
                             "producer"                                  : "producer",
                             "editor"                                      : "editor"}

        # A list of countries with their 2 digit version:
        self.coutrytrans = {}

        # List of titles not to split with title_split().
        # these are mainly spin-off series like NCIS: Los Angeles
        self.notitlesplit = []

        # Parts to remove from a title
        self.groupnameremove = ['kro detectives', 'detectives', 'premiére']
        # Titles to rename
        self.titlerename = {'navy ncis': 'NCIS',
                                        'inspector banks': 'DCI Banks'}

        self.chan_opt = {}
        self.chan_opt['bool'] = ['fast', 'compat', 'logos', 'cattrans', 'mark_hd', 'add_hd_id',
                'append_tvgidstv', 'disable_ttvdb', 'use_split_episodes', 'legacy_xmltvids']
        self.chan_opt['string'] = ['xmltvid_alias']
        self.chan_opt['int'] = ['slowdays', 'max_overlap', 'desc_length']
        self.chan_opt['all'] = ['prime_source', 'prefered_description', 'overlap_strategy']
        self.chan_opt['all'].extend(self.chan_opt['bool'])
        self.chan_opt['all'].extend(self.chan_opt['string'])
        self.chan_opt['all'].extend(self.chan_opt['int'])

        # Create a category translation dictionary
        # Look in mythtv/themes/blue/ui.xml for all category names
        # The keys are the categories used by tvgids.nl (lowercase please)
        # See the file ~/.xmltv/tv_grab_nl_py.set created after the first run and edit there!
        self.cattrans = { (u'', u'')                                                 : u'Unknown',
                             (u'amusement', u'')                                      : u'Talk',
                             (u'amusement', u'quiz')                              : u'Game',
                             (u'amusement', u'spelshow')                      : u'Game',
                             (u'amusement', u'muziekshow')                  : u'Art/Music',
                             (u'amusement', u'muziekprogramma')        : u'Art/Music',
                             (u'amusement', u'dansprogramma')            : u'Art/Music',
                             (u'amusement', u'cabaret')                        : u'Art/Music',
                             (u'amusement', u'variete')                        : u'Art/Music',
                             (u'amusement', u'sketches')                      : u'Art/Music',
                             (u'amusement', u'stand-up comedy')        : u'Art/Music',
                             (u'amusement', u'stand-up comedy, sketches'): u'Art/Music',
                             (u'amusement', u'erotisch programma')  : u'Adult',
                             (u'amusement', u'komedie')                        : u'Comedy',
                             (u'amusement', u'klusprogramma')            : u'Home/How-to',
                             (u'amusement', u'hobbyprogramma')          : u'Home/How-to',
                             (u'amusement', u'lifestyleprogramma')  : u'Home/How-to',
                             (u'amusement', u'modeprogramma')            : u'Home/How-to',
                             (u'amusement', u'kookprogramma')            : u'Cooking',
                             (u'amusement', u'realityserie')              : u'Reality',
                             (u'documentaire', u'')                                : u'Documentary',
                             (u'educatief', u'')                                      : u'Educational',
                             (u'film', u'')                                                : u'Film',
                             (u'korte film', u'')                                    : u'Film',
                             (u'info', u'')                                                : u'News',
                             (u'info', u'business')                                : u'Bus./financial',
                             (u'info', u'documentary')                          : u'Documentary',
                             (u'info', u'science')                                  : u'Science/Nature',
                             (u'informatief, amusement', u'')            : u'Educational',
                             (u'informatief, amusement', u'kookprogramma'): u'Cooking',
                             (u'informatief, kunst en cultuur', u''): u'Arts/Culture',
                             (u'informatief, wetenschap', u'')          : u'Science/Nature',
                             (u'informatief', u'')                                  : u'Educational',
                             (u'informatief', u'wetenschappelijk programma'): u'Science/Nature',
                             (u'informatief', u'techniek')                  : u'Science/Nature',
                             (u'informatief', u'documentaire')          : u'Documentary',
                             (u'informatief', u'gezondheid')              : u'Health',
                             (u'informatief', u'fitnessprogramma')  : u'Health',
                             (u'informatief', u'gymnastiekprogramma'): u'Health',
                             (u'informatief', u'medisch programma'): u'Health',
                             (u'informatief', u'medisch praatprogramma'): u'Health',
                             (u'informatief', u'docusoap')                  : u'Reality',
                             (u'informatief', u'realityprogramma')  : u'Reality',
                             (u'informatief', u'realityserie')          : u'Reality',
                             (u'informatief', u'praatprogramma')      : u'Talk',
                             (u'informatief', u'jeugdprogramma')      : u'Children',
                             (u'jeugd', u'')                                              : u'Children',
                             (u'kunst/cultuur', u'')                              : u'Arts/Culture',
                             (u'kunst en cultuur', u'')                        : u'Arts/Culture',
                             (u'magazine', u'')                                        : u'Talk',
                             (u'muziek', u'')                                            : u'Art/Music',
                             (u'natuur', u'')                                            : u'Science/Nature',
                             (u'nieuws/actualiteiten', u'')                : u'News',
                             (u'news', u'')                                                : u'News',
                             (u'religieus', u'')                                      : u'Religion',
                             (u'serie/soap', u'')                                    : u'Drama',
                             (u'serie/soap', u'jeugdserie')                : u'Children',
                             (u'serie/soap', u'animatieserie')          : u'Children',
                             (u'serie/soap', u'tekenfilmserie')        : u'Children',
                             (u'serie/soap', u'soap')                            : u'Soap',
                             (u'serie/soap', u'comedyserie')              : u'Comedy',
                             (u'serie/soap', u'komedieserie')            : u'Comedy',
                             (u'serie/soap', u'detectiveserie')        : u'Crime/Mystery',
                             (u'serie/soap', u'misdaadserie')            : u'Crime/Mystery',
                             (u'serie/soap', u'fantasyserie')            : u'Sci-fi/Fantasy',
                             (u'serie/soap', u'sciencefictionserie'): u'Sci-fi/Fantasy',
                             (u'serie/soap', u'actieserie')                : u'Action',
                             (u'sport', u'')                                              : u'Sports',
                             (u'talks', u'')                                              : u'Talk',
                             (u'talkshow', u'')                                        : u'Talk',
                             (u'wetenschap', u'')                                    : u'Science/Nature',
                             (u'overige', u'')                                          : u'Unknown'}

        self.genre_list = []

        # These ara all dicts used in merging the sources
        self.source_cattrans = {}
        self.new_cattrans = {}
        # tvgids.tv subgenre to genre translation table
        self.source_cattrans[1] = {'euromillions': 'Amusement',
                                 'erotisch magazine': 'Amusement',
                                 'reality-reeks': 'Amusement',
                                 'keno': 'Amusement',
                                 'loterij': 'Amusement',
                                 'spektakel': 'Amusement',
                                 'informatief programma': 'Informatief',
                                 'reportage': 'Informatief',
                                 'biografie': 'Informatief',
                                 'schooltelevisie ': 'Informatief',
                                 'peuterprogramma': 'Jeugd',
                                 'kleuterprogramma': 'Jeugd',
                                 'tekenfilm': 'Jeugd',
                                 'animatiereeks': 'Jeugd',
                                 'theatershow': 'Kunst en Cultuur',
                                 'concert': 'Muziek',
                                 'musical': 'Muziek',
                                 'weerbericht': 'Nieuws/Actualiteiten',
                                 'verkeersinfo': 'Nieuws/Actualiteiten',
                                 'actualiteitenmagazine': 'Nieuws/Actualiteiten',
                                 'actuele reportage': 'Nieuws/Actualiteiten',
                                 'praatprogramma over de actualiteit': 'Nieuws/Actualiteiten',
                                 'voetbal': 'Sport',
                                 'darts': 'Sport',
                                 'golf': 'Sport',
                                 'wielrennen op de weg': 'Sport',
                                 'baanwielrennen': 'Sport',
                                 'tennis': 'Sport',
                                 'veldrijden': 'Sport',
                                 'volleybal': 'Sport',
                                 'motorcross': 'Sport',
                                 'religieuze uitzending': 'Religieus',
                                 'docusoap': 'Informatief',
                                 'sitcom': 'Serie/Soap'}

        self.new_cattrans[1] = []

        # teveblad.be genre translation table
        self.source_cattrans[3] = {'amusement'           : (u'Amusement', u''),
                                 'documentaire'      : (u'Informatief', u'Documentaire'),
                                 'film'                      : (u'Film', u''),
                                 'kinderen'              : (u'Jeugd', u''),
                                 'kunst & cultuur': (u'Kunst en Cultuur', u''),
                                 'magazine'              : (u'Magazine', u''),
                                 'muziek'                  : (u'Muziek', u''),
                                 'nieuws'                  : (u'Nieuws/Actualiteiten', u''),
                                 'reality'                : (u'informatief', u'realityprogramma'),
                                 'serie'                    : (u'Serie/Soap', u''),
                                 'sport'                    : (u'Sport', u''),
                                 'andere'                  : (u'Overige', u'')}

        self.new_cattrans[3] = {}

        # npo.nl genre translation table
        self.source_cattrans[4] = {('nieuws-actualiteiten', ): (u'nieuws/actualiteiten', u''),
                                     ('amusement', ): (u'amusement', u''),
                                     ('amusement', 'komisch', ): (u'amusement', u'komedie'),
                                     ('amusement', 'spel-quiz', ): (u'amusement', u'quiz'),
                                     ('informatief', ): (u'informatief', u''),
                                     ('informatief', 'nieuws-actualiteiten', ): (u'nieuws/actualiteiten', u''),
                                     ('informatief', 'kunst-cultuur', ): (u'informatief', u'kunst/cultuur'),
                                     ('informatief', 'gezondheid-opvoeding', ): (u'informatief', u'gezondheid'),
                                     ('informatief', 'consumenten-informatie', ): (u'informatief', u'consument'),
                                     ('informatief', 'spel-quiz', ): (u'informatief', u'quiz'),
                                     ('informatief', 'koken-eten', ): (u'informatief', u'kookprogramma'),
                                     ('informatief', 'natuur', ): (u'natuur', u''),
                                     ('informatief', 'religieus' ): (u'religieus', u''),
                                     ('religieus', ): (u'religieus', u''),
                                     ('jeugd', ): (u'jeugd', u''),
                                     ('jeugd', 'animatie', ): (u'jeugd', u'animatieserie'),
                                     ('jeugd', 'spel-quiz', ): (u'jeugd', u'quiz'),
                                     ('documentaire', ): (u'documentaire', u''),
                                     ('documentaire', 'kunst-cultuur', ): (u'documentaire', u'kunst/cultuur'),
                                     ('sport', ): (u'sport', u''),
                                     ('sport', 'sport-informatie', ): (u'sport', u'journaal'),
                                     ('animatie', ): (u'serie/soap', u'animatieserie'),
                                     ('natuur', ): (u'natuur', u''),
                                     ('muziek', ): (u'muziek', u''),
                                     ('muziek', 'muziek-populair', ): (u'muziek', u'populair'),
                                     ('muziek', 'muziek-klassiek', ): (u'muziek', u'klassiek'),
                                     ('film', ): (u'film', u''),
                                     ('film', 'animatie', ): (u'film', u'animatieserie'),
                                     ('film', 'spanning', ): (u'film', u'thriller'),
                                     ('wetenschap', ): (u'wetenschap', u''),
                                     ('drama', ): (u'serie/soap', u'drama'),
                                     ('reizen', ): (u'reizen', u''),
                                     ('serie', ): (u'serie/soap', u''),
                                     ('serie', 'soap-serie', ): (u'serie/soap', u'soap')}
                                     #~ '14': (u'serie/soap', u''),
                                     #~ '15': (u'overige', u''),
                                     #~ '18': (u'serie/soap', u'misdaadserie'),
                                     #~ '19': (u'kunst/cultuur', u''),
                                     #~ '20': (u'amusement', u'erotisch programma'),
                                     #~ '23': (u'amusement', u'komedie'),
                                     #~ '26': (u'educatief', u''),
                                     #~ '27': (u'informatief', u'fitnessprogramma'),
                                     #~ '29': (u'jeugd', u'6-12'),
                                     #~ '30': (u'maatschappij', u''),
                                     #~ '32': (u'jeugd', u'2-5'),
                                     #~ '34': (u'muziek', u'klassiek'),
                                     #~ '77': (u'gezondheid-opvoeding', u''),
                                     #~ '79': (u'komisch', u''),
                                     #~ '80': (u'spanning', u''),
                                     #~ '81': (u'consumenten-informatie', u''),
                                     #~ '82': (u'wonen-tuin', u''),
                                     #~ '83': (u'muziek-populair', u''),
                                     #~ '84': (u'spel-quiz', u''),
                                     #~ '85': (u'cabaret', u''),
                                     #~ '86': (u'sport-informatie', u''),
                                     #~ '87': (u'muziek-klassiek', u''),
                                     #~ '88': (u'koken-eten', u''),
                                     #~ '89': (u'geschiedenis', u''),
                                     #~ '90': (u'sport-wedstrijd', u''),
                                     #~ '91': (u'soap-serie', u'')}

        self.new_cattrans[4] = {}
        self.npo_fill = 'Programmainfo en Reclame'

        # horizon.tv genre translation table
        self.source_cattrans[5] ={('13946319', ): ('nieuws/actualiteiten',''),
                                     ('13946319', '13946323'): ('informatief', 'Documentaire'),
                                     ('13946319', '13946324'): ('informatief', 'Discussie'),
                                     ('13946336', ): ('amusement',''),
                                     ('13946336', '13946338'): ('kunst en cultuur', 'Variété'),
                                     ('13946336', '13946340'): ('talkshow', ''),
                                     ('13946352', ): ('sport',''),
                                     ('13946369', ): ('jeugd', ''),
                                     ('13946386', ): ('muziek', ''),
                                     ('13946404', ): ('kunst en cultuur', ''),
                                     ('13946404', '13946407'): ('religieus', ''),
                                     ('13946455', ): ('informatief', ''),
                                     ('13946420', ): ('informatief', ''),
                                     ('13946438', ): ('informatief', ''),
                                     ('13946472', ): ('informatief', ''),
                                     ('13948023', ): ('serie/soap', ''),
                                     ('13948023', '13948024'): ('serie/soap', 'thriller'),
                                     ('13948023', '13948025'): ('serie/soap', 'actieserie'),
                                     ('13948023', '13948026'): ('serie/soap', 'sciencefictionserie'),
                                     ('13948023', '13948027'): ('serie/soap', 'comedyserie'),
                                     ('13948023', '13948028'): ('serie/soap', 'melodrama'),
                                     ('13948023', '13948031'): ('serie/soap', 'historisch'),
                                     ('13948023', '13948032'): ('serie/soap', 'waar gebeurt'),
                                     ('13948023', '13948033'): ('serie/soap', 'detectiveserie')}
        self.new_cattrans[5] = {}

        # humo.be genre translation table
        self.source_cattrans[6] = {'nieuws'       : (u'Nieuws/Actualiteiten', u''),
                                 'current-affairs': (u'Nieuws/Actualiteiten', u'Actualiteiten'),
                                 'magazine'              : (u'Magazine', u''),
                                 'reportage'            : (u'Informatief', u'Reportage'),
                                 'documentaire'      : (u'Informatief', u'Documentaire'),
                                 'talkshow'              : (u'Talkshow', u''),
                                 'reality'                : (u'Amusement', u'Realityserie'),
                                 'kinderen'              : (u'jeugd', u''),
                                 'animated-cartoon': (u'serie/soap', u'animatieserie'),
                                 'serie'                    : (u'Serie/Soap', u''),
                                 'miniserie'            : (u'Serie/Soap', u''),
                                 'soap'                      : (u'Serie/Soap', u'Soap'),
                                 'film'                      : (u'Film', u''),
                                 'tv-film'                : (u'Film', u'TV Film'),
                                 'movie-short'        : (u'Film', u'Korte Film'),
                                 'quiz'                      : (u'Amusement', u'Quiz'),
                                 'spel'                      : (u'amusement', u'spelshow'),
                                 'amusement'            : (u'Amusement', u''),
                                 'religion'              : (u'Religieus', u''),
                                 'muziek'                  : (u'Muziek', u''),
                                 'kunst-cultuur'    : (u'kunst en cultuur', u''),
                                 'sports-football': (u'Sports', u'Voetbal'),
                                 'sports-cycling'  : (u'Sport', u'Wielrennen'),
                                 'sports-formula-1-racing'  : (u'Sport', u'Formule-1'),
                                 'sports-tennis'    : (u'Sport', u'Tennis'),
                                 'sport'                    : (u'Sport', u''),
                                 'andere'                  : (u'Overige', u'')}

        self.new_cattrans[6] = {}

        # vpro.nl genre translation table
        self.source_cattrans[7] ={('g3011', ): ('jeugd', ''),
                                     ('g3012', ): ('film', ''),
                                     ('g3013', ): ('serie/soap', ''),
                                     ('g3014', ): ('sport',''),
                                     ('g3015', ): ('muziek', ''),
                                     ('g3016', ): ('amusement',''),
                                     ('g3017', ): ('informatief', ''),
                                     ('g301721', ): ('Nieuws/Actualiteiten', ''),
                                     ('g3017','g301721'): ('Nieuws/Actualiteiten', ''),
                                     ('g301724', ): (u'kunst/cultuur', u''),
                                     ('g3017', 'g301724'): (u'informatief', u'kunst/cultuur'),
                                     ('g301725'): (u'natuur', u''),
                                     ('g3017', 'g301725', ): (u'natuur', u''),
                                     ('g301726', ): (u'religieus', u''),
                                     ('g3017', 'g301726' ): (u'religieus', u''),
                                     ('g301727', ): (u'informatief, wetenschap', u''),
                                     ('g3017', 'g301727',): (u'informatief, wetenschap', u''),
                                     ('g3018', ): ('informatief', 'Documentaire')}
        self.new_cattrans[7] = {}

        # nieuwsblad.be genre translation table
        self.source_cattrans[8] ={}
        self.new_cattrans[8] = {}

        # primo.eu genre translation table
        self.source_cattrans[9] ={'actua': ('nieuws/actualiteiten', 'actua'),
                                     'amusement': ('amusement', ''),
                                     'documentaire': ('documentaire', ''),
                                     'film': ('film', ''),
                                     'kortfilm': ('film', ''),
                                     'magazine': ('magazine', ''),
                                     'muziek': ('muziek', ''),
                                     'nieuws': ('nieuws/actualiteiten', 'nieuws'),
                                     'reality-tv': ('amusement', 'realityprogramma'),
                                     'religie': ('religieus', ''),
                                     'reportage': ('nieuws/actualiteiten', 'reportage'),
                                     'serie': ('serie/soap', ''),
                                     'spel': ('amusement', 'spelshow'),
                                     'sport': ('sport', ''),
                                     'talkshow': ('amusement', 'talkshow'),
                                     'varia': ('informatief', 'varia'),
                                     'zwart/wit film': ('film', ''),
                                     ('serie', 'actiereeks'): (u'serie/soap', u'actieserie'),
                                     ('serie', 'advocatenreeks'): (u'serie/soap', u'advocatenserie'),
                                     ('serie', 'avonturenreeks'): (u'serie/soap', u'avonturenserie'),
                                     ('serie', 'detectivereeks'): (u'serie/soap', u'detectiveserie'),
                                     ('serie', 'dramareeks'): (u'serie/soap', u'dramaserie'),
                                     ('serie', 'fictiereeks'): (u'serie/soap', u'fictieserie'),
                                     ('serie', 'jeugdreeks'): (u'jeugd', u'serie'),
                                     ('serie', 'komische reeks'): (u'serie/soap', u'comedyserie'),
                                     ('serie', 'minireeks'): (u'serie/soap', u'miniserie'),
                                     ('serie', 'misdaadreeks'): (u'serie/soap', u'misdaadserie'),
                                     ('serie', 'romantische reeks'): (u'serie/soap', u'romantische serie'),
                                     ('serie', 'soapreeks'): (u'serie/soap', u'soap'),
                                     ('serie', 'thrillerreeks'): (u'serie/soap', u'thrillerserie'),
                                     ('serie', 'tragikomische reeks'): (u'serie/soap', u'tragikomische serie'),
                                     ('serie', 'ziekenhuisreeks'): (u'serie/soap', u'ziekenhuisserie')}
        self.new_cattrans[9] = {}

        #vrt.be genre translation table
        self.source_cattrans[10] ={(u'1',): (u'film', u''),
                                                (u'2',): (u'nieuws/actualiteiten', u''),
                                                (u'3',): (u'amusement', u''),
                                                (u'3', u'1'): (u'amusement', u'spelshow'),
                                                (u'4',): (u'sport', u''),
                                                (u'5',): (u'jeugd', u''),
                                                (u'6',): (u'muziek', u''),
                                                (u'7',): (u'kunst en cultuur', u''),
                                                (u'7', u'3'): (u'religieus', u''),
                                                (u'8',): (u'documentaire', u''),
                                                (u'8', u'2'): (u'informatief', u'economie'),
                                                (u'9',): (u'informatief', u''),
                                                (u'10',): (u'informatief', u''),
                                                (u'13', ): (u'serie/soap', u''),
                                                (u'13', u'1'): (u'serie/soap', u''),
                                                (u'13', u'2'): (u'serie/soap', u'detectiveserie'),
                                                (u'13', u'3'): (u'serie/soap', u'actieserie'),
                                                (u'13', u'4'): (u'serie/soap', u'sciencefictionserie'),
                                                (u'13', u'5'): (u'serie/soap', u'komedieserie'),
                                                (u'13', u'6'): (u'serie/soap', u'soap'),
                                                (u'13', u'8'): (u'serie/soap', u'animatieserie'),
                                                (u'14',): (u'informatief', u''),
                                                (u'14', u'1'): (u'informatief', u'realityprogramma'),
                                                (u'14', u'3'): (u'informatief', u'docusoap')}
        self.new_cattrans[10] = {}

        # The following two list get replaced by their sourcematching counterparts
        # Program group names to exclude from a primesource if the counterpart contains details
        self.groupslot_names = ("ochtend- en dagprogramma's", "ochtend - en dagprogramma's",
                                                "nachtprogramma's", "nachtprogrammering", "kinderprogramma's",
                                                "kinder-tv", "kindertijd", "pause", "geen programmagegevens beschikbaar.")

        self.combined_channels = {'5-24443943013': ['0-300'],
                                                     '5-24443943080': ['0-301', '1-cbeebies'],
                                                     '1-veronica': ['0-34', '0-311'],
                                                     '1-ketnet-canvas-2': ['8-ketnet', '8-eenplus']}

        #Channel group names as used in tvgids.tv
        self.group_names = {1: 'Nederlandse kanalen',
                                          2: 'Vlaamse kanalen',
                                          3: 'Engelse kanalen',
                                          4: 'Duitse kanalen',
                                          5: 'Franse kanalen',
                                          6: 'Nederlands Regionaal',
                                          7: 'Overige Nederlands kanalen',
                                          8: 'Vlaams Regionaal',
                                          9: 'Overige Vlaamse kanalen ',
                                         10: 'Internationale kanalen',
                                         11: 'Nederlandse Radio',
                                         12: 'Vlaamse Radio',
                                         13: 'Overige Radio',
                                         99: 'Overig kanalen',
                                         -1: 'Alleen geselecteerde kanalen'}

        self.group_order = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,99]
        # DO NOT CHANGE THIS!
        self.configversion = None
        self.__CONFIG_SECTIONS__ = { 1: u'Configuration',
                                                            2: u'tvgids.nl Channels',
                                                            3: u'Channels'}


        self.__DEFAULT_SECTIONS__ = {1: u'genre conversion table',
                                                             2: u'no title split list',
                                                             3: u'remove groupname list',
                                                             4: u'rename title list',
                                                             5: u'teveblad.be genres',
                                                             6: u'tvgids.tv genres',
                                                             7: u'role translation',
                                                             8: u'npo.nl genres',
                                                             9: u'horizon.tv genres',
                                                             10: u'humo.be genres',
                                                             11: u'vpro.nl genres',
                                                             13: u'primo.eu genres',
                                                             14: u'vrt.be genres'}

    # end Init()

    def version(self, as_string = False):
        """
        return tuple or string with version info
        """
        if as_string and self.alfa:
            return u'%s (Version: %s.%s.%s-p%s-alpha)' % (self.name, self.major, self.minor, '{:0>2}'.format(self.patch), self.patchdate)

        if as_string and self.beta:
            return u'%s (Version: %s.%s.%s-p%s-beta)' % (self.name, self.major, self.minor, '{:0>2}'.format(self.patch), self.patchdate)

        if as_string and not self.beta:
            return u'%s (Version: %s.%s.%s-p%s)' % (self.name, self.major, self.minor, '{:0>2}'.format(self.patch), self.patchdate)

        else:
            return (self.name, self.major, self.minor, self.patch, self.patchdate, self.beta)

    # end version()

    def create_dayoffset(self, current_date):
        dayoffset = {}
        dayoffset['eergisteren'] = -2
        dayoffset['gisteren'] = -1
        dayoffset['vandaag'] = 0
        dayoffset['morgen'] = 1
        dayoffset['overmorgen'] = 2
        for d in range(7):
            dd = self.weekdagen[int(datetime.date.fromordinal(current_date + d).strftime("%w"))]
            dayoffset[dd] = d

        for d in range(14):
            day = datetime.date.fromordinal(current_date + d).strftime("%d")
            month = datetime.date.fromordinal(current_date + d).strftime("%m")
            monthname = self.monthnames[int(month)]
            year = datetime.date.fromordinal(current_date + d).strftime("%Y")
            dayoffset['%s %s' % (int(day), monthname)] = d
            dayoffset['%s %s' % (day, monthname)] = d
            dayoffset['%s %s' % (int(day), monthname.capitalize())] = d
            dayoffset['%s %s' % (day, monthname.capitalize())] = d
            dayoffset['%s %s %s' % (int(day), monthname, year)] = d
            dayoffset['%s %s %s' % (day, monthname, year)] = d
            dayoffset['%s %s %s' % (int(day), monthname.capitalize(), year)] = d
            dayoffset['%s %s %s' % (day, monthname.capitalize(), year)] = d
            dayoffset['%s-%s' % (int(day), int(month))] = d
            dayoffset['%s-%s' % (day, month)] = d
            dayoffset['%s-%s-%s' % (int(day), int(month), year)] = d
            dayoffset['%s-%s-%s' % (day, month, year)] = d

        return dayoffset
    # end create_dayoffset()

    def save_oldfile(self, fle):
        """ save the old file to .old if it exists """
        if os.path.isfile(fle + '.old'):
            os.remove(fle + '.old')

        if os.path.isfile(fle):
            os.rename(fle, fle + '.old')

    # end save_old()

    def open_file(self, file_name, mode = 'rb', encoding = None):
        """ Open a file and return a file handler if success """
        if encoding == None:
            encoding = self.file_encoding

        try:
            if 'b' in mode:
                file_handler =  io.open(file_name, mode = mode)
            else:
                file_handler =  io.open(file_name, mode = mode, encoding = encoding)

        except IOError as e:
            if e.errno == 2:
                log('File: "%s" not found.\n' % file_name)
            else:
                log('File: "%s": %s.\n' % (file_name, e.strerror))
            return None

        return file_handler

    # end open_file ()

    def get_line(self, fle, byteline, isremark = False, encoding = None):
        """
        Check line encoding and if valid return the line
        If isremark is True or False only remarks or non-remarks are returned.
        If None all are returned
        """
        if encoding == None:
            encoding = self.file_encoding

        try:
            line = byteline.decode(encoding)
            line = line.lstrip()
            line = line.replace('\n','')
            if isremark == None:
                return line

            if len(line) == 0:
                return False

            if isremark and line[0:1] == '#':
                return line

            if not isremark and not line[0:1] == '#':
                return line

        except UnicodeError:
            log('%s is not encoded in %s.\n' % (fle.name, encoding))

        return False

    # end get_line()

    def check_encoding(self, fle, encoding = None, check_version = False):
        """
        Check file encoding. Return True or False
        Encoding is stored in self.encoding
        Optionally check for a version string
        and store it in self.configversion
        """
        # regex to get the encoding string
        reconfigline = re.compile(r'#\s*(\w+):\s*(.+)')

        self.encoding = None
        self.configversion = None

        if encoding == None:
            encoding = self.file_encoding

        for byteline in fle.readlines():
            line = self.get_line(fle, byteline, True, self.encoding)
            if not line:
                continue

            else:
                match = reconfigline.match(line)
                if match is not None and match.group(1) == "encoding":
                    encoding = match.group(2)

                    try:
                        codecs.getencoder(encoding)
                        self.encoding = encoding

                    except LookupError:
                        log('%s has invalid encoding %s.\n' % (fle.name, encoding))
                        return False

                    if (not check_version) or self.configversion != None:
                        return True

                    continue

                elif match is not None and match.group(1) == "configversion":
                    self.configversion = float(match.group(2))
                    if self.encoding != None:
                        return True

                continue

        if check_version and self.configversion == None:
            fle.seek(0,0)
            for byteline in fle.readlines():
                line = self.get_line(fle, byteline, False, self.encoding)
                if not line:
                    continue

                else:
                    config_title = re.search('[(.*?)]', line)
                    if config_title != None:
                        self.configversion = float(2.0)
                        break

            else:
                self.configversion = float(1.0)

        if self.encoding == None:
            return False

        else:
            return True

    # end check_encoding()

    def read_commandline(self):
        """Initiate argparser and read the commandline"""
        self.description = 'The Netherlands: %s\n' % self.version(True) + \
                        '  A grabber that grabs tvguide data from tvgids.nl, tvgids.tv, rtl.nl,\n' + \
                        '  teveblad.be, npo.nl and horizon.tv for up to 230+ channels and up to 14 days.\n' + \
                        '  Which it then stores in XMLTV-compatible format.'

        parser = argparse.ArgumentParser(description = self.description, formatter_class=argparse.RawTextHelpFormatter)

        parser.add_argument('-V', '--version', action = 'store_true', default = False, dest = 'version',
                        help = 'display version')

        parser.add_argument('--description', action = 'store_true', default = False, dest = 'description',
                        help = 'prints the above short description of the grabber')

        parser.add_argument('-d', '--long-descr', action = 'store_true', default = False, dest = 'description_long',
                        help = 'prints a long description in english of the grabber')

        parser.add_argument('--capabilities', action = 'store_true', default = False, dest = 'capabilities',
                        help = 'xmltv required option')

        parser.add_argument('--preferredmethod', action = 'store_true', default = False, dest = 'preferredmethod',
                        help = 'returns the preferred method to be called')

        parser.add_argument('--show-sources', action = 'store_true', default = False, dest = 'show_sources',
                        help = 'returns the available sources')

        parser.add_argument('--disable-source', action = 'append', default = [], dest = 'disable_source',
                        metavar = '<source ID>', type = int,
                        help = 'disable a numbered source.\nSee "--show-sources" for a list.')

        parser.add_argument('--show-detail-sources', action = 'store_true', default = False, dest = 'show_detail_sources',
                        help = 'returns the available detail sources')

        parser.add_argument('--show-logo-sources', action = 'store_true', default = False, dest = 'show_logo_sources',
                        help = 'returns the available logo sources')

        parser.add_argument('--disable-detail-source', action = 'append', default = [], dest = 'disable_detail_source',
                        metavar = '<source ID>', type = int,
                        help = 'disable a numbered source for detailfetches.\nSee "--show-detail-sources" for a list.')

        parser.add_argument('--disable-ttvdb', action = 'store_true', default = None, dest = 'disable_ttvdb',
                        help = 'disable fetching extra data from ttvdb.com.')

        parser.add_argument('--add-ttvdb-title', nargs = '*', metavar = '<title>', dest = 'ttvdb_title',
                        help = 'Query ttvdb.com for a series-title and optionally store\n' +
                                    'it with the ID in the DB. Enclose the title in quotes!\n' +
                                    'Optionally add a language code after the title.\n' +
                                    '<cs, da, de, el, en, es, fi, fr, he, hr, hu, it, ja,\n' +
                                    ' ko, nl, no, pl, pt, ru, sl, sv, tr, zh>')

        parser.add_argument('-N', '--nouse-NPO', action = 'store_false', default = None, dest = 'use_npo',
                        help = 'This argument is deprecated, use "--disable-source 4".')

        parser.add_argument('-x', '--compat', action = 'store_true', default = None, dest = 'compat',
                        help = 'append tvgids.nl to the xmltv id\n(use this if you were using tv_grab_nl)')

        parser.add_argument('-X', '--legacy_xmltvids', action = 'store_true', default = None, dest = 'legacy_xmltvids',
                        help = 'remove as in pre 2.2.8 for source 0 and 1 the sourceid\n' +
                                    'from the chanid to get the xmltvid.')

        parser.add_argument('-u', '--utc', action = 'store_true', default = None, dest = 'use_utc',
                        help = 'generate all data in UTC time (use with timezone "auto"\nin mythtv)')

        parser.add_argument('-c', '--configure', action = 'store_true', default = False, dest = 'configure',
                        help = 'create configfile; rename an existing file to *.old.')

        parser.add_argument('--group_active_channels', action = 'store_true', default = False, dest = 'group_active_channels',
                        help = 'After running configure, place all active channels in\n' +
                                    'a separate group on top of the list.\n' +
                                    'Only relevant together with the configure option.')

        parser.add_argument('-C', '--config-file', type = str, default = self.config_file, dest = 'config_file',
                        metavar = '<file>',
                        help = 'name of the configuration file\n<default = \'%s\'>' % self.config_file)

        parser.add_argument('-O', '--save-options', action = 'store_true', default = False, dest = 'save_options',
                        help = 'save the currently defined options to the config file\n' +
                                    'add options to the command-line to adjust the file.')

        parser.add_argument('-A', '--cache', type = str, default = self.program_cache_file, dest = 'program_cache_file',
                        metavar = '<file>',
                        help = 'cache descriptions and use the file to store\n<default = \'%s\'>' % self.program_cache_file)

        parser.add_argument('--clean_cache', action = 'store_true', default = self.clean_cache, dest = 'clean_cache',
                        help = 'clean the cache of outdated data before fetching')

        parser.add_argument('--clear_cache', '--clear-cache', action = 'store_true', default = self.clear_cache, dest = 'clear_cache',
                        help = 'empties the program table before fetching data')

        parser.add_argument('--clear_ttvdb', '--clear-ttvdb', action = 'store_true', default = self.clear_cache, dest = 'clear_ttvdb',
                        help = 'empties the ttvdb table before fetching data')

        parser.add_argument('-W', '--output', type = str, default = None, dest = 'output_file',
                        metavar = '<file>',
                        help = 'file where to send the output <default to the screen>')

        parser.add_argument('--output-windows-codeset',  action = 'store_true', default = False, dest = 'output_codeset',
                        help = 'use for the outputfile Windows codeset (cp1252)\n' +
                                    'instead of utf-8')

        parser.add_argument('-q', '--quiet', action = 'store_true', default = None, dest = 'quiet',
                        help = 'suppress all output.')

        parser.add_argument('-v', '--verbose', action = 'store_false', default = None, dest = 'quiet',
                        help = 'Sent log-info also to the screen.')

        parser.add_argument('-f', '--fast', action = 'store_true', default = None, dest = 'fast',
                        help = 'do not grab details of programming (tvgids.nl/tv)')

        parser.add_argument('-s', '--slow', action = 'store_false', default = None, dest = 'fast',
                        help = '<default> grab details of programming (tvgids.nl/tv)')

        parser.add_argument('-o', '--offset', type = int, default = None, dest = 'offset',
                        metavar = '<days>',
                        help = 'The day to start grabbing <defaults to 0 is today>')

        parser.add_argument('-g', '--days', type = int, default = None, dest = 'days',
                        metavar = '<days>',
                        help = '# number of days to grab from the several sources.\n<max 14 = default>\n' +
                                     'Where every source has itś own max.\n')

        parser.add_argument('-G', '--slowdays', type = int, default = None, dest = 'slowdays',
                        metavar = '<days>',
                        help = 'number of days to grab slow and the rest in fast mode\nDefaults to all days (tvgids.nl/tv)')

        parser.add_argument('-r', '--rtldays', type = int, default = None, dest = 'rtldays',
                        metavar = '<days>', help = 'This option is deprecated and no longer used')

        parser.add_argument('-b', '--tevedays', type = int, default = None, dest = 'tevedays',
                        metavar = '<days>',
                        help = 'This option is deprecated and no longer used')

        parser.add_argument('--logos', action = 'store_true', default = None, dest = 'logos',
                        help = '<default> insert urls to channel icons\n(mythfilldatabase will then use these)')

        parser.add_argument('-n', '--nologos', action = 'store_false', default = None, dest = 'logos',
                        help = 'do not insert urls to channel icons')

        parser.add_argument('-H', '--mark-HD', action = 'store_true', default = None, dest = 'mark_hd',
                        help = 'mark HD programs,\ndo not set if you only record analog SD')

        parser.add_argument('--cattrans', action = 'store_true', default = None, dest = 'cattrans',
                        help = '<default> translate the grabbed genres into\nMythTV-genres. See the tv_grab_nl_py.set file')

        parser.add_argument('-t', '--nocattrans', action = 'store_false', default = None, dest = 'cattrans',
                        help = 'do not translate the grabbed genres into MythTV-genres.\n' +
                                    'it then only uses the basic genres without possibility\n' +
                                     'to differentiate on subgenre.')

        parser.add_argument('-l ', '--desc-length', type = int, default = None, dest = 'desc_length',
                        metavar = '<bytes>',
                        help = 'maximum allowed length of program descriptions in bytes.')

        parser.add_argument('-a', '--overlap_strategy', type = str, default = None, dest = 'overlap_strategy',
                        metavar = '<strategy>',
                        help = 'what strategy to use to correct overlaps:\n' +
                                     '\'avarage\' use average of stop and start of next program.\n          <default>\n' +
                                     '\'stop\'    keep stop time of current program and adjust\n          start time.\n' +
                                     '\'start\'   keep start time of next program and adjust\n          stop time.\n' +
                                     '\'none\'    do not use any strategy and see what happens.\n')

        parser.add_argument('-m', '--max_overlap', type = int, default = None, dest = 'max_overlap',
                        metavar = '<minutes>',
                        help = 'maximum length of overlap between programming to correct\n<default 10 minutes>')

        # Handle the sys.exit(0) exception on --help more gracefull
        try:
            self.args = parser.parse_args()

        except:
            return(0)

    # end read_commandline()

    def read_config(self):
        """Read the configurationfile Return False on failure."""
        self.config_dict = {1:[], 2:[], 3:[], 9:{}}
        f = self.open_file(self.config_file)
        if f == None:
            if not self.args.configure:
                log('Re-run me with the --configure flag.\n')
            return False

        if not self.check_encoding(f, None, True):
            return False

        if self.configversion == 1.0:
            type = 2
            section = self.__CONFIG_SECTIONS__[2]

        else:
            type = 0
            section = ''
            if self.configversion < 2.208:
                log("Adding 'legacy_xmltvids = True'\n", 0)
                self.opt_dict['legacy_xmltvids'] = True
                if not self.args.configure:
                    log("Run with '--configure' to make it permanent\n", 0)

        # Read the configuration into the self.config_dict dictionary
        f.seek(0,0)
        for byteline in f.readlines():
            try:
                line = self.get_line(f, byteline, None, self.encoding)
                if not line:
                    continue

                if line[0:1] == '#' and type != 3:
                    continue

                # Look for section headers
                config_title = re.search('\[(.*?)\]', line)
                if config_title != None:
                    if (config_title.group(1) in self.__CONFIG_SECTIONS__.values()):
                        section = config_title.group(1)
                        for i, v in self.__CONFIG_SECTIONS__.items():
                            if v == config_title.group(1):
                                type = i
                                continue

                        continue

                    # Itś a channel confuration
                    elif config_title.group(1)[0:7].lower() == 'channel':
                        section = config_title.group(1)[7:].strip().lower()
                        type = 9
                        if not section in self.config_dict[9].keys():
                            self.config_dict[9][section] = []
                        continue

                    # Unknown Section header, so ignore
                    else:
                        log('Ignoring unknown section "%s"\n' % config_title.group(1))
                        section = ''
                        type = 0

                # Unknown Section header, so ignore
                elif line[0:1] == '[':
                    section = ''
                    type = 0
                    continue

                if type in (1, 2, 3):
                    self.config_dict[type].append(line)

                elif type == 9:
                    self.config_dict[9][section].append(line)

                else:
                    log('Ignoring configuration line "%s". Outside any known section.\n' % line)


            except:
                log([u'Error reading Config\n', traceback.format_exc()])
                continue

        f.close()
        # Read the General Configuration options
        for line in self.config_dict[1]:
            try:
                # Strip the name from the value
                a = re.split('=',line)
                cfg_option = a[0].lower().strip()
                # Boolean Values
                if cfg_option in ('write_info_files', 'quiet', 'fast', 'compat', 'logos', 'cattrans', 'mail_log', \
                  'mark_hd', 'use_utc', 'disable_ttvdb', 'use_split_episodes', 'group_active_channels', \
                  'always_use_json', 'legacy_xmltvids'):
                    if len(a) == 1:
                        self.opt_dict[cfg_option] = True

                    elif a[1].lower().strip() in ('true', '1' , 'on'):
                        self.opt_dict[cfg_option] = True

                    else:
                        self.opt_dict[cfg_option] = False

                # String values that can be None
                elif cfg_option in ('output_file', 'mail_info_address'):
                    self.opt_dict[cfg_option] = None if (len(a) == 1 or a[1].lower().strip() == 'none') else a[1].strip()

                elif len(a) == 2:
                    cfg_value = a[1].lower().strip()
                    if cfg_option == 'use_npo':
                        if cfg_value in ('false', '0' , 'off'):
                            self.validate_option('disable_source', value = 4)

                    # String values
                    elif cfg_option in ('mailserver', 'mail_log_address'):
                        self.opt_dict[cfg_option] = a[1].strip()

                    # Select Values
                    elif cfg_option == 'overlap_strategy':
                        if cfg_value in ('average', 'stop', 'start'):
                            self.opt_dict[cfg_option] = cfg_value

                        else:
                            self.opt_dict[cfg_option] = 'none'

                    elif cfg_option == 'kijkwijzerstijl':
                        if cfg_value in ('long', 'short', 'single'):
                            self.opt_dict[cfg_option] = cfg_value

                        else:
                            self.opt_dict[cfg_option] = 'none'

                    # Integer Values
                    elif cfg_option in ('log_level', 'match_log_level', 'offset', 'days', 'slowdays', 'mailport', \
                      'max_overlap', 'desc_length', 'disable_source', 'disable_detail_source', 'data_version', \
                      'max_simultaneous_fetches', 'global_timeout'):
                        try:
                            cfg_value = int(cfg_value)

                        except ValueError:
                            if (cfg_option == 'slowdays') and (cfg_value == 'none'):
                                self.opt_dict[cfg_option] = None

                        else:
                            if cfg_option in ('disable_source', 'disable_detail_source'):
                                self.validate_option(cfg_option, value = cfg_value)

                            else:
                                self.opt_dict[cfg_option] = cfg_value

            except:
                log(['Invalid line in Configuration section of config file %s:' % (self.config_file),'%r\n' % (line)])

        # Read the channel stuff up to version 2.0
        channel_names = {}
        old_chanids = {}
        if self.configversion <= 2.0:
            for line in self.config_dict[2]:
                try:
                    channel = line.split(None, 1) # split on first whitespace
                    chanid = unicode(channel[0]).strip()
                    channame = unicode(channel[1]).strip()
                    self.channels[chanid] = Channel_Config(chanid, channame)
                    self.channels[chanid].active = True
                    if chanid in channel_names.keys():
                        channel_names[chanid].append(channame.lower())

                    else:
                        channel_names[chanid] = [channame.lower()]

                except:
                    log(['Invalid line in Channels section of config file %s:' % (self.config_file),'%r\n' % (line)])

        # Changed Channel config since version 2.1
        if self.configversion >= 2.1:
            test_as_21 = False
            for line in self.config_dict[3]:
                try:
                    if line[0:1] == '#':
                        active = False
                        line = line.lstrip('#').lstrip()

                    else:
                        active = True

                    channel = re.split(';', line)
                    if len(channel) < 6:
                        # A configuration line with less then six has no chanids
                        continue

                    # Test to catch early 2.2 configurations without a chanid
                    try:
                        if self.configversion > 2.1:
                            if channel[2].strip() == '':
                                test_as_21 = True

                            else:
                                x = int(channel[2])
                                test_as_21 = True

                    except:
                        pass

                    if self.configversion == 2.1 or test_as_21:
                        # Read an old channel string
                        if len(channel) != 8 and not test_as_21:
                            # A 2.1  configuration line must contain 8 items
                            continue

                        for index in range(xml_output.source_count):
                            if channel[index + 2].strip() != '':
                                old_chanid = unicode(channel[index + 2]).strip()
                                break

                        else:
                            # No sources!
                            continue

                        chanid = u'%s-%s' % (index, old_chanid)
                        old_chanids[old_chanid] = chanid
                        self.channels[chanid] = Channel_Config(chanid, unicode(channel[0]).strip(), int(channel[1]))
                        for index in range(4):
                            self.channels[chanid].source_id[index] = unicode(channel[index + 2]).strip()

                    else:
                        # And the new version 2.2 one
                        for index in range(min(xml_output.source_count,len(channel) - 5)):
                            if channel[index + 3].strip() != '':
                                break

                        else:
                            # No sources!
                            continue

                        chanid = unicode(channel[2])
                        channame = unicode(channel[0]).strip()
                        if chanid in channel_names.keys():
                            channel_names[chanid].append(channame.lower())

                        else:
                            channel_names[chanid] = [channame.lower()]

                        self.channels[chanid] = Channel_Config(chanid, channame, int(channel[1]))
                        for index in range(min(xml_output.source_count,len(channel) - 5)):
                            self.channels[chanid].source_id[index] = unicode(channel[index + 3]).strip()

                    # The icon defenition
                    self.channels[chanid].icon_source = int(channel[-2])
                    self.channels[chanid].icon = unicode(channel[-1]).strip()
                    # Set active if not remarked out
                    self.channels[chanid].active = active
                    if active:
                        self.chan_count += 1

                except:
                    log(['Invalid line in Channels section of config file %s:\n' % (self.config_file),'%r\n' % (line), traceback.print_exc()])

            # Read the channel specific configuration
            if self.configversion == 2.1 or test_as_21:
                for k, v in old_chanids.items():
                    self.process_channel_config(v, [k])

            else:
                for chanid in self.channels.keys():
                    if chanid in channel_names.keys():
                        self.process_channel_config(chanid, channel_names[chanid])

                    else:
                        self.process_channel_config(chanid)

        # an extra option for gathering extra info to better the code
        if 'write_info_files' in self.opt_dict.keys():
            self.write_info_files = self.opt_dict['write_info_files']
            if self.write_info_files :
                infofiles.open_files()

        return True
    # end read_config()

    def process_channel_config(self, chanid, channames = None):
        if chanid in self.config_dict[9].keys():
            chan_conf = self.config_dict[9][chanid]

        elif channames != None:
            for name in channames:
                if name in self.config_dict[9].keys():
                    chan_conf = self.config_dict[9][name]
                    break

            else:
                return

        else:
            return

        for line in chan_conf:
            try:
                # Strip the name from the value
                a = re.split('=',line)
                cfg_option = a[0].lower().strip()
                # Boolean Values
                if cfg_option in self.chan_opt['bool']:
                    if len(a) == 1:
                        self.channels[chanid].opt_dict[cfg_option] = True

                    elif a[1].lower().strip() in ('true', '1' , 'on'):
                        self.channels[chanid].opt_dict[cfg_option] = True

                    else:
                        self.channels[chanid].opt_dict[cfg_option] = False

                elif len(a) == 2:
                    cfg_value = a[1].lower().strip()
                    if cfg_option == 'use_npo':
                        if cfg_value in ('false', '0' , 'off'):
                            self.validate_option('disable_source', self.channels[chanid], 4)

                    # String values
                    elif cfg_option in self.chan_opt['string']:
                        self.channels[chanid].opt_dict[cfg_option] = cfg_value.strip()

                    # Select Values
                    elif cfg_option == 'overlap_strategy':
                        if cfg_value in ('average', 'stop', 'start'):
                            self.channels[chanid].opt_dict[cfg_option] = cfg_value

                        else:
                            self.channels[chanid].opt_dict[cfg_option] = 'none'

                    # Integer Values
                    elif cfg_option in self.chan_opt['int']:
                        try:
                            cfg_value = int(cfg_value)

                        except ValueError:
                            if (cfg_option == 'slowdays') and (cfg_value == 'none'):
                                self.channels[chanid].opt_dict[cfg_option] = None

                        else:
                            self.channels[chanid].opt_dict[cfg_option] = cfg_value

                    # Source Values
                    elif cfg_option in ('prime_source', 'prefered_description', 'disable_source', 'disable_detail_source'):
                        try:
                            cfg_value = int(cfg_value)

                        except ValueError:
                            continue

                        else:
                            if cfg_option == 'prime_source':
                                # We have to validate this value after reading sourcematching.json
                                self.channels[chanid].prevalidate_opt[cfg_option] = cfg_value

                            else:
                                self.validate_option(cfg_option, self.channels[chanid], cfg_value)

            except:
                log(['Invalid line in %s section of config file %s:' % (section, self.config_file),'%r\n' % (line), traceback.format_exc()])

    # end process_channel_config()

    def read_defaults_list(self):
        """
        Get the genre conversion table, the title split exception list and others
        """
        f = self.open_file(self.settings_file)
        if f == None:
            return False

        if not self.check_encoding(f):
            return False

        f.seek(0,0)
        type = 0
        for byteline in f.readlines():
            try:
                line = self.get_line(f, byteline, False, self.encoding)
                if not line:
                    continue

                # Look for section headers
                config_title = re.search('\[(.*?)\]', line)
                if config_title != None and (config_title.group(1) in self.__DEFAULT_SECTIONS__.values()):
                    for i, v in self.__DEFAULT_SECTIONS__.items():
                        if v == config_title.group(1):
                            type = i
                            continue

                    # Ignore the defaults if section exists
                    if type == 3:
                        self.groupnameremove = []

                    elif type == 4:
                        self.titlerename = {}

                    continue

                # Unknown Section header, so ignore
                if line[0:1] == '[':
                    type = 0
                    continue

                if type == 1:
                    # split of the translation (if present) or supply an empty one
                    a = re.split('=',line)
                    if len(a) == 1:
                        a.append('')

                    # split main and sub-genre (if present) and if they have a translation
                    # (overwriting the default) or are not yet pressent add them to cattrans
                    g = re.split(':', a[0].lower() )
                    if len(g) == 2:
                        if not ((g[0].strip(), g[1].strip()) in self.cattrans and a[1].strip() == ''):
                            self.cattrans[ (g[0].strip(), g[1].strip())] = a[1].strip()

                    elif len(g) == 1:
                        if not ((g[0].strip(), g[1].strip()) in self.cattrans and a[1].strip() == ''):
                            self.cattrans[ (g[0].strip(), '')] = a[1].strip()

                elif type == 2:
                    line = line.lower()
                    if not line in self.notitlesplit:
                        self.notitlesplit.append(line)

                elif type == 3:
                    line = line.lower()
                    if not line in self.groupnameremove:
                        self.groupnameremove.append(line)

                elif type == 4:
                    # split of the translation (if present) or supply an empty one
                    a = re.split('=',line)
                    if len(a) == 1:
                        continue

                    self.titlerename[a[0].lower().strip()] = a[1].strip()

                elif type == 5:
                    # split of the translation (if present) or supply an empty one
                    a = re.split('=',line)
                    if len(a) == 1:
                        self.source_cattrans[3][a[0].lower().strip()] = (u'Overige', u'')
                        continue

                    # split main and sub-genre (if present) and add or overwrite the default
                    g = re.split(':', a[1].lower() )
                    if len(g) == 2:
                        self.source_cattrans[3][a[0].lower().strip()] = (g[0].strip(), g[1].strip())
                        continue

                    self.source_cattrans[3][a[0].lower().strip()] = (g[0].strip(), u'')

                elif type == 6:
                    # split of the translation (if present) or supply an empty one
                    a = re.split('=',line)
                    if len(a) == 1:
                        continue

                    if len(a[1]) > 20:
                        continue

                    self.source_cattrans[1][a[1].lower().strip()] = a[0].strip()

                elif type == 7:
                    # split of the translation (if present) or supply an empty one
                    a = re.split('=',line)
                    if len(a) == 1:
                        continue

                    self.roletrans[a[0].lower().strip()] = a[1].strip()
                elif type in (8, 9, 10, 11, 13, 14):
                    source = type - 4
                    # split of the translation (if present) or supply an empty one
                    a = line.split('=',1)
                    # split the source in main and sub
                    s = a[0].lower().strip()
                    if type in (8, 9):
                        s = s.split(':', 1)
                        if len(s) == 2:
                            s = (s[0].strip(), s[1].strip())

                    if len(a) == 1:
                        self.source_cattrans[source][s] = (u'Overige', u'')
                        continue

                    # split main and sub-genre (if present) and add or overwrite the default
                    g = a[1].lower().split(':', 1)
                    if len(g) == 2:
                        self.source_cattrans[source][s] = (g[0].strip(), g[1].strip())
                        continue

                    self.source_cattrans[source][s] = (g[0].strip(), u'')

            except:
                log(['Error reading Defaults\n', traceback.format_exc()])
                continue

        f.close()
        self.groupnameremove.sort(key=lambda p: len(p), reverse = True)

        return True

    #end read_defaults_list()

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in a file.
        """
        for channel in self.channels.values():
            # First we clear out all existing source_id's, because they can have become invalid!
            for index in range(xml_output.source_count):
                channel.source_id[index] = ''

            # These groupids have changed, so to be sure
            if self.opt_dict['always_use_json'] and channel.group != -1:
                channel.group = 99

            # And all not custom set icons
            if self.opt_dict['always_use_json'] and channel.icon_source != 99:
                channel.icon_source = -1
                channel.icon = ''

        db_icon = []
        db_channel = []
        db_channel_source = []
        # These channels contain no data!
        source_keys = {}
        reverse_channels = {}
        if not isinstance(self.channels, dict):
            self.channels = {}

        for chanid, icon in xml_output.logo_names.items():
            db_icon.append({'sourceid': icon[0], 'chanid': str(chanid),'icon': icon[1]})

        # Get the sources
        for index in (0, 1, 10, 6, 5, 2, 4, 7, 9, 8, 11, 12):
            xml_output.channelsource[index].init_channels()
            if xml_output.channelsource[index].get_channels() == 69:
                log("Not all channel info could be retreived.\n")
                if not index in self.opt_dict['disable_source']:
                    log("Try again in 15 minutes or so; or disable the failing source.\n")
                    return 69

            if config.write_info_files:
                infofiles.check_new_channels(xml_output.channelsource[index])

            # a dict with coresponding source, id and chanid
            reverse_channels[index] = {}
            # a list of all ids for the source
            source_keys[index] = []
            for i, v in self.source_channels[index].items():
                reverse_channels[index][v] = {}
                reverse_channels[index][v]['chanid'] = unicode(i)
                i =i.split('-',1)
                reverse_channels[index][v]['source'] = int(i[0])
                reverse_channels[index][v]['chan_scid'] = unicode(i[1])

            for chan_scid in xml_output.channelsource[index].all_channels.keys():
                if not (chan_scid in self.empty_channels[index]):
                    source_keys[index].append(chan_scid)

        for index in (0, 1, 10, 6, 5, 2, 4, 7, 9, 8, 11, 12):
            for chan_scid, channel in xml_output.channelsource[index].all_channels.items():
                if chan_scid in reverse_channels[index].keys():
                    chanid = reverse_channels[index][chan_scid]['chanid']

                else:
                    chanid = '%s-%s' % (index, chan_scid)

                chan ={}
                chan['chanid'] = chanid
                chan['sourceid'] = index
                chan['scid'] = chan_scid
                chan['cgroup'] = channel['group'] if 'group' in channel else 99
                chan['name'] = channel['name']
                if 'HD' in channel:
                    chan['hd'] = channel['HD']

                elif channel['name'][-3:].lower() == ' hd':
                    chan['hd'] = True

                else:
                    chan['hd'] = False

                # These channels are for show, but we like the icons from source 2, 6 and 5!
                if (chan_scid in self.empty_channels[index]):
                    chan['scid'] = ''
                    chan_scid = ''

                if not chanid in self.channels:
                    if (chan_scid in self.empty_channels[index]):
                        continue

                    self.channels[chanid] = Channel_Config(chanid, chan['name'] )

                self.channels[chanid].source_id[index] = chan_scid
                # Set the group
                if ((not self.opt_dict['always_use_json'] and self.channels[chanid].group >= 99)
                    or self.opt_dict['always_use_json']) and index in self.channel_grouping:
                        for g in self.channel_grouping[index]:
                            if chan_scid in self.channel_grouping[index][g]:
                                channel['group'] = g
                                break

                if self.channels[chanid].group >= 99:
                    self.channels[chanid].group = channel['group'] if 'group' in channel else 99

                # Move Dutch/Flemish channels from other to main if any sources places them there
                if 'group' in channel and channel['group'] == 1 and self.channels[chanid].group == 7:
                    self.channels[chanid].group = channel['group']

                if 'group' in channel and channel['group'] == 2 and self.channels[chanid].group == 9:
                    self.channels[chanid].group = channel['group']

                # Force Regional radio group
                if 'group' in channel and channel['group'] == 17 and self.channels[chanid].group == 11:
                    self.channels[chanid].group = channel['group']

                # Set the Icon
                icon ={}
                icon['sourceid'] = -1
                icon['chanid'] = chanid
                if 'icon' in channel and channel['icon'] != '':
                    icon['sourceid'] = index if 'icongrp' not in channel else channel['icongrp']
                    icon['icon'] = channel['icon']
                    db_icon.append(icon)

                for iconsource in xml_output.logo_source_preference:
                    if self.channels[chanid].icon_source == iconsource:
                        # A higher preference icon is already set
                        break

                    elif chanid in xml_output.logo_names.keys() and int(xml_output.logo_names[chanid][0]) == iconsource:
                        self.channels[chanid].icon_source = int(xml_output.logo_names[chanid][0])
                        self.channels[chanid].icon = xml_output.logo_names[chanid][1]
                        break

                    elif icon['sourceid'] == iconsource:
                        self.channels[chanid].icon_source = icon['sourceid']
                        self.channels[chanid].icon = icon['icon']
                        break

                db_channel_source.append(chan)

        for channel in self.channels.values():
            # Some channel title renaming
            if self.opt_dict['always_use_json'] and channel.chanid in self.channel_rename.keys():
                channel.chan_name = self.channel_rename[channel.chanid]

            # mark HD channels
            if channel.chan_name[-3:].lower() == ' hd':
                channel.opt_dict['mark_hd'] = True

            if channel.get_source_id(5) != '' and channel.get_source_id(5) in xml_output.channelsource[5].all_channels \
              and xml_output.channelsource[5].all_channels[channel.get_source_id(5)]['HD']:
                channel.opt_dict['mark_hd'] = True

            # set the default prime_source
            self.validate_option('prime_source', channel, -1)
            if channel.get_source_id(0) in ('3',):
                channel.opt_dict['append_tvgidstv'] = False

            db_channel.append({'name': channel.chan_name,
                                               'chanid': channel.chanid,
                                               'cgroup': channel.group})

        xml_output.program_cache.cache_request.put({'task':'add',
                                               'channel': db_channel,
                                               'channelsource': db_channel_source,
                                               'icon': db_icon})
        return 0

    # end get_channels()

    def get_page(self, url, encoding = "default encoding", accept_header = None):
        """
        Wrapper around get_page_internal to catch the
        timeout exception
        """
        try:
            fu = FetchURL(url, encoding, accept_header)
            self.max_fetches.acquire()
            fu.start()
            fu.join(self.opt_dict['global_timeout'])
            page = fu.result
            config.max_fetches.release()
            if (page == None) or (page.replace('\n','') == '') or (page.replace('\n','') =='{}'):
                with xml_output.output_lock:
                    xml_output.fail_count += 1

                return None

            else:
                return page

        except(urllib.URLError, socket.timeout):
            log('get_page timed out on (>%s s): %s\n' % (self.opt_dict['global_timeout'], url), 1, 1)
            if self.write_info_files:
                infofiles.add_url_failure('Fetch timeout: %s\n' % url)

            with xml_output.output_lock:
                xml_output.fail_count += 1

            config.max_fetches.release()
            return None
    # end get_page()

    def get_sourcematching_file(self, configuring = False, show_info = True):
        # This gets grabed after reading the config
        def get_githubdict(gvar, intlevels = 0):
            lvar = {}
            try:
                if not gvar in githubdata:
                    return lvar.copy()

                if intlevels == 0:
                    return githubdata[gvar]

                for s, v in githubdata[gvar].items():
                    if re.match('--.*?--', s):
                        continue

                    if intlevels == 1:
                        lvar[int(s)] = v

                    elif intlevels == 2:
                        lvar[int(s)] = {}
                        for g, clist in v.items():
                            if re.match('--.*?--', g):
                                continue

                            lvar[int(s)][int(g)] = clist

                return lvar.copy()

            except:
                log('Error loading %s data from sourcematching!' % gvar)
                return lvar.copy()

        def get_githubdata(gvar, default = []):
            try:
                if not gvar in githubdata:
                    if isinstance(default,(list, tuple)):
                        return list(default)[:]

                    else:
                        return default

                if isinstance(githubdata[gvar],list):
                    lvar = default[:]
                    for t in githubdata[gvar]:
                        if not t in default:
                            lvar.append(t)

                    return lvar[:]

            except:
                return default

        def get_time(tvar, default = 0):
            try:
                st = tvar.split(':')
                self.time = datetime.time(int(st[0]), int(st[1]), tzinfo=CET_CEST)
                return True

            except:
                if default == 0:
                    self.time = datetime.time(0, 0, tzinfo=CET_CEST)

                elif default == 24:
                    self.time = datetime.time(23, 59, 59, 999999, tzinfo=CET_CEST)

                return False

        try:
            url = 'https://raw.githubusercontent.com/tvgrabbers/sourcematching/master/sourcematching.json'
            githubdata = json.loads(self.get_page(url, 'utf-8'))
            # Check on data or program updates
            dv = int(githubdata["data_version"])
            nsv = githubdata["program_version"].split('.')
            nv = []
            for i in range(len(nsv)):
                nv.append(int(nsv[i]))

            pv = [self.major, self.minor, self.patch]
            if not "data_version" in self.opt_dict:
                self.opt_dict["data_version"] = max(0, dv - 2)

            if pv < nv or (pv == nv and (self.alfa or self.beta)):
                loglist = ['There is a newer stable release available on github!\n']
                if "version_message" in githubdata:
                    if isinstance(githubdata["version_message"], (str, unicode)):
                        loglist.append(githubdata["version_message"])

                    elif isinstance(githubdata["version_message"], list):
                        loglist.extend(githubdata["version_message"])

                loglist.append("Goto: https://github.com/tvgrabbers/tvgrabnlpy/releases/latest\n")
                if show_info:
                    log(loglist, 0)

            elif dv > self.opt_dict["data_version"]:
                loglist = ['The channel/source matching data on github is newer!\n']
                if "warning_message_2" in githubdata:
                    for v, tekst in githubdata["warning_message_2"].items():
                        if int(v) > self.opt_dict["data_version"]:
                            if isinstance(tekst, (str, unicode)):
                                loglist.append(tekst)

                            elif isinstance(tekst, list):
                                loglist.extend(tekst)

                if not configuring:
                    loglist.append("Run with '--configure' to implement it\n")

                if show_info:
                    log(loglist, 0)

        except:
            githubdata = {}
            log(['Error reading the datafile on github.\n', traceback.print_exc()], 0)
            if configuring:
                log(['Unable to continue with configure!\n'], 0)
                return 2

        # Check on disabled sources
        active_sources = get_githubdata("active_sources")
        for c in xml_output.channelsource.keys():
            if c not in active_sources:
                self.validate_option('disable_source', value = c)

        # Remove any source that's not (jet) there
        xml_output.source_order = active_sources[:]
        for s in active_sources:
            if not s in xml_output.channelsource.keys():
                xml_output.source_order.remove(s)

        # Remove any source that's not (jet) there
        xml_output.prime_source_order = get_githubdata("prime_source_order")
        for s in xml_output.prime_source_order[:]:
            if not s in xml_output.source_order:
                xml_output.prime_source_order.remove(s)

        for s in xml_output.sourceid_order[:]:
            if not s in xml_output.source_order:
                xml_output.sourceid_order.remove(s)

        # Read in the tables needed for normal grabbing
        self.source_base_url = get_githubdict("source_base_url", 1)
        self.source_regexes = get_githubdict("source_regexes", 1)
        self.text_replace = get_githubdict("text_replace", 1)
        self.unquote_html = get_githubdict("unquote_html", 1)
        xml_output.logo_provider = get_githubdata("logo_provider")
        virtual_sub_channels = get_githubdict("virtual-sub-channels")
        self.virtual_sub_channels = {}
        for chanid, chandef in virtual_sub_channels.items():
            self.virtual_sub_channels[chanid] = {}
            if 'start' in chandef:
                if not get_time(chandef['start'], 0):
                    log('Invalid starttime for %s\n' % (chanid))

                self.virtual_sub_channels[chanid]['start'] = self.time

            else:
                self.virtual_sub_channels[chanid]['start'] = datetime.time(0, 0, tzinfo=CET_CEST)

            if 'end' in chandef:
                if not get_time(chandef['end'], 24):
                    log('Invalid endtime for %s\n' % (chanid))

                self.virtual_sub_channels[chanid]['end'] = self.time

            else:
                self.virtual_sub_channels[chanid]['end'] = datetime.time(23, 59, 59, 999999, tzinfo=CET_CEST)

            self.virtual_sub_channels[chanid]['channelids'] = {}
            for s in xml_output.source_order:
                if unicode(s) in chandef.keys():
                    self.virtual_sub_channels[chanid]['channelids'][s] = chandef[unicode(s)]

                else:
                    self.virtual_sub_channels[chanid]['channelids'][s] = ''

        combined_channels = get_githubdict("combined_channels_2")
        self.combined_channels = {}
        for chanid, chanlist in combined_channels.items():
            clist = []
            for child in chanlist:
                if isinstance(child, dict) and 'chanid' in child:
                    if "virtual-chanid" in child and child["virtual-chanid"] in self.virtual_sub_channels.keys():
                        child['chanid'] = child['virtual-chanid']

                    if 'start' in child:
                        if not get_time(child['start'], 0):
                            log('Invalid starttime for %s in combined channel: %s\n' % (child['chanid'], chanid))

                        child['start'] = self.time

                    if 'end' in child:
                        if not get_time(child['end'], 24):
                            log('Invalid endtime for %s in combined channel: %s\n' % (child['chanid'], chanid))

                        child['end'] = self.time

                    if 'start' in child and not 'end' in child:
                        child['end'] = datetime.time(23, 59, 59, 999999, tzinfo=CET_CEST)

                    elif 'end' in child and not 'start' in child:
                        child['start'] = datetime.time(0, 0, tzinfo=CET_CEST)

                    clist.append(child)

                elif isinstance(child, (str, unicode)):
                    clist.append({'chanid': child})

            self.combined_channels[chanid] = clist

        self.groupslot_names = get_githubdata("groupslot_names")
        self.ttvdb_aliasses = get_githubdict("ttvdb_aliasses")
        self.generic_channel_genres = get_githubdict("generic_channel_genres")
        self.coutrytrans = get_githubdict("coutrytrans")
        self.prime_source = get_githubdict("prime_source")
        # Remove any source that's not (jet) there
        for c, s in self.prime_source.items()[:]:
            if s not in xml_output.source_order:
                del self.prime_source[c]

        self.notitlesplit = get_githubdata("notitlesplit", self.notitlesplit)
        self.user_agents = get_githubdata("user_agents", self.user_agents)
        self.no_genric_matching = get_githubdict("no_genric_matching", 1)

        # Read the tables only needed during configuring
        xml_output.logo_source_preference = get_githubdata("logo_source_preference")
        xml_output.logo_names = get_githubdict("logo_names")
        for icon in xml_output.logo_names.values():
            if icon[1][-4:] not in ('.png', '.jpg', '.gif'):
                if icon[0] in ('0', '1'):
                    icon[1] = icon[1] + '.gif'

                elif icon[0] in ('2', '6'):
                    icon[1] = icon[1] + '.jpg'

                elif icon[0] in ('3', '4', '5', '7', '8', '10', '11'):
                    icon[1] = icon[1] + '.png'

        self.chan_groups = get_githubdict("channel_groups", 1)
        self.group_order = get_githubdata("group_order")
        for g in self.chan_groups.keys():
            if g not in self.group_order[:]:
                self.group_order.append(g)

        for g in self.group_order:
            if g not in self.chan_groups.keys():
                self.chan_groups[g] = 'Channel groep %s' % g

        self.prime_source_groups = get_githubdict("prime_source_groups", 1)
        # Remove any source that's not (jet) there
        for g, s in self.prime_source_groups.items():
            if s not in xml_output.channelsource.keys():
                del self.prime_source_groups[g]

        self.source_channels = get_githubdict("source_channels", 1)
        self.empty_channels = get_githubdict("empty_channels", 1)
        self.channel_grouping = get_githubdict("channel_grouping", 2)
        self.rtl_channellist = get_githubdict("rtl_channellist")
        self.virtual_channellist = get_githubdict("virtual_channellist")
        self.channel_rename = get_githubdict("channel_rename")
        self.merge_into = get_githubdict("merge_into")
        if configuring:
            self.opt_dict["data_version"] = dv

        logarray = []
        for newch, oldch  in self.merge_into.items():
            newpresent = bool(newch in self.channels)
            newactive = newpresent and self.channels[newch].active
            oldpresent = bool(oldch['chanid'] in self.channels)
            oldactive = oldpresent and self.channels[oldch['chanid']].active
            if configuring:
                if oldpresent:
                    logarray.append('We merged %s into %s\n' % (oldch['chanid'], newch))
                    logarray.extend(oldch['message'])
                    # Initiate an alias if the old chanid is active
                    if oldactive and newactive:
                        logarray.append('Since both were active, we have not set an alias\n')
                        logarray.append('If you want to use the old chanid %s as xmltvid\n' % oldch['chanid'])
                        logarray.append('you have to add:\n')
                        logarray.append('  xmltvid_alias = %s\n' % oldch['chanid'])
                        logarray.append('to the channel configuration for %s\n' % newch)

                    elif oldactive and self.channels[newch].opt_dict['xmltvid_alias'] == None:
                        logarray.append('Since the old chanid was active, we have set an alias\n')
                        logarray.append('  xmltvid_alias = %s\n' % oldch['chanid'])
                        logarray.append('to the channel configuration for %s\n' % newch)
                        self.channels[newch].opt_dict['xmltvid_alias'] = oldch['chanid']
                        self.channels[newch].active = True

                    elif oldactive:
                        logarray.append('Since %s already has an xmltvid_alias set\n' % newch)
                        logarray.append('we have not changed this.\n')
                        logarray.append('If you want to use the old chanid %s as xmltvid\n' % oldch['chanid'])
                        logarray.append('you have to change:\n')
                        logarray.append('  xmltvid_alias = %s\n' % self.channels[newch].opt_dict['xmltvid_alias'])
                        logarray.append('to:')
                        logarray.append('  xmltvid_alias = %s\n' % oldch['chanid'])
                        logarray.append('in the channel configuration for %s\n' % newch)
                        self.channels[newch].active = True

                    self.channels[oldch['chanid']].active = False
                    logarray.append('We could not check for any selfset options on the old chanid: %s\n' % oldch['chanid'])
                    logarray.append('So check the settings for the new chanid: %s\n' % newch)
                    logarray.append('\n')

                for source, id in oldch['sources'].items():
                    # Link the ids from the old chanid to the new chanid
                    if oldch['chanid'] in self.source_channels[int(source)]:
                        self.source_channels[int(source)][newch] = self.source_channels[int(source)][oldch['chanid']]
                        del self.source_channels[int(source)][oldch['chanid']]

            else:
                if newpresent:
                    # Remove the old one from combined_channels if in there
                    if newch in self.combined_channels.keys():
                        for c in self.combined_channels[newch]:
                            if c['chanid'] == oldch['chanid']:
                                self.combined_channels[newch].remove(c)
                                break

                    # And link the ids to the new chanid
                    for source, id in oldch['sources'].items():
                        self.channels[newch].source_id[int(source)] = id

            log(logarray, 0)

    # get_sourcematching_file()

    def send_mail(self, message, mail_address, subject=None):
        try:
            if isinstance(message, (list,tuple)):
                msg = u''.join(message)

            elif isinstance(message, (str,unicode)):
                msg = unicode(message)

            else:
                return

            if subject == None:
                subject = 'Tv_grab_nl_py %s' % datetime.datetime.now().strftime('%Y-%m-%d %H:%M')

            msg = MIMEText(msg, _charset='utf-8')
            msg['Subject'] = subject
            msg['From'] = mail_address
            msg['To'] = mail_address
            try:
                mail = smtplib.SMTP(self.opt_dict['mailserver'], self.opt_dict['mailport'])

            except:
                sys.stderr.write(('Error mailing message: %s\n' % sys.exc_info()[1]).encode(logging.local_encoding, 'replace'))
                return

            mail.sendmail(mail_address, mail_address, msg.as_string())

        except:
            sys.stderr.write('Error mailing message\n'.encode(logging.local_encoding, 'replace'))
            sys.stderr.write(traceback.format_exc())

        mail.quit()

    # send_mail()

    def validate_commandline(self):
        """Read the commandline and validate the values"""
        if self.read_commandline() == 0:
             return(0)

        # The Query options
        for (a, o) in ((self.args.version, 'version'), \
                              (self.args.description, 'description'), \
                              (self.args.description_long, 'description_long'), \
                              (self.args.capabilities, 'capabilities'), \
                              (self.args.preferredmethod, 'preferredmethod'), \
                              (self.args.show_sources, 'show_sources'), \
                              (self.args.show_logo_sources, 'show_logo_sources'), \
                              (self.args.show_detail_sources, 'show_detail_sources')):
            if a:
                self.validate_option(o)
                return(0)

        # Check config and log file
        if self.args.config_file != self.config_file:
            # use the provided name for configuration and logging
            self.config_file = self.args.config_file
            self.log_file = self.args.config_file+'.log'

        x = self.validate_option('config_file')
        if x != None:
            return(x)

        self.max_fetches = Semaphore(self.opt_dict['max_simultaneous_fetches'])
        x = self.get_sourcematching_file(self.args.configure)
        if x != None:
            return(x)

        if self.args.quiet != None:
            self.opt_dict['quiet'] = self.args.quiet

        logging.quiet = self.opt_dict['quiet']
        logging.log_level = self.opt_dict['log_level']

        #check for cache
        if self.validate_option('program_cache_file') != None:
            return(2)

        if self.args.clear_cache or self.args.clear_ttvdb:
            return(0)

        if self.args.disable_ttvdb != None:
            self.opt_dict['disable_ttvdb'] = self.args.disable_ttvdb

        if self.opt_dict['disable_ttvdb'] == False:
            xml_output.ttvdb.start()

        if self.args.ttvdb_title != None :
            self.validate_option('check_ttvdb_title')
            return(0)

        # Check a possible output file
        if self.args.output_file != None:
            self.opt_dict['output_file'] = self.args.output_file

        if self.validate_option('output_file') != None:
            return(2)

        # Validate the options
        for s in self.args.disable_source:
            self.validate_option('disable_source', value = s)

        if self.args.use_npo != None:
            self.validate_option('disable_source', value = 4)

        for s in self.args.disable_detail_source:
            self.validate_option('disable_detail_source', value = s)

        for (a, o) in ((self.args.compat, 'compat'), \
                              (self.args.legacy_xmltvids, 'legacy_xmltvids'), \
                              (self.args.fast, 'fast'), \
                              (self.args.logos, 'logos'), \
                              (self.args.mark_hd, 'mark_hd'), \
                              (self.args.cattrans, 'cattrans'), \
                              (self.args.slowdays, 'slowdays'), \
                              (self.args.desc_length, 'desc_length'), \
                              (self.args.overlap_strategy, 'overlap_strategy'), \
                              (self.args.max_overlap, 'max_overlap')):
            if a != None:
                self.opt_dict[o] = a

        self.offset = self.opt_dict['offset']
        for (a, o) in ((self.args.use_utc, 'use_utc'), \
                              (self.args.offset, 'offset'), \
                              (self.args.days, 'days')):
            if a != None:
                self.opt_dict[o] = a

        # limit days to maximum supported by the several sites
        for o in ('offset', 'days'):
            self.validate_option(o)

        # Continue validating the settings for the individual channels
        self.validate_option('channel_settings')
        if not self.args.configure and self.configversion < float('%s.%s' % (self.major, self.minor)):
            # Update to the current version config
            if self.configversion == 1.0:
                self.write_defaults_list()
            if not self.write_config(None):
                log(['Error updating to new Config.\n', 'Please remove the old config and Re-run me with the --configure flag.\n'])
                return(1)

            log(['Updated the configfile %s!\n' % self.config_file, \
                'Check if you are fine with the settings.\n', \
                'If this is a first install, you have to enable the desired channels!\n'], 1, 1)
            return(0)

        self.write_opts_to_log()
        if self.args.configure:
            self.args.group_active_channels = self.opt_dict['group_active_channels'] | self.args.group_active_channels
            log('Creating config file: %s\n' % self.config_file)
            if self.get_channels() == 69:
                return 69

            self.validate_option('channel_settings')
            if not self.write_config(True):
                log('Error writing new Config. Trying to restore an old one.\n')
                try:
                    if os.path.exists(self.config_file + '.old'):
                        os.rename(self.config_file + '.old', self.config_file)

                except:
                    pass

                return(1)

            log(['Created the configfile %s!\n' % self.config_file, \
                'Check if you are fine with the settings.\n', \
                'If this is a first install, you have to enable the desired channels!\n'], 1, 1)
            return(0)

        if self.args.save_options:
            if not self.write_config(False):
                log('Error writing new Config. Trying to restore an old one.\n')
                try:
                    if os.path.exists(self.config_file + '.old'):
                        os.rename(self.config_file + '.old', self.config_file)

                except:
                    pass

                return(1)

            log(['Updated the options in the configfile %s!\n' % self.config_file, 'Check if you are fine with the settings.\n'])
            return(0)

        self.read_defaults_list()

    # end validate_commandline()

    def validate_option(self, option, channel = None, value = None, stdoutput = True, check_default = False):
        """Validate an option"""
        if not (channel == None or channel in self.channels.values()):
            return

        if option == 'version':
            print("The Netherlands: %s" % self.version(True))
            return(0)

        elif option == 'description':
            if stdoutput:
                v=self.version()
                if v[5]:
                    print("Dutch/Flemish grabber combining multiple sources. v%s.%s.%s-beta" % (v[1], v[2], v[3]))

                else:
                    print("Dutch/Flemish grabber combining multiple sources. v%s.%s.%s" % (v[1], v[2], v[3]))

            else:
                v=self.version()
                if v[5]:
                    return("Dutch/Flemish grabber combining multiple sources. v%s.%s.%s-beta" % (v[1], v[2], v[3]))

                else:
                    return("Dutch/Flemish grabber combining multiple sources. v%s.%s.%s" % (v[1], v[2], v[3]))

        elif option == 'description_long':
            print("The Netherlands: %s" % self.version(True))
            print(description_text)
            return(0)

        elif option == 'capabilities':
            if stdoutput:
                print("baseline")
                print("cache")
                print("manualconfig")
                print("preferredmethod")

            else:
                return ("baseline", "cache", "manualconfig", "preferredmethod")

        elif option == 'preferredmethod':
            if stdoutput:
                 print('allatonce')

            else:
                 return('allatonce')

        elif option == 'show_sources':
            if stdoutput:
                print('The available sources are:')
                for i, s in xml_output.channelsource.items():
                    print('  %s: %s' % (i, s.source))

            else:
                tdict = {}
                for i, s in xml_output.channelsource.items():
                    tdict[i] = s.source

                return tdict

        elif option == 'show_detail_sources':
            if stdoutput:
                print('The available detail sources are:')
                for i, s in xml_output.channelsource.items():
                    if i in xml_output.detail_sources:
                        print('  %s: %s' % (i, s.source))

            else:
                tdict = {}
                for i, s in xml_output.channelsource.items():
                    if s.detail_processor:
                        tdict[i] = s.source

                return tdict
        elif option == 'show_logo_sources':
            self.get_sourcematching_file(show_info=False)
            if stdoutput:
                print('The available logo sources are:')
                for i in range(len(xml_output.logo_provider)):
                    print('  %s: %s' % (i, xml_output.logo_provider[i]))

                print(' 99: Your own full logo url')

            else:
                tdict = {}
                for i in range(len(xml_output.logo_provider)):
                    tdict[i] = xml_output.logo_provider[i]

                    tdict[99] = 'free-form'
                return tdict

        elif option == 'check_ttvdb_title':
            if self.opt_dict['disable_ttvdb']:
                log('Sorry, thetvdb.com lookup is disabled!\n')
                return

            if len(self.args.ttvdb_title) == 0:
                log('Please supply a series title!\n')
                return

            series_title = unicode(self.args.ttvdb_title[0], 'utf-8')
            lang = 'nl'
            if len(self.args.ttvdb_title) >1:
                lang = unicode(self.args.ttvdb_title[1], 'utf-8')[:2]
                if not lang in ('cs', 'da', 'de', 'el', 'en', 'es', 'fi', 'fr', 'he', 'hr', 'hu', 'it',
                                        'ja', 'ko', 'nl', 'no', 'pl', 'pt', 'ru', 'sl', 'sv', 'tr', 'zh'):
                    log("Invalid language code: '%s' supplied falling back to 'nl'\n" % lang)

            xml_output.ttvdb.check_ttvdb_title(series_title, lang)

        elif option == 'disable_source':
            if value in xml_output.channelsource.keys():
                if channel == None:
                    if value not in self.opt_dict['disable_source']:
                        self.opt_dict['disable_source'].append(value)

                elif channel.get_source_id(value) != '':
                    if value not in channel.opt_dict['disable_source']:
                        channel.opt_dict['disable_source'].append(value)

                    if value in xml_output.detail_sources and value not in channel.opt_dict['disable_detail_source']:
                        channel.opt_dict['disable_detail_source'].append(value)

        elif option == 'disable_detail_source':
            if value in xml_output.detail_sources:
                if channel == None:
                    if value not in self.opt_dict['disable_detail_source']:
                        self.opt_dict['disable_detail_source'].append(value)

                elif channel.get_source_id(value) != '':
                    if value not in channel.opt_dict['disable_detail_source']:
                        channel.opt_dict['disable_detail_source'].append(value)

        elif option == 'channel_settings':
            for chanid, chanlist in self.combined_channels.items():
                if chanid in self.channels.keys() and self.channels[chanid].active:
                    for child in chanlist:
                        childid = child['chanid']
                        if childid in self.virtual_sub_channels.keys():
                            if not childid in self.channels.keys():
                                # We start a channel thread
                                self.channels[childid] = Channel_Config(childid, unicode(childid), -1)

                            for s, channelid in self.virtual_sub_channels[childid]['channelids'].items():
                                self.channels[childid].source_id[s] = channelid

                            self.process_channel_config(childid)
                            self.channels[childid].is_child = True
                            self.channels[childid].is_virtual_sub = True
                            self.channels[childid].virtual_start = self.virtual_sub_channels[childid]['start']
                            self.channels[childid].virtual_end = self.virtual_sub_channels[childid]['end']

            for channel in self.channels.values():
                channel.validate_settings()

        elif option == 'prime_source':
            if channel == None:
                return

            # First get the default
            def_value = -1
            if channel.group in self.prime_source_groups and channel.get_source_id(self.prime_source_groups[channel.group]) != '' \
                and not (self.prime_source_groups[channel.group] in self.opt_dict['disable_source'] \
                or self.prime_source_groups[channel.group] in channel.opt_dict['disable_source'] \
                or channel.get_source_id(self.prime_source_groups[channel.group]) in config.no_genric_matching[self.prime_source_groups[channel.group]]):
                    # A group default in sourcematching.json
                    def_value = self.prime_source_groups[channel.group]

            else:
                for s in xml_output.prime_source_order:
                    if channel.get_source_id(s) != '' \
                        and not (s in self.opt_dict['disable_source'] or s in channel.opt_dict['disable_source'] \
                        or channel.get_source_id(s) in config.no_genric_matching[s]):
                            # The first available not set in no_genric_matching
                            def_value = s
                            break

                else:
                    for s in xml_output.prime_source_order:
                        if channel.get_source_id(s) != '' \
                            and not (s in self.opt_dict['disable_source'] or s in channel.opt_dict['disable_source']):
                                # The first available
                                def_value = s
                                break

            # Now we check for a custom value
            if value == None:
                value = channel.opt_dict['prime_source']

            custom_value = None
            if value in xml_output.channelsource.keys() and channel.get_source_id(value) != '' \
                and not (value in self.opt_dict['disable_source'] or value in channel.opt_dict['disable_source']) \
                and value != def_value:
                    # It's a valid custom value not equal to the default
                    custom_value = value

            json_value = None
            if channel.chanid in self.prime_source.keys() and channel.get_source_id(self.prime_source[channel.chanid]) != '' \
                and not (self.prime_source[channel.chanid] in self.opt_dict['disable_source'] \
                or self.prime_source[channel.chanid] in channel.opt_dict['disable_source']):
                    # Use an override in sourcematching.json
                    json_value = self.prime_source[channel.chanid]

            if self.opt_dict['always_use_json']:
                for v in (json_value, custom_value):
                    if v != None and not channel.get_source_id(v) in config.no_genric_matching[v]:
                        value = v
                        break

                else:
                    value = def_value

            else:
                for v in (custom_value, json_value):
                    if v != None and not channel.get_source_id(v) in config.no_genric_matching[v]:
                        value = v
                        break

                else:
                    value = def_value

            if check_default:
                return bool((value == def_value) or (value == -1))

            else:
                channel.opt_dict['prime_source'] = value

        elif option == 'prefered_description':
            if channel == None:
                return

            if value == None:
                value = channel.opt_dict['prefered_description']

            if value in xml_output.channelsource.keys() and channel.get_source_id(value) != '' \
                and not (value in self.opt_dict['disable_source'] or value in channel.opt_dict['disable_source']):
                    channel.opt_dict['prefered_description'] = value

            else:
                channel.opt_dict['prefered_description'] = -1

        elif option == 'offset':
            if self.opt_dict['offset'] > 14:
                if self.offset < 14:
                    log("Een zo hoge offset van: %s is belachelijk. We resetten naar %s\n" % (self.opt_dict['offset'], self.offset),1,1)
                    self.opt_dict['offset'] = self.offset

                else:
                    log("Een zo hoge offset van: %s is belachelijk. We resetten naar 0\n" % (self.opt_dict['offset']),1,1)
                    self.opt_dict['offset'] = 0

        elif option == 'days':
            if self.opt_dict['days'] > (14 - self.opt_dict['offset']):
                log("tvgids.nl/tvgids.tv kunnen maximaal 14 dagen vooruit kijken. Resetting\n",1,1)

            self.opt_dict['days'] = min(self.opt_dict['days'],(14 - self.opt_dict['offset']))

            #~ if self.opt_dict['slowdays'] != None and self.opt_dict['slowdays'] > config.opt_dict['days']:
                #~ self.opt_dict['slowdays'] = config.opt_dict['days']

        elif option == 'output_file':
            if self.opt_dict['output_file'] != None:
                try:
                    output_dir = os.path.dirname(self.opt_dict['output_file'])
                    if (output_dir != '') and not os.path.exists(output_dir):
                        log('Creating %s directory,\n' % output_dir)
                        os.mkdir(output_dir)

                    if self.args.output_codeset:
                        xml_output.xmlencoding = 'CP1252'
                        self.output = self.open_file(self.opt_dict['output_file'],'w', 'windows-1252')

                    else:
                        self.output = self.open_file(self.opt_dict['output_file'],'w')

                    if self.output == None:
                        log('Cannot write to outputfile: %s\n' % self.opt_dict['output_file'])
                        return(2)

                except:
                    log(['Cannot write to outputfile: %s\n' % self.opt_dict['output_file'], traceback.format_exc()])
                    return(2)

            else: self.output = None

        elif option == 'config_file':
            # Save an old session log and open a new one
            try:
                # check for the config/log dir
                config_dir = os.path.dirname(self.config_file)
                if (config_dir != '') and not os.path.exists(config_dir):
                    log('Creating %s directory,\n' % config_dir)
                    os.mkdir(config_dir)

                else:
                    self.save_oldfile(self.log_file)

            except:
                logging.writelog('Cannot access the config/log directory: %s\n' % config_dir, 0,1)
                logging.writelog(traceback.format_exc(), 0,1)
                return(2)

            try:
                self.log_output = self.open_file(self.log_file, mode = 'a')
                if self.log_output == None:
                    logging.writelog('Cannot open the logfile: %s\n' % self.log_file, 0,1)
                    return(2)

                logging.start()

            except:
                logging.writelog('Cannot open the logfile: %s\n' % self.log_file, 0,1)
                logging.writelog(traceback.format_exc(), 0,1)
                return(2)

            log('Using config file: %s\n' % self.config_file)
            # get config if available Overrule if set by commandline
            if not self.read_config() and not self.args.configure:
                return(1)

        elif option == 'program_cache_file':
            if self.program_cache_file.lower() == 'none' or self.program_cache_file == None:
                self.program_cache_file = None
                xml_output.program_cache = ProgramCache(self.program_cache_file)
                xml_output.program_cache.start()
                return

            try:
                cache_dir = os.path.dirname(self.program_cache_file)
                if (cache_dir != '') and not os.path.exists(cache_dir):
                    log('Creating %s directory,\n' % cache_dir)
                    os.mkdir(cache_dir)

                if os.access(self.program_cache_file, os.F_OK and os.W_OK):
                    pass

                elif not os.path.isfile(self.program_cache_file) and os.access(cache_dir, os.W_OK):
                    pass

                else:
                    log('Cannot write to cachefile: %s\n' % self.program_cache_file)
                    return(2)

            except:
                log(['Error accessing cachefile(directory): %s\n' % self.program_cache_file, traceback.format_exc()])
                return(2)

            xml_output.program_cache = ProgramCache(self.program_cache_file)
            xml_output.program_cache.start()
            if self.args.clear_cache:
                xml_output.program_cache.cache_request.put({'task':'clear'})

            elif self.args.clean_cache:
                xml_output.program_cache.cache_request.put({'task':'clean'})

            if self.args.clear_ttvdb:
                xml_output.program_cache.cache_request.put({'task':'clear', 'table':['ttvdb', 'episodes']})

        elif option == 'slowdays':
            if channel == None:
                if self.opt_dict['slowdays'] != None:
                    self.opt_dict['slowdays'] = min(self.opt_dict['days'], self.opt_dict['slowdays'])
                    # slowdays implies fast == False
                    if self.opt_dict['slowdays'] < self.opt_dict['days']:
                        self.opt_dict['fast']  = False

            elif 'slowdays' in channel.opt_dict.keys():
                if channel.get_opt('slowdays') != None:
                    channel.opt_dict['slowdays'] = min(self.opt_dict['days'], channel.get_opt('slowdays'))
                    # slowdays implies fast == False
                    if channel.get_opt('slowdays') < self.opt_dict['days']:
                        channel.opt_dict['fast']  = False

        elif option == 'desc_length':
            if channel != None and channel.get_opt('desc_length') != self.opt_dict['desc_length']:
                log('Using description length: %d for Cannel: %s\n' % (channel.opt_dict['desc_length'], channel.chan_name),1,1)

        elif option == 'overlap_strategy':
            if channel == None:
                if not self.opt_dict['overlap_strategy'] in ['average', 'stop', 'start']:
                    self.opt_dict['overlap_strategy'] = 'none'

            elif not channel.get_opt('overlap_strategy') in ['average', 'stop', 'start']:
                channel.opt_dict['overlap_strategy'] = 'none'

        elif option == 'max_overlap':
            if channel == None:
                if self.opt_dict['max_overlap'] == 0:
                    # no max_overlap implies strategie == 'None'
                    self.opt_dict['overlap_strategy'] = 'None'
                    log('Maximum overlap 0 means overlap strategy set to: "%s"\n' % (self.opt_dict['overlap_strategy']),1,1)

            elif channel.get_opt('max_overlap') == 0:
                # no max_overlap implies strategie == 'None'
                channel.opt_dict['overlap_strategy'] = 'None'
                log('Maximum overlap 0 means overlap strategy for Channel: %s set to: "%s"\n' % (channel.chan_name, channel.get_opt('overlap_strategy')),1,1)

            elif channel.get_opt('max_overlap') != self.opt_dict['max_overlap']:
                log('Using Maximum Overlap: %d for Channel %s\n' % (channel.get_opt('max_overlap'), channel.chan_name),1,1)
                if channel.get_opt('overlap_strategy') != self.opt_dict['overlap_strategy']:
                    log('overlap strategy for Channel: %s set to: "%s"\n' % (channel.chan_name, channel.get_opt('overlap_strategy')),1,1)

    # end validate_option()

    def write_opts_to_log(self):
        """
        Save the the used options to the logfile
        """
        if self.log_output == None:
            return(0)

        log_array = [u'Python versie: %s.%s.%s' % (sys.version_info[0], sys.version_info[1], sys.version_info[2])]
        log_array.append(u'The Netherlands: %s' % self.version(True))
        log_array.append(u'Capabilities:"baseline" ,"cache" ,"manualconfig" ,"preferredmethod")')
        log_array.append(u'Preferred Methode: "allatonce"')
        log_array.append(u'log level = %s' % (self.opt_dict['log_level']))
        log_array.append(u'match log level = %s' % (self.opt_dict['match_log_level']))
        log_array.append(u'global_timeout = %s' % (self.opt_dict['global_timeout']))
        log_array.append(u'max_simultaneous_fetches = %s' % (self.opt_dict['max_simultaneous_fetches']))
        log_array.append(u'config_file = %s' % (self.config_file))
        log_array.append(u'program_cache_file = %s.db' % (self.args.program_cache_file))
        log_array.append(u'clean_cache = %s' % (self.args.clean_cache))
        log_array.append(u'disable_ttvdb = %s' % (self.opt_dict['disable_ttvdb']))
        log_array.append(u'quiet = %s' % (self.opt_dict['quiet']))
        log_array.append(u'output_file = %s' % (self.opt_dict['output_file']))
        log_array.append(u'fast = %s' % (self.opt_dict['fast']))
        log_array.append(u'offset = %s' % (self.opt_dict['offset']))
        log_array.append(u'days = %s' % (self.opt_dict['days']))
        log_array.append(u'slowdays = %s' % (self.opt_dict['slowdays']))
        log_array.append(u'compat = %s' % (self.opt_dict['compat']))
        log_array.append(u'legacy_xmltvids = %s' % (self.opt_dict['legacy_xmltvids']))
        log_array.append(u'max_overlap = %s' % (self.opt_dict['max_overlap']))
        log_array.append(u'overlap_strategy = %s' % (self.opt_dict['overlap_strategy']))
        log_array.append(u'logos = %s' % (self.opt_dict['logos']))
        log_array.append(u'desc_length = %s' % (self.opt_dict['desc_length']))
        log_array.append(u'cattrans = %s' % (self.opt_dict['cattrans']))
        log_array.append(u'use_split_episodes = %s' % (self.opt_dict['use_split_episodes']))
        log_array.append(u'kijkwijzerstijl = %s' % (self.opt_dict['kijkwijzerstijl']))
        log_array.append(u'mark_hd = %s' % (self.opt_dict['mark_hd']))
        log_array.append(u'use_utc = %s' % (self.opt_dict['use_utc']))
        for index in xml_output.source_order:
            if index in self.opt_dict['disable_source']:
                log_array.append(u'Source %s (%s) disabled' % (index, xml_output.channelsource[index].source))

            elif index in self.opt_dict['disable_detail_source']:
                log_array.append(u'No detailfetches from Source %s (%s)' % (index, xml_output.channelsource[index].source))

        log_array.append(u'Channel specific settings other then the above (only for the active channels):')
        for chan_def in self.channels.values():
            if not (chan_def.active or chan_def.is_child):
                continue

            log_array.append(u'[%s (Chanid=%s)]\n' % (chan_def.chan_name, chan_def.chanid))
            if chan_def.get_opt('xmltvid_alias') != None:
                log_array.append(u'  xmltvID_alias = %s\n' % (chan_def.get_opt('xmltvid_alias')))

            src_id = chan_def.opt_dict['prime_source']
            log_array.append(u'  prime_source = %s (%s)\n' % (src_id, xml_output.channelsource[src_id].source))
            if not self.opt_dict['always_use_json'] and chan_def.chanid in self.prime_source and self.prime_source[chan_def.chanid] != src_id:
                log_array.append(u'  prime_source setting: %s (%s) in sourcematching.json not used\n' % \
                    (self.prime_source[chan_def.chanid], xml_output.channelsource[self.prime_source[chan_def.chanid]].source))

            for index in chan_def.opt_dict['disable_source']:
                if index in self.opt_dict['disable_source']:
                    continue

                log_array.append(u'  Source %s (%s) disabled\n' % (index, xml_output.channelsource[index].source))

            for index in chan_def.opt_dict['disable_detail_source']:
                if index in chan_def.opt_dict['disable_source'] or index in self.opt_dict['disable_source'] or index in self.opt_dict['disable_detail_source']:
                    continue

                log_array.append(u'  Detail Source %s (%s) disabled\n' % (index, xml_output.channelsource[index].source))

            if not chan_def.get_opt('append_tvgidstv'):
                log_array.append(u'  append_tvgidstv = False\n')

            src_id = chan_def.opt_dict['prefered_description']
            if src_id != -1:
                if chan_def.get_source_id(src_id) != '':
                    log_array.append(u'  prefered_description = %s (%s)\n' % (src_id, xml_output.channelsource[src_id].source))

            if chan_def.get_opt('add_hd_id'):
                log_array.append(u'  add_hd_id = True\n')

            elif chan_def.get_opt('mark_hd'):
                log_array.append(u'  mark_hd = True\n')

            if 'slowdays' in chan_def.opt_dict.keys() and chan_def.opt_dict['slowdays'] not in (self.opt_dict['slowdays'], None):
                log_array.append(u'  slowdays = %s' % (chan_def.get_opt('slowdays')))

            if 'compat' in chan_def.opt_dict.keys() and chan_def.opt_dict['compat'] != self.opt_dict['compat']:
                log_array.append(u'  compat = %s' % (chan_def.get_opt('compat')))

            for val in ( 'fast', 'legacy_xmltvids', 'max_overlap', 'overlap_strategy', \
              'logos', 'desc_length', 'cattrans', 'disable_ttvdb', 'use_split_episodes'):
                if chan_def.get_opt(val) != self.opt_dict[val]:
                    log_array.append(u'  %s = %s\n' % (val, chan_def.get_opt(val)))

        log_array.append(u' \n')
        log(log_array, 1, 2)

    # end write_opts_to_log()

    def write_config(self, add_channels = None):
        """
        Save the channel info and the default options
        if add_channels is False or None we copy over the Channels sections, called on save_options
        If add_channels is None we convert the channel info to the new form, called on version update
        if add_channels is True we create a fresh channels section                , called on configure
        """
        self.save_oldfile(self.config_file)
        f = self.open_file(self.config_file, 'w')
        if f == None:
            return False

        f.write(u'# encoding: utf-8\n')
        f.write(u'# configversion: %s.%s%s\n' % (self.major, self.minor, '{:0>2}'.format(self.patch)))
        f.write(u'\n')

        # Save the options
        f.write(u'# See: https://github.com/tvgrabbers/tvgrabnlpy/wiki/Over_de_configuratie\n')
        f.write(u'# This is a list with default options set by the --save-options (-O)\n')
        f.write(u'# argument. Most can be overruled on the commandline.\n')
        f.write(u'# Be carefull with manually editing. Invalid options will be\n')
        f.write(u'# silently ignored. Boolean options can be set with True/False,\n')
        f.write(u'# On/Off or 1/0. Leaving it blank sets them on. Setting an invalid\n')
        f.write(u'# value sets them off. You can always check the log for the used values.\n')
        f.write(u'# To edit you beter run --save-options with all the desired defaults.\n')
        f.write(u'# Options not shown here can not be set this way.\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__CONFIG_SECTIONS__[1])
        f.write(u'# DO NOT CHANGE THIS VALUE!\n')
        f.write(u'data_version = %s\n' % self.opt_dict['data_version'])
        f.write(u'# Set always_use_json to False to ignore Channelname, Channelgroup \n')
        f.write(u'# and prime_source set in sourcematching.json if they are set different\n')
        f.write(u'# in this configuration file. If you do not have set any of those yourself\n')
        f.write(u'# leave the value to True to profite from all updates.\n')
        f.write(u'always_use_json = %s\n' % self.opt_dict['always_use_json'])
        f.write(u'group_active_channels = %s\n' % self.opt_dict['group_active_channels'])
        if self.write_info_files:
            f.write(u'write_info_files = True\n')
            if self.opt_dict['mail_log'] and self.opt_dict['mail_info_address'] != None:
                f.write(u'mail_info_address = %s\n' % self.opt_dict['mail_info_address'])

            f.write(u'\n')
        f.write(u'# The following are tuning parameters. You normally do not need to change them.\n')
        f.write(u'# global_timeout is the maximum time in seconds to wait for a fetch to complete\n')
        f.write(u'#    before calling it a time-out failure.\n')
        f.write(u'# max_simultaneous_fetches is the maximum number of simultaneous fetches\n')
        f.write(u'#    that are allowed.\n')
        f.write(u'#    With the growing number of sources it is possible that they all together\n')
        f.write(u'#    try to get their page. This could lead to congestion and failure.\n')
        f.write(u'#    If you see often "incomplete read failures" or "get_page timed out", you\n')
        f.write(u'#    can try raising the first or lowering the second.\n')
        f.write(u"#    This won't significantly alter the total runtime as this is mostley determined\n")
        f.write(u'#    by the the highest number of fetches from a single source and the mandatory.\n')
        f.write(u'#    waittime in between those fetches to not overload their resources.\n')
        f.write(u'#    However all basepage fetches are retried on failure and a detailpagefailure\n')
        f.write(u'#    can triger a retry on one of the other detailsources. So a lot of failures\n')
        f.write(u'#    especially on source 0, 1 and 9 can increase the total runtime.\n')
        f.write(u'global_timeout = %s\n' % self.opt_dict['global_timeout'])
        f.write(u'max_simultaneous_fetches = %s\n' % self.opt_dict['max_simultaneous_fetches'])
        f.write(u'\n')
        f.write(u'# This handles what goes to the log and screen\n')
        f.write(u'# 0 Nothing (use quiet mode to turns off screen output, but keep a log)\n')
        f.write(u'# 1 include Errors and Warnings\n')
        f.write(u'# 2 include page fetches\n')
        f.write(u'# 4 include (merge) summaries\n')
        f.write(u'# 8 include detail fetches and ttvdb lookups to the screen\n')
        f.write(u'# 16 include detail fetches and ttvdb lookups to the log\n')
        f.write(u'# 32 include matchlogging (see below)\n')
        f.write(u'# 64 Title renames\n')
        f.write(u'# 128 ttvdb failures\n')
        f.write(u'log_level = %s\n' % self.opt_dict['log_level'])
        f.write(u'# What match results go to the log/screen (needs code 32 above)\n')
        f.write(u'# 0 = Log Nothing (just the overview)\n')
        f.write(u'# 1 = log not matched programs added\n')
        f.write(u'# 2 = log left over programs\n')
        f.write(u'# 4 = Log matches\n')
        f.write(u'# 8 = Log group slots\n')
        f.write(u'match_log_level = %s\n' % self.opt_dict['match_log_level'])
        f.write(u'\n')
        f.write(u'mail_log = %s\n' % self.opt_dict['mail_log'])
        f.write(u'# Set "mail_log" to True to send the log to the mailaddress below\n')
        f.write(u'# Also set the mailserver and port apropriate\n')
        f.write(u'# SSL/startTLS is NOT sopported at present. Neither is authentication\n')
        f.write(u'# Make sure to first test on a console as mailing occures after \n')
        f.write(u'# closing of the logfile!\n')
        f.write(u'mail_log_address = %s\n' % self.opt_dict['mail_log_address'])
        f.write(u'mailserver = %s\n' % self.opt_dict['mailserver'])
        f.write(u'mailport = %s\n' % self.opt_dict['mailport'])
        f.write(u'\n')
        f.write(u'quiet = %s\n' % self.opt_dict['quiet'])
        f.write(u'output_file = %s\n' % self.opt_dict['output_file'])
        for index in xml_output.source_order:
            if index in self.opt_dict['disable_source']:
                f.write(u'disable_source = %s\n' % index)

            elif index in self.opt_dict['disable_detail_source']:
                f.write(u'disable_detail_source = %s\n' % index)

        f.write(u'disable_ttvdb = %s\n' % self.opt_dict['disable_ttvdb'])
        f.write(u'compat = %s\n' % self.opt_dict['compat'])
        f.write(u'legacy_xmltvids = %s\n' % self.opt_dict['legacy_xmltvids'])
        f.write(u'logos = %s\n' % self.opt_dict['logos'])
        f.write(u'use_utc = %s\n' % self.opt_dict['use_utc'])
        f.write(u'fast = %s\n' % self.opt_dict['fast'])
        f.write(u'offset = %s\n' % self.opt_dict['offset'])
        f.write(u'days = %s\n' % self.opt_dict['days'])
        f.write(u'slowdays = %s\n' % self.opt_dict['slowdays'])
        f.write(u'cattrans = %s\n' % self.opt_dict['cattrans'])
        f.write(u'mark_hd = %s\n' % self.opt_dict['mark_hd'])
        f.write(u'overlap_strategy = %s\n' % self.opt_dict['overlap_strategy'] )
        f.write(u'max_overlap = %s\n' % self.opt_dict['max_overlap'])
        f.write(u'desc_length = %s\n' % self.opt_dict['desc_length'])
        f.write(u'# Possible values for kijkwijzerstijl are:\n')
        f.write(u'#   long  : add the long descriptions and the icons\n')
        f.write(u'#   short : add the one word descriptions and the icons\n')
        f.write(u'#   single: add a single string (mythtv only reads the first item)\n')
        f.write(u"#   none  : don't add any\n")
        f.write(u'kijkwijzerstijl = %s\n' % self.opt_dict['kijkwijzerstijl'])
        f.write(u'use_split_episodes = %s\n' % self.opt_dict['use_split_episodes'])
        f.write(u'\n')

        f.write(u'# These are the channeldefinitions. You can disable a channel by placing\n')
        f.write(u'# a \'#\' in front. Seperated by \';\' you see on every line: The Name,\n')
        f.write(u'# the group, the channelID, the ID\'s for the sources in the order as\n')
        f.write(u'# returned by the "--show-sources" option and finally the iconsource and name.\n')
        f.write(u'# You can change the names to suit your own preferences.\n')
        f.write(u'# A missing ID means the source doesn\'t supply the channel.\n')
        f.write(u'# Removing an ID disables fetching from that source, but keep the \';\'s in place.\n')
        f.write(u'# But you better use the "disable_source" option as described below.\n')
        f.write(u'# Set iconsource to 99, to add your own full url.\n')
        f.write(u'\n')
        f.write(u'# To specify further Channel settings you can add sections in the form of\n')
        f.write(u'# [Channel <channelID>], where <channelID> is the third item on the line, \n')
        f.write(u'# You can use the following tags:\n')
        f.write(u'# Boolean values (True, 1, on or no value means True. Everything else False):\n')
        f.write(u'#   fast, compat, legacy_xmltvids, logos, cattrans, mark_hd, add_hd_id,\n')
        f.write(u'#   append_tvgidstv, disable_ttvdb, use_split_episodes\n')
        f.write(u'#     append_tvgidstv is True by default, which means: \'Don\'t get data\n')
        f.write(u'#     from tvgids.tv if there is from tvgids.nl\' tvgids.tv data normally is\n')
        f.write(u'#     inferiour, except for instance that for Veronica it fills in Disney XD\n')
        f.write(u'#     add_hd_id: if set to True will create two listings for the given channel.\n')
        f.write(u'#     One normal one without HD tagging and one with \'-hd\' added to the ID\n')
        f.write(u'#     and with the HD tags. This will overrule any setting of mark_hd\n')
        f.write(u'# Integer values:\n')
        f.write(u'#   slowdays, max_overlap, desc_length, prime_source, prefered_description\n')
        f.write(u'#   disable_source, disable_detail_source\n')
        f.write(u'#     prime_source (0-8) is the source whose timings and titles are dominant\n')
        f.write(u'#     It defaults to 2 for rtl channels, 4 for NPO channels, 5 for Dutch regional\n')
        f.write(u'#     and 6 for group 2 and 9 (Flemmisch) channels or else the first available\n')
        f.write(u'#     source as set in sourcematching.json (2, 4, 7, 0, 5, 1, 9, 6, 8)\n')
        f.write(u'#     prefered_description (0-8) is the source whose description, if present,\n')
        f.write(u'#     is used. It defaults to the longest description found.\n')
        f.write(u'#     with disable_source and disable_detail_source you can disable a source\n')
        f.write(u'#     for that channel either al together or only for the detail fetches\n')
        f.write(u'#     disabling an unavailable source has no effect.\n')
        f.write(u'#     With the commandline options: "--show-sources" and "--show-detail-sources"\n')
        f.write(u'#     you can get a list of available sources and their ID\n')
        f.write(u'# String values:\n')
        f.write(u'#   overlap_strategy (With possible values): \n')
        f.write(u'#     average, stop, start; everything else sets it to none\n')
        f.write(u'#   xmltvid_alias: This is a string value to be used in place of the chanid\n')
        f.write(u'#     for the xmltvID. Be careful not to set it to an existing chanid.\n')
        f.write(u'#     It can get set by configure on chanid changes! See also the WIKI\n')
        f.write(u'\n')

        def get_channel_string(chanid, active = None, chan_string = None, icon_string = None):
            chan = self.channels[chanid]
            for index in range(xml_output.source_count):
                if chan.get_source_id(index) != '':
                    break

            else:
                # There are no Source ids so we remove it
                return {'chan_string': None, 'active': None}

            if active == None:
                active = chan.active

            if chan_string == None:
                chan_string = '%s;%s;%s' % (chan.chan_name, chan.group, chanid)

            for index in range(xml_output.source_count):
                chan_string = '%s;%s' % (chan_string, chan.get_source_id(index))

            if icon_string == None:
                chan_string = '%s;%s;%s\n' % (chan_string, chan.icon_source, chan.icon)

            else:
                chan_string = '%s;%s\n' % (chan_string, icon_string)

            if active:
                return {'chan_string': chan_string, 'active': active}

            else:
                return {'chan_string': '# %s' % chan_string, 'active': active}

        # just copy over the channels section
        if add_channels == False and self.configversion == float('%s.%s' % (self.major, self.minor)):
            fo = self.open_file(self.config_file + '.old')
            if fo == None or not self.check_encoding(fo):
                # We cannot read the old config, so we create a new one
                log('Error Opening the old config. Creating a new one.\n')
                add_channels = True

            else:
                fo.seek(0,0)
                type = 0
                for byteline in fo.readlines():
                    line = self.get_line(fo, byteline, None)
                    try:
                        if  line == False:
                            continue

                        # Look for section headers
                        config_title = re.search('\[(.*?)\]', line)
                        if config_title != None and (config_title.group(1) in self.__CONFIG_SECTIONS__.values()):
                            for i, v in self.__CONFIG_SECTIONS__.items():
                                if v == config_title.group(1):
                                    type = i
                                    break

                        elif config_title != None and (config_title.group(1)[0:8] == 'Channel '):
                            type = 9

                        # Unknown Section header, so ignore
                        if line[0:1] == '[':
                            type = 0
                            continue

                        if type > 1:
                            # We just copy everything except the old configuration (type = 1)
                            f.write(line + u'\n')
                    except:
                        log('Error reading old config\n')
                        continue

                fo.close()
                f.close()
                return True

        # This is an upgrade
        chan_not_updated = []
        if add_channels != True:
            configlines = {}
            configlines['2remarks'] = []
            configlines['2'] = []
            configlines['3remarks'] = []
            configlines['3'] = []
            configlines['9'] = {}
            # Get the old channels section to convert
            fo = self.open_file(self.config_file + '.old')
            if fo == None or not self.check_encoding(fo, None, True):
                # We cannot read the old config, so we create a new one
                log('Error Opening the old config. Creating a new one.\n')
                self.get_channels()
                add_channels = True

            else:
                fo.seek(0,0)
                if self.configversion == 1.0:
                    type = 2

                else:
                    type = 0

                # Read the old configuration
                for byteline in fo.readlines():
                    line = self.get_line(fo, byteline, None, self.encoding)
                    try:
                        if line == '# encoding: utf-8' or line[0:17] == '# configversion: ' or line == False or line == '':
                            continue

                        if self.configversion != 1.0:
                            # Look for section headers
                            config_title = re.search('\[(.*?)\]', line)
                            if config_title != None and (config_title.group(1) in self.__CONFIG_SECTIONS__.values()):
                                section = config_title.group(1)
                                for i, v in self.__CONFIG_SECTIONS__.items():
                                    if v == config_title.group(1):
                                        type = i
                                        continue

                                continue

                            elif config_title != None and (config_title.group(1)[0:8] == 'Channel '):
                                section = config_title.group(1)
                                type = 9
                                chanid = config_title.group(1)[7:].strip()
                                configlines['9'][chanid] = []
                                continue

                            # Unknown Section header, so ignore
                            if line[0:1] == '[' or re.sub('#', '', line).strip() == u'Channel specific settings other then the above or the default:':
                                type = 0
                                continue

                        if type == 2 and self.configversion <= 2.0:
                            if line[0:1] == '#':
                                configlines['2remarks'].append(line)

                            else:
                                configlines['2'].append(line)

                        elif type == 3 and self.configversion > 2.0:
                            if line[0:1] == '#':
                                configlines['3remarks'].append(line)

                            else:
                                configlines['3'].append(line)

                        elif type == 9 and self.configversion > 2.0:
                            configlines['9'][chanid].append(line)

                    except:
                        log('Error reading old config\n')
                        continue

                fo.close()

                self.get_channels()
                chan_added = []
                chan_list = {}
                for g in self.group_order:
                    chan_list[unicode(g)] =[]

                if self.configversion <= 2.0:
                    for item in configlines['2']:
                        chan = item.split(None, 1) # split on first whitespace
                        if len(chan) != 2:
                            chan_not_updated.append(u'# %s\n' % (item))
                            continue

                        if chan[0].strip() in self.channels.keys():
                            chanid = chan[0].strip()
                            grp = u'0' if self.args.group_active_channels else unicode(self.channels[chanid].group)
                            chan_list[grp].append(get_channel_string(chanid, True, '%s;%s;%s' % \
                                (chan[1], self.channels[chanid].group, chanid)))
                            chan_added.append(chanid)

                        else:
                            chan_not_updated.append(u'# %s\n' % (item))

                    for item in configlines['2remarks']:
                        chan = re.sub('#', '', item)
                        chan = chan.split(None, 1) # split on first whitespace
                        if len(chan) != 2:
                            chan_not_updated.append(u'# %s\n' % (item))
                            continue

                        if chan[0].strip() in self.channels.keys():
                            chanid = chan[0].strip()
                            chan_list[unicode(self.channels[chanid].group)].append(get_channel_string(chanid, False, '%s;%s;%s' % \
                                (chan[1], self.channels[chanid].group, chanid)))
                            chan_added.append(chanid)

                        else:
                            chan_not_updated.append(u'# %s\n' % (item))

                if self.configversion > 2.0:
                    for item in configlines['3']:
                        chan = re.split(';', item)
                        if self.configversion == 2.1 and len(chan) != 8:
                            chan_not_updated.append(u'# %s\n' % (item))
                            continue

                        chan_found = False
                        for chanid, channel in self.channels.items():
                            for index in range(min(xml_output.source_count,len(chan) - 4)):
                                if (chan[index + 2].strip() !='') and (chan[index + 2].strip() == channel.get_source_id(index)):
                                    chan_found = True
                                    grp = u'0' if self.args.group_active_channels else chan[1]
                                    chan_list[grp].append(get_channel_string(chanid, True, '%s;%s;%s' % \
                                        (chan[0], chan[1], chanid), '%s;%s' % (chan[-2], chan[-1])))
                                    chan_added.append(chanid)
                                    chan_found = True
                                    break

                            if chan_found:
                                break

                        else:
                            chan_not_updated.append(u'# %s\n' % (item))

                    for item in configlines['3remarks']:
                        chan = re.sub('#', '', item)
                        if chan.strip() in self.group_order:
                            continue

                        chan = re.split(';', chan)
                        if self.configversion == 2.1 and len(chan) != 8:
                            chan_not_updated.append(u'# %s\n' % (item))
                            continue

                        chan_found = False
                        for chanid, channel in self.channels.items():
                            for index in range(min(xml_output.source_count,len(chan) - 4)):
                                if (chan[index + 2].strip() !='') and (chan[index + 2].strip() == channel.get_source_id(index)):
                                    chan_list[chan[1]].append(get_channel_string(chanid, False, '%s;%s;%s' % \
                                        (chan[0], chan[1], chanid), '%s;%s' % (chan[-2], chan[-1])))
                                    chan_added.append(chanid)
                                    chan_found = True
                                    break

                            if chan_found:
                                break

                        else:
                            chan_not_updated.append(u'# %s\n' % (item))

                del configlines['2']
                del configlines['3']
                del configlines['2remarks']
                del configlines['3remarks']
                for chanid, channel in self.channels.items():
                    if not chanid in chan_added:
                        chan_list[unicode(channel.group)].append(get_channel_string(chanid, False))

                # At a later config upgrade we here have to parse the type 9 sections

        if add_channels:
            chan_list = {}
            for g in self.group_order:
                chan_list[unicode(g)] =[]

            for chanid, channel in self.channels.items():
                if channel.group == -1:
                    continue

                grp = u'0' if self.args.group_active_channels and channel.active else unicode(channel.group)
                chan_list[grp].append(get_channel_string(chanid))

        f.write(u'[%s]\n' % self.__CONFIG_SECTIONS__[3])
        for g in self.group_order:
            if g == 0 and not self.opt_dict['group_active_channels']:
                continue

            f.write('\n')
            f.write('# %s\n' % self.chan_groups[g])
            chan_list[unicode(g)].sort(key=lambda channel: (channel['chan_string']))
            chan_list[unicode(g)].sort(key=lambda channel: (channel['active']), reverse=True)
            for channel in chan_list[unicode(g)]:
                if channel['chan_string'] != None:
                    f.write(channel['chan_string'])

        if len(chan_not_updated) > 0:
            f.write('\n')
            f.write('# Following are not converted lines!\n')
            for line in chan_not_updated:
                f.write(line)

        f.write(u'\n')
        f.write(u'# Channel specific settings other then the above or the default:\n')
        chan_names = []
        for chan_def in self.channels.values():
            for index in range(xml_output.source_count):
                if chan_def.get_source_id(index) != '':
                    # Only add specific settings if at least one sourceid present
                    chan_names.append({'active': chan_def.active, 'name': chan_def.chan_name, 'id': chan_def.chanid})
                    break

        chan_names.sort(key=lambda channel: (channel['name']))
        chan_names.sort(key=lambda channel: (channel['active']), reverse=True)

        for chan in chan_names:
            chan_def = self.channels[chan['id']]
            chan_name_written = False
            for index in chan_def.opt_dict['disable_source']:
                if index in self.opt_dict['disable_source']:
                    continue

                if chan_def.get_source_id(index) != '':
                    if not chan_name_written:
                        f.write(u'\n')
                        f.write(u'# %s\n' % (chan_def.chan_name))
                        f.write(u'[Channel %s]\n' % (chan_def.chanid))
                        chan_name_written = True

                    f.write(u'disable_source = %s\n' % index)

            for index in chan_def.opt_dict['disable_detail_source']:
                if index in chan_def.opt_dict['disable_source'] or index in self.opt_dict['disable_source'] or index in self.opt_dict['disable_detail_source']:
                    continue

                if chan_def.get_source_id(index) != '':
                    if not chan_name_written:
                        f.write(u'\n')
                        f.write(u'# %s\n' % (chan_def.chan_name))
                        f.write(u'[Channel %s]\n' % (chan_def.chanid))
                        chan_name_written = True

                    f.write(u'disable_detail_source = %s\n' % index)

            if not chan_def.get_opt('append_tvgidstv'):
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'append_tvgidstv = False\n')

            if chan_def.opt_dict['xmltvid_alias'] != None and chan_def.opt_dict['xmltvid_alias'] != chan_def.chanid:
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'xmltvid_alias = %s\n' % (chan_def.opt_dict['xmltvid_alias']))

            if not self.validate_option('prime_source', chan_def, check_default = True):
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'prime_source = %s\n' % chan_def.opt_dict['prime_source'])

            opt_val = chan_def.opt_dict['prefered_description']
            if chan_def.get_source_id(opt_val) != '':
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'prefered_description = %s\n' % ( opt_val))

            if chan_def.get_opt('add_hd_id'):
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'add_hd_id = True\n')

            if 'slowdays' in chan_def.opt_dict.keys() and chan_def.opt_dict['slowdays'] not in (self.opt_dict['slowdays'], None):
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'slowdays = %s\n' % (chan_def.get_opt('slowdays')))

            if 'compat' in chan_def.opt_dict.keys() and chan_def.opt_dict['compat'] != self.opt_dict['compat']:
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'compat = %s\n' % (chan_def.get_opt('compat')))

            if self.opt_dict['disable_ttvdb'] == False and chan_def.opt_dict['disable_ttvdb'] == True:
                if not chan_name_written:
                    f.write(u'\n')
                    f.write(u'# %s\n' % (chan_def.chan_name))
                    f.write(u'[Channel %s]\n' % (chan_def.chanid))
                    chan_name_written = True

                f.write(u'disable_ttvdb = True\n')

            for val in ( 'fast', 'legacy_xmltvids', 'max_overlap', 'overlap_strategy', \
              'logos', 'desc_length', 'cattrans', 'mark_hd', 'use_split_episodes'):
                if chan_def.get_opt(val) != self.opt_dict[val]:
                    if not chan_name_written:
                        f.write(u'\n')
                        f.write(u'# %s\n' % (chan_def.chan_name))
                        f.write(u'[Channel %s]\n' % (chan_def.chanid))
                        chan_name_written = True

                    f.write(u'%s = %s\n' % (val, chan_def.get_opt(val)))

        f.close()
        return True
    # end write_config()

    def write_defaults_list(self):
        """
        Save the genre conversion table, the title split exception list and othe translation tables
        """
        self.save_oldfile(self.settings_file)
        f = self.open_file(self.settings_file, 'w')
        if f == None:
            return False

        f.write(u'# encoding: utf-8\n')
        f.write(u'\n')
        f.write(u'# This is a list of the role-titles encountered\n')
        f.write(u'# with a translation to the english titles as used in MythTV.\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[7])

        l = []
        for i, t in self.roletrans.iteritems():
            l.append(u'%s = %s' % (i,t))
        l.sort()
        for string in l:
            f.write(u'%s\n' % string)

        f.write(u'\n')
        f.write(u'# This is a list of titles containing a \':\' not to split\n')
        f.write(u'# in a title and a subtitle\n')
        f.write(u'# These will mainly be spin-off series like \'NCIS: Los Angeles\'\n')
        f.write(u'# Movies and programs already having a subtitle are already excluded.\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[2])

        self.notitlesplit.sort()
        for t in self.notitlesplit:
             f.write(u'%s\n' % t)

        f.write(u'\n')
        f.write(u'# This is a list of grouptitles in titles containing a \':\'\n')
        f.write(u'# to remove from the title\n')
        f.write(u'# For instance \"KRO detectives\".\n')
        f.write(u'# This among others to cover diferent naming on separate sources.\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[3])

        self.groupnameremove.sort()
        for t in self.groupnameremove:
             f.write(u'%s\n' % t)

        f.write(u'\n')
        f.write(u'# This is a list of titles to rename.\n')
        f.write(u'# For instance \"navy NCIS\" to \"NCIS\".\n')
        f.write(u'# This among others to cover diferent naming on separate sources.\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[4])

        l = []
        for i, t in self.titlerename.iteritems():
            l.append(u'%s = %s' % (i,t))
        l.sort()
        for string in l:
            f.write(u'%s\n' % string)

        f.write(u'\n')
        f.write(u'# These are the translation lists for npo.nl, horizon.tv, humo.be, vpro.nl,\n')
        f.write(u'# primo.eu and vrt.be genres to tvgids.nl genre:subgenre. If you have cattrans\n')
        f.write(u'# enabled, they will next be converted according to the list further down.\n')
        f.write(u"# Notice you don't see any Movie category in the horizon list. This is ruled by\n")
        f.write(u'# a separate flag\n')
        for index in (4, 5, 6, 7, 9, 10):
            f.write(u'\n')
            f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[index+4])

            # remove doubles and sort
            for k in self.new_cattrans[index].keys():
                if not (k in self.source_cattrans[index].keys()):
                    self.source_cattrans[index][k] = self.new_cattrans[index][k]

            # format for export
            gl1 = []
            gl = []
            for k, (v1, v2) in self.source_cattrans[index].iteritems():
                if isinstance(k, (str, unicode)):
                    gl1.append('%s = %s: %s' % (k, v1, v2))

                elif isinstance(k, (list, tuple)) and len(k) == 1:
                    gl1.append('%s: = %s: %s' % (k[0], v1, v2))

                elif isinstance(k, (list, tuple)) and len(k) == 2 and k[1] != '':
                    gl.append('%s: %s = %s: %s' % (k[0], k[1], v1, v2))

            gl1.sort()
            for string in gl1:
                f.write(u'%s\n' % string)

            gl.sort()
            for string in gl:
                f.write(u'%s\n' % string)

        f.write(u'\n')
        f.write(u'# This is the list of genres to add the tvgidstv genres as subgenre\n')
        f.write(u'# tvgids.tv genres are like tvgids.nl subgenres. This is a list of what\n')
        f.write(u'# genre to add to a subgenre. Available genres are:\n')
        f.write(u'#   Amusement             Magazine                Serie/Soap\n')
        f.write(u'#   Film                  Muziek                  Sport\n')
        f.write(u'#   Informatief           Natuur                  Wetenschap\n')
        f.write(u'#   Jeugd                 Nieuws/Actualiteiten    Overige\n')
        f.write(u'#   Kunst en Cultuur      Religieus\n')
        f.write(u'# New found "subgenres" are automatically added and matched on generic rules\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[6])

        # remove doubles and sort
        gs = set(self.new_cattrans[1])
        gl = []
        for k, v in gs:
            if not (k in self.source_cattrans[1]):
                self.source_cattrans[1][k] = v

        # format for export
        for k, v in self.source_cattrans[1].iteritems():
            gl.append('%s = %s' % (v, k))
        gl.sort()

        for string in gl:
            f.write(u'%s\n' % string)

        f.write(u'\n')
        f.write(u'# This is the \'Genre:Subgenre\' conversion table\n')
        f.write(u'# \n')
        f.write(u'# \'Genre:Subgenre\' will automatically be converted to lowercase\n')
        f.write(u'# and leading and following spaces will be removed\n')
        f.write(u'# It will automatically get sorted with the genres without\n')
        f.write(u'# a subgenre at the top.\n')
        f.write(u'# Also new found values will be added on every new scan\n')
        f.write(u'# \n')
        f.write(u'# Behind the \'=\' you can supply the category to be used\n')
        f.write(u'# If a category value is empty the main category or an existing\n')
        f.write(u'# default will be used\n')
        f.write(u'# If a main category is empty the default will be supplied\n')
        f.write(u'# and used. If no default exists \'Unknown\' will be used.\n')
        f.write(u'# You should regualary check on new main categories\n')
        f.write(u'# so they don\'t get translated to \'Unknown\'\n')
        f.write(u'\n')
        f.write(u'[%s]\n' % self.__DEFAULT_SECTIONS__[1])

        l1 = []
        l2 = []

        # remove doubles
        gs = set(self.genre_list)
        # add them to cattrans if not yet known
        for i in gs:
            if not (i in self.cattrans):
                self.cattrans[i]=''

        # format for export and add main genres and sub-genres to seperate lists to get sorted
        # add missing maingenres for new subgenres
        for k1, k2 in self.cattrans:
            if k2 == '':
                l1.append('%s: = %s' % (k1, self.cattrans[(k1,k2)]))
            else:
                l2.append('%s: %s = %s' % (k1, k2, self.cattrans[(k1,k2)]))
                i = (k1, u'')
                if not (i in self.cattrans):
                    l1.append('%s:=' % k1)
        l1.sort()
        l2.sort()

        for string in l1:
            f.write(u'%s\n' % string)

        f.write(u'\n')
        for string in l2:
            f.write(u'%s\n' % string)

        f.close()
    # end write_defaults_list()

    def close(self):

        try:
            infofiles.close()

        except:
            log(['\n', 'An unexpected error has occured closing infofiles: %s\n' %  (traceback.print_exc())], 0)

        # Quiting any remaining Threads
            for source in xml_output.channelsource.values():
                if source.is_alive():
                    source.cache_return.put('quit')
                    source.quit = True

            for channel in self.channels.values():
                if channel.is_alive():
                    channel.cache_return.put('quit')
                    channel.quit = True

        if xml_output.program_cache != None and xml_output.program_cache.is_alive():
            xml_output.program_cache.cache_request.put({'task':'quit'})
            xml_output.program_cache = None

        if xml_output.ttvdb != None and xml_output.ttvdb.is_alive():
            xml_output.ttvdb.detail_request.put({'task':'quit'})
            xml_output.ttvdb = None

        # close everything neatly
        if self.opt_dict['output_file'] != None:
            try:
                self.output.close()

            except:
                pass

        logging.log_queue.put('Closing down\n')
        if logging.is_alive():
            logging.join()

        if self.log_output != None:
            self.log_output.close()

    # end close()

# end Configure
config = Configure()

# used for gathering extra info to better the code
class InfoFiles:
    """used for gathering extra info to better the code"""
    def __init__(self):

        self.detail_list = []
        self.raw_list = []
        self.raw_string = ''
        self.fetch_strings = {}
        self.info_lock = Lock()
        self.cache_return = Queue()
        self.lineup_changes = []
        self.url_failure = []

    def open_files(self):

        if config.write_info_files:
            config.save_oldfile(config.xmltv_dir + '/fetched-programs')
            self.fetch_list = config.open_file(config.xmltv_dir + '/fetched-programs','w')
            config.save_oldfile(config.xmltv_dir+'/raw_output')
            self.raw_output =  config.open_file(config.xmltv_dir+'/raw_output', 'w')

    def check_new_channels(self, source):
        if not config.write_info_files:
            return

        if source.all_channels == {}:
            source.get_channels()

        for chan_scid, channel in source.all_channels.items():
            if not (chan_scid in config.source_channels[source.proc_id].values() or chan_scid in config.empty_channels[source.proc_id]):
                self.lineup_changes.append( u'New channel on %s => %s (%s)\n' % (source.source, chan_scid, channel['name']))

        for chanid, chan_scid in config.source_channels[source.proc_id].items():
            if not (chan_scid in source.all_channels.keys() or chan_scid in config.empty_channels[source.proc_id]):
                self.lineup_changes.append( u'Removed channel on %s => %s (%s)\n' % (source.source, chan_scid, chanid))

        for chan_scid in config.empty_channels[source.proc_id]:
            if not chan_scid in source.all_channels.keys():
                self.lineup_changes.append( u"Empty channelID %s on %s doesn't exist\n" % (chan_scid, source.source))

    def add_url_failure(self, string):
        self.url_failure.append(string)

    def addto_raw_string(self, string):
        if config.write_info_files:
            with self.info_lock:
                self.raw_string = unicode(self.raw_string + string)

    def write_raw_string(self, string):
        if config.write_info_files:
            with self.info_lock:
                self.raw_string = unicode(self.raw_string + string)
                self.raw_output.write(self.raw_string + u'\n')
                self.raw_output.flush()
                self.raw_string = ''

    def addto_raw_list(self, raw_data = None):

        if config.write_info_files:
            with self.info_lock:
                if raw_data == None:
                    self.raw_list.append(self.raw_string)
                    self.raw_string = ''
                else:
                    self.raw_list.append(raw_data)

    def write_raw_list(self, raw_data = None):

        if (not config.write_info_files) or (self.raw_output == None):
            return

        with self.info_lock:
            if raw_data != None:
                self.raw_list.append(raw_data)

            self.raw_list.sort()
            for i in self.raw_list:
                i = re.sub('\n +?\n', '\n', i)
                i = re.sub('\n+?', '\n', i)
                if i.strip() == '\n':
                    continue

                self.raw_output.write(i + u'\n')

            self.raw_output.flush()
            self.raw_list = []
            self.raw_string = ''

    def addto_detail_list(self, detail_data):

        if config.write_info_files:
            with self.info_lock:
                self.detail_list.append(detail_data)

    def write_fetch_list(self, programs, chanid, source, sid = None, ismerge = False):

        if (not config.write_info_files) or (self.fetch_list == None):
            return

        with self.info_lock:
            plist = deepcopy(programs)
            if not chanid in  self.fetch_strings:
                 self.fetch_strings[chanid] = {}

            if not source in  self.fetch_strings[chanid]:
                self.fetch_strings[chanid][source] = ''

            if ismerge:
                self.fetch_strings[chanid][source] += u'(%3.0f) merging channel: %s from: %s\n' % \
                    (len(plist), config.channels[chanid].chan_name, source)

            else:
                self.fetch_strings[chanid][source] += u'(%3.0f) channel: %s from: %s\n' % \
                    (len(plist), config.channels[chanid].chan_name, source)

            plist.sort(key=lambda program: (program['start-time']))

            for tdict in plist:
                if sid == None:
                    psid = tdict['ID']

                elif sid in tdict['prog_ID']:
                    psid = tdict['prog_ID'][sid]

                else:
                    psid = ''

                self.fetch_strings[chanid][source] += u'  %s-%s: [%s][%s] %s: %s [%s/%s]\n' % (\
                                tdict['start-time'].strftime('%d %b %H:%M'), \
                                tdict['stop-time'].strftime('%H:%M'), \
                                psid.rjust(15), tdict['genre'][0:10].rjust(10), \
                                tdict['name'], tdict['titel aflevering'], \
                                tdict['season'], tdict['episode'])

            if ismerge: self.fetch_strings[chanid][source] += u'#\n'

    def write_xmloutput(self, xml):

        if config.write_info_files:
            xml_output =config.open_file(config.xmltv_dir+'/xml_output', 'w')
            if xml_output == None:
                return

            xml_output.write(xml)
            xml_output.close()

    def close(self):
        if not config.write_info_files:
            return

        if config.opt_dict['mail_info_address'] == None:
            config.opt_dict['mail_info_address'] = config.opt_dict['mail_log_address']

        if config.opt_dict['mail_log'] and len(self.lineup_changes) > 0:
            config.send_mail(self.lineup_changes, config.opt_dict['mail_info_address'], 'Tv_grab_nl_py lineup changes')

        if config.opt_dict['mail_log'] and len(self.url_failure) > 0:
            config.send_mail(self.url_failure, config.opt_dict['mail_info_address'], 'Tv_grab_nl_py url failures')

        if self.fetch_list != None:
            for chanid in config.channels.keys():
                if (config.channels[chanid].active or config.channels[chanid].is_child) and chanid in self.fetch_strings:
                    for s in config.channels[chanid].merge_order:
                        if xml_output.channelsource[s].source in self.fetch_strings[chanid].keys():
                            self.fetch_list.write(self.fetch_strings[chanid][xml_output.channelsource[s].source])

                    if chanid in config.combined_channels.keys():
                        for c in config.combined_channels[chanid]:
                            if c['chanid'] in config.channels and config.channels[c['chanid']].chan_name in self.fetch_strings[chanid]:
                                self.fetch_list.write(self.fetch_strings[chanid][config.channels[c['chanid']].chan_name])


            self.fetch_list.close()

        if self.raw_output != None:
            self.raw_output.close()

        if len(self.detail_list) > 0:
            f = config.open_file(config.xmltv_dir+'/detail_output')
            if (f != None):
                f.seek(0,0)
                for byteline in f.readlines():
                    line = config.get_line(f, byteline, False)
                    if line:
                        self.detail_list.append(line)

                f.close()

            f = config.open_file(config.xmltv_dir+'/detail_output', 'w')
            if (f != None):
                ds = set(self.detail_list)
                ds = set(self.detail_list)
                tmp_list = []
                tmp_list.extend(ds)
                tmp_list.sort()
                for i in tmp_list:
                    f.write(u'%s\n' % i)

                f.close()

# end InfoFiles
infofiles = InfoFiles()

class ProgramCache(Thread):
    """
    A cache to hold program name and category info.
    TVgids stores the detail for each program on a separate URL with an
    (apparently unique) ID. This cache stores the fetched info with the ID.
    New fetches will use the cached info instead of doing an (expensive)
    page fetch.
    """
    def __init__(self, filename=None):
        Thread.__init__(self, name = 'caching')
        """
        Create a new ProgramCache object, optionally from file
        """
        self.ID_list = {}
        self.url_list = {}
        for key in xml_output.source_order:
            self.ID_list[xml_output.channelsource[key].detail_id] = key
            self.url_list[xml_output.channelsource[key].detail_url] = key

        xml_output.channelsource[0].checkout_program_dict()
        self.field_list = ['genre', 'kijkwijzer']
        self.field_list.extend( xml_output.channelsource[0].text_values)
        self.field_list.extend( xml_output.channelsource[0].date_values)
        self.field_list.extend( xml_output.channelsource[0].datetime_values)
        self.field_list.extend( xml_output.channelsource[0].bool_values)
        self.field_list.extend( xml_output.channelsource[0].num_values)
        self.field_list.extend( xml_output.channelsource[0].video_values)
        sqlite3.register_adapter(list, self.adapt_kw)
        sqlite3.register_converter(str('kijkwijzer'), self.convert_kw)
        sqlite3.register_adapter(list, self.adapt_list)
        sqlite3.register_converter(str('listing'), self.convert_list)
        sqlite3.register_adapter(bool, self.adapt_bool)
        sqlite3.register_converter(str('boolean'), self.convert_bool)
        sqlite3.register_adapter(datetime.datetime, self.adapt_datetime)
        sqlite3.register_converter(str('datetime'), self.convert_datetime)
        sqlite3.register_adapter(datetime.date, self.adapt_date)
        sqlite3.register_converter(str('date'), self.convert_date)

        # where we store our info
        self.filename  = filename
        self.quit = False
        self.cache_request = Queue()

    def adapt_kw(self, val):
        ret_val = ''
        for k in val:
            ret_val += k

        return ret_val

    def convert_kw(self, val):
        ret_val = []
        for k in val:
            ret_val.append(k)

        return ret_val

    def adapt_list(self, val):
        if isinstance(val, (str, unicode)):
            return val

        if not isinstance(val, (list, tuple, set)) or len(val) == 0:
            return ''

        ret_val = ''
        for k in val:
            ret_val += ';%s' % k

        return ret_val[1:]

    def convert_list(self, val):
        ret_val = []
        val = val.split(';')
        for k in val:
            ret_val.append(k)

        return ret_val

    def adapt_bool(self, val):
        if val:
            return 'True'

        elif val == None:
            return 'None'

        else:
            return 'False'

    def convert_bool(self, val):
        if val == 'True':
            return True

        elif val == 'False':
            return False

        else:
            return None

    def adapt_datetime(self, val):
        if isinstance(val, (datetime.datetime)):
            if val.tzinfo == CET_CEST:
                return time.mktime(val.timetuple())*1000

            else:
                return time.mktime(val.astimezone(CET_CEST).timetuple())*1000

        else:
            return 0

    def convert_datetime(self, val):
        try:
            if int(val) == 0 or val == '':
                return None

            if len(val) < 10:
                return datetime.date.fromordinal(int(val))

            return datetime.datetime.fromtimestamp(int(val)/1000, CET_CEST)

        except:
            return None

    def adapt_date(self, val):
        if isinstance(val, (datetime.date)):
            return val.toordinal()

        return 0

    def convert_date(self, val):
        try:
            if int(val) == 0 or val == '':
                return None

            return datetime.date.fromordinal(int(val))

        except:
            return None

    def run(self):
        self.open_db()
        try:
            while True:
                if self.quit and self.cache_request.empty():
                    self.pconn.close()
                    break

                try:
                    crequest = self.cache_request.get(True, 5)

                except Empty:
                    continue

                if (not isinstance(crequest, dict)) or (not 'task' in crequest):
                    continue

                if crequest['task'] == 'query_id':
                    if not 'parent' in crequest:
                        continue

                    if self.filename == None:
                        qanswer = None

                    else:
                        for t in ('program', 'ttvdb', 'ttvdb_alias', 'tdate'):
                            if t in crequest:
                                qanswer = self.query_id(t, crequest[t])
                                break

                            else:
                                qanswer = None

                    crequest['parent'].cache_return.put(qanswer)
                    continue

                if crequest['task'] == 'query':
                    if not 'parent' in crequest:
                        continue

                    if self.filename == None:
                        qanswer = None

                    else:
                        for t in ('pid', 'ttvdb', 'ttvdb_aliasses', 'ttvdb_langs', 'ep_by_id', 'ep_by_title', 'icon', 'chan_group', 'chan_scid'):
                            if t in crequest:
                                qanswer = self.query(t, crequest[t])
                                break

                            else:
                                qanswer = None

                    crequest['parent'].cache_return.put(qanswer)
                    continue

                if self.filename == None:
                    continue

                if crequest['task'] == 'add':
                    for t in ('program', 'channelsource', 'channel', 'icon', 'ttvdb', 'ttvdb_alias', 'ttvdb_lang', 'episode'):
                        if t in crequest:
                            self.add(t, crequest[t])
                            continue

                if crequest['task'] == 'delete':
                    for t in ('ttvdb', ):
                        if t in crequest:
                            self.delete(t, crequest[t])
                            continue

                if crequest['task'] == 'clear':
                    if 'table' in crequest:
                        for t in crequest['table']:
                            self.clear(t)

                    else:
                        self.clear('programs')
                        self.clear('credits')

                    continue

                if crequest['task'] == 'clean':
                    self.clean()
                    continue

                if crequest['task'] == 'quit':
                    self.quit = True
                    continue

        except:
            log_list = ['\n', 'An unexpected error has occured in the ProgramCache thread\n']
            log_list.extend([traceback.print_exc(), '\n', \
                'If you want assistence, please attach your configuration and log files!\n', \
                '     %s\n' % (config.config_file), '     %s\n' % (config.log_file)])

            log(log_list,0)

            self.ready = True
            for source in xml_output.channelsource.values():
                if source.is_alive():
                    source.cache_return.put('quit')
                    source.quit = True

            for channel in config.channels.values():
                if channel.is_alive():
                    channel.cache_return.put('quit')
                    channel.quit = True

            return(98)

    def open_db(self):
        if self.filename == None:
            log('Cache function disabled!\n')
            return

        if os.path.isfile(self.filename) and \
          (datetime.date.today() - datetime.date.fromtimestamp(os.stat(self.filename).st_mtime)).days > 14:
            os.remove(self.filename)

        if os.path.isfile(self.filename +'.db'):
            # There is already a db file
            self.load_db()
            return

        # Check the directory
        if not os.path.exists(os.path.dirname(self.filename)):
            try:
                os.makedirs(os.path.dirname(self.filename), 0755)
                self.load_db
                return

            except:
                log('The cache directory is not accesible. Cache function disabled!\n')
                self.filename = None
                return

        self.load_db()
        # Check for an old cache file to convert
        if os.path.isfile(self.filename +'.tmp'):
            # Trying to recover a backup cache file
            if os.path.isfile(self.filename):
                if os.stat(self.filename +'.tmp').st_size > os.stat(self.filename).st_size:
                    try:
                        os.remove(self.filename)
                        os.rename(self.filename + '.tmp', self.filename)

                    except:
                        pass

                else:
                    try:
                        os.remove(self.filename + '.tmp')

                    except:
                        pass

            else:
                try:
                    os.rename(self.filename + '.tmp', self.filename)

                except:
                    pass

        if os.path.isfile(self.filename) and \
          (datetime.date.today() - datetime.date.fromtimestamp(os.stat(self.filename).st_mtime)).days < 14:
            self.load_old()

    def load_db(self):
        """
        Opens a sqlite cache db
        """
        for try_loading in (0,1):
            try:
                self.pconn = sqlite3.connect(database=self.filename + '.db', isolation_level=None, detect_types=sqlite3.PARSE_DECLTYPES)
                self.pconn.row_factory = sqlite3.Row
                pcursor = self.pconn.cursor()
                log('Verifying the database\n')
                pcursor.execute("PRAGMA main.integrity_check")
                if pcursor.fetchone()[0] == 'ok':
                    # Making a backup copy
                    self.pconn.close()
                    if os.path.isfile(self.filename +'.db.bak'):
                        os.remove(self.filename + '.db.bak')

                    shutil.copy(self.filename + '.db', self.filename + '.db.bak')
                    self.pconn = sqlite3.connect(database=self.filename + '.db', isolation_level=None, detect_types=sqlite3.PARSE_DECLTYPES)
                    self.pconn.row_factory = sqlite3.Row
                    pcursor = self.pconn.cursor()
                    break

                if try_loading == 0:
                    log(['Error loading the database: %s.db (possibly corrupt)\n' % self.filename, \
                        'Trying to load a backup copy', traceback.format_exc()])

            except:
                if try_loading == 0:
                    log(['Error loading the database: %s.db (possibly corrupt)\n' % self.filename, \
                        'Trying to load a backup copy', traceback.format_exc()])

            try:
                self.pconn.close()

            except:
                pass

            try:
                if os.path.isfile(self.filename +'.db'):
                    os.remove(self.filename + '.db')

                if os.path.isfile(self.filename +'.db.bak'):
                    if try_loading == 0:
                        shutil.copy(self.filename + '.db.bak', self.filename + '.db')

                    else:
                        os.remove(self.filename + '.db.bak')

            except:
                log(['Failed to load the database: %s\n' % self.filename, traceback.format_exc(), 'Disableing Cache function'])
                self.filename = None
                config.opt_dict['disable_ttvdb'] = True
                return

        try:
            pcursor.execute("PRAGMA main.synchronous = OFF")
            pcursor.execute("PRAGMA main.temp_store = MEMORY")
            for t in ( 'programs',  'credits', 'channels', 'channelsource', 'iconsource', 'ttvdb', 'ttvdb_alias', 'episodes'):
                # (cid, Name, Type, Nullable = 0, Default, Pri_key index)
                pcursor.execute("PRAGMA main.table_info('%s')" % (t,))
                trows = pcursor.fetchall()
                if len(trows) == 0:
                    # Table does not exist
                    self.create_table(t)
                    continue

                else:
                    clist = {}
                    for r in trows:
                        clist[r[1].lower()] = r

                    self.check_collumns(t, clist)

                self.check_indexes(t)

            for a, t in config.ttvdb_aliasses.items():
                if not self.query_id('ttvdb_alias', {'title': t, 'alias': a}):
                    self.add('ttvdb_alias', {'title': t, 'alias': a})

        except:
            log(['Failed to load the database: %s\n' % self.filename, traceback.format_exc(), 'Disableing Cache function'])
            self.filename = None
            config.opt_dict['disable_ttvdb'] = True

    def create_table(self, table):
        if table == 'programs':
            create_string = u"CREATE TABLE IF NOT EXISTS %s ('pid' TEXT PRIMARY KEY ON CONFLICT REPLACE, 'genre' TEXT DEFAULT 'overige'" % table
            for key in xml_output.channelsource[0].text_values:
                create_string = u"%s, '%s' TEXT DEFAULT ''" % (create_string, key)

            for key in xml_output.channelsource.keys():
                create_string = u"%s, '%s' TEXT DEFAULT ''" % (create_string, xml_output.channelsource[key].detail_id.lower())
                create_string = u"%s, '%s' TEXT DEFAULT ''" % (create_string, xml_output.channelsource[key].detail_url.lower())

            for key in xml_output.channelsource[0].date_values:
                create_string = u"%s, '%s' date" % (create_string, key)

            for key in xml_output.channelsource[0].datetime_values:
                create_string = u"%s, '%s' datetime" % (create_string, key)

            for key in xml_output.channelsource[0].bool_values:
                create_string = u"%s, '%s' boolean DEFAULT 'False'" % (create_string, key)

            for key in xml_output.channelsource[0].num_values:
                create_string = u"%s, '%s' INTEGER DEFAULT 0" % (create_string, key)

            for key in xml_output.channelsource[0].video_values:
                create_string = u"%s, '%s' boolean DEFAULT 'False'" % (create_string, key)

            create_string = u"%s, 'kijkwijzer' kijkwijzer DEFAULT '')" % create_string

        elif table == 'credits':
            create_string = u"CREATE TABLE IF NOT EXISTS %s " % table
            create_string += u"('pid' TEXT"
            create_string += u", 'title' TEXT"
            create_string += u", 'name' TEXT"
            create_string += u", PRIMARY KEY ('pid', 'title', 'name') ON CONFLICT REPLACE)"
            if (sqlite3.sqlite_version_info >= (3, 8, 2)):
                create_string += u" WITHOUT ROWID"


        elif table == 'ttvdb':
            create_string = u"CREATE TABLE IF NOT EXISTS %s "  % table
            create_string += u"('title' TEXT PRIMARY KEY ON CONFLICT REPLACE"
            create_string += u", 'tid' INTEGER"
            create_string += u", 'langs' listing"
            create_string += u", 'tdate' date)"
            if (sqlite3.sqlite_version_info >= (3, 8, 2)):
                create_string += u" WITHOUT ROWID"


        elif table == 'ttvdb_alias':
            create_string = u"CREATE TABLE IF NOT EXISTS %s "  % table
            create_string += u"('alias' TEXT PRIMARY KEY ON CONFLICT REPLACE"
            create_string += u", 'title' TEXT)"
            if (sqlite3.sqlite_version_info >= (3, 8, 2)):
                create_string += u" WITHOUT ROWID"

        elif table == 'episodes':
            create_string = u"CREATE TABLE IF NOT EXISTS %s "  % table
            create_string += u"('tid' INTEGER"
            create_string += u", 'sid' INTEGER"
            create_string += u", 'eid' INTEGER"
            create_string += u", 'lang' TEXT DEFAULT 'nl'"
            create_string += u", 'title' TEXT"
            create_string += u", 'description' TEXT"
            create_string += u", 'airdate' date"
            create_string += u", PRIMARY KEY ('tid', 'sid', 'eid', 'lang') ON CONFLICT REPLACE)"
            if (sqlite3.sqlite_version_info >= (3, 8, 2)):
                create_string += u" WITHOUT ROWID"


        elif table == 'channels':
            create_string = u"CREATE TABLE IF NOT EXISTS %s " % table
            create_string += u"('chanid' TEXT PRIMARY KEY ON CONFLICT REPLACE"
            create_string += u", 'cgroup' INTEGER DEFAULT 10"
            create_string += u", 'name' TEXT)"

        elif table == 'channelsource':
            create_string = u"CREATE TABLE IF NOT EXISTS %s " % table
            create_string += u"( 'chanid' TEXT"
            create_string += u", 'sourceid' INTEGER"
            create_string += u", 'scid' TEXT"
            create_string += u", 'name' TEXT"
            create_string += u", 'hd' boolean DEFAULT 'False'"
            create_string += u", 'emptycount' INTEGER DEFAULT 0"
            create_string += u", PRIMARY KEY ('chanid', 'sourceid') ON CONFLICT REPLACE)"
            if (sqlite3.sqlite_version_info >= (3, 8, 2)):
                create_string += u" WITHOUT ROWID"


        elif table == 'iconsource':
            create_string = u"CREATE TABLE IF NOT EXISTS %s " % table
            create_string += u"('chanid' TEXT"
            create_string += u", 'sourceid' INTEGER"
            create_string += u", 'icon' TEXT"
            create_string += u", PRIMARY KEY ('chanid', 'sourceid') ON CONFLICT REPLACE)"
            if (sqlite3.sqlite_version_info >= (3, 8, 2)):
                create_string += u" WITHOUT ROWID"

        else:
            return

        with self.pconn:
            try:
                self.pconn.execute(create_string)

            except:
                log(['Error creating the %s table!\n' % table, traceback.format_exc()])

    def check_collumns(self, table, clist):
        def add_collumn(table, collumn):
            try:
                with self.pconn:
                    self.pconn.execute(u"ALTER TABLE %s ADD %s" % (table, collumn))

            except:
                log('Error updating the %s table with collumn "%s"!\n' % (table, collumn))

        def drop_table(table):
            with self.pconn:
                self.pconn.execute(u"DROP TABLE IF EXISTS %s" % (table,))

        if table == 'programs':
            if 'pid' not in clist.keys():
                drop_table(table)
                self.create_table(table)
                return

            if 'genre' not in clist.keys():
                add_collumn(table, u"'genre' TEXT DEFAULT 'overige'")

            if 'kijkwijzer' not in clist.keys():
                add_collumn(table, u"'kijkwijzer' kijkwijzer DEFAULT ''")

            for c in xml_output.channelsource[0].text_values:
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' TEXT DEFAULT ''" % c)

            for key in xml_output.channelsource.keys():
                if xml_output.channelsource[key].detail_id.lower() not in clist.keys():
                    add_collumn(table, u"'%s' TEXT DEFAULT ''" % xml_output.channelsource[key].detail_id.lower())

                if xml_output.channelsource[key].detail_url.lower() not in clist.keys():
                    add_collumn(table, u"'%s' TEXT DEFAULT ''" % xml_output.channelsource[key].detail_url.lower())

            for c in xml_output.channelsource[0].date_values:
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' date" % c)

            for c in xml_output.channelsource[0].datetime_values:
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' datetime" % c)

            for c in xml_output.channelsource[0].bool_values:
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' boolean DEFAULT 'False'" % c)

            for c in xml_output.channelsource[0].num_values:
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' INTEGER DEFAULT 0" % c)

            for c in xml_output.channelsource[0].video_values:
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' boolean DEFAULT 'False'" % c)

        elif table == 'credits':
            for c in ('pid', 'title', 'name'):
                if c.lower() not in clist.keys():
                    drop_table(table)
                    self.create_table(table)
                    return

        elif table == 'ttvdb':
            for c in ('title', ):
                if c.lower() not in clist.keys():
                    drop_table(table)
                    self.create_table(table)
                    drop_table('episodes')
                    self.create_table('episodes')
                    return

            if 'tid' not in clist.keys():
                add_collumn(table, u"'tid' INTEGER")

            if 'langs' not in clist.keys():
                add_collumn(table, u"'langs' listing")

            if 'tdate' not in clist.keys():
                add_collumn(table, u"'tdate' date")

        elif table == 'ttvdb_alias':
            for c in ('alias', ):
                if c.lower() not in clist.keys():
                    drop_table(table)
                    self.create_table(table)
                    return

            if 'title' not in clist.keys():
                add_collumn(table, u"'title' TEXT")

        elif table == 'episodes':
            for c in ('tid', 'sid', 'eid', 'lang'):
                if c.lower() not in clist.keys():
                    drop_table(table)
                    self.create_table(table)
                    return

            for c in ('title', 'description'):
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' TEXT" % c)

            if 'airdate' not in clist.keys():
                add_collumn(table, u"'airdate' date")

        elif table == 'channels':
            if 'chanid' not in clist.keys():
                drop_table(table)
                self.create_table(table)
                return

            if 'cgroup' not in clist.keys():
                add_collumn(table, u"'cgroup' INTEGER")

            if 'name' not in clist.keys():
                add_collumn(table, u"'name' TEXT")

        elif table == 'channelsource':
            for c in ('chanid', 'sourceid'):
                if c.lower() not in clist.keys():
                    drop_table(table)
                    self.create_table(table)
                    return

            for c in ('scid', 'name'):
                if c.lower() not in clist.keys():
                    add_collumn(table, u"'%s' TEXT" % c)

            if 'hd' not in clist.keys():
                add_collumn(table, u"'hd' boolean DEFAULT 'False'")

            if 'emptycount' not in clist.keys():
                add_collumn(table, u"'emptycount' INTEGER DEFAULT 0")

        elif table == 'iconsource':
            for c in ('chanid', 'sourceid'):
                if c.lower() not in clist.keys():
                    drop_table(table)
                    self.create_table(table)
                    return

            if 'icon' not in clist.keys():
                add_collumn(table, u"'icon' TEXT")

    def check_indexes(self, table):
        def add_index(table, i, clist):
            try:
                with self.pconn:
                    self.pconn.execute(u"CREATE INDEX IF NOT EXISTS '%s' ON %s %s" % (i, table, clist))

            except:
                log('Error updating the %s table with Index "%s"!\n' % (table, i))

        pcursor = self.pconn.cursor()
        # (id, Name, Type, Nullable = 0, Default, Pri_key index)
        pcursor.execute("PRAGMA main.index_list(%s)" % (table,))
        ilist = {}
        for r in pcursor.fetchall():
            ilist[r[1].lower()] = r

        if table == 'programs':
            if 'stoptime' not in ilist:
                add_index( table, 'stoptime', "('stop-time')")

        elif table == 'credits':
            if 'credtitle' not in ilist:
                add_index( table, 'credtitle', "('pid', 'title')")

        elif table == 'ttvdb':
            if 'ttvdbtid' not in ilist:
                add_index( table, 'ttvdbtid', "('tid')")

        elif table == 'episodes':
            if 'eptitle' not in ilist:
                add_index( table, 'eptitle', "('title')")

        elif table == 'channels':
            if 'cgroup' not in ilist:
                add_index( table, 'cgroup', "('cgroup')")

            if 'chan_name' not in ilist:
                add_index( table, 'chan_name', "('name')")

        elif table == 'channelsource':
            if 'scid' not in ilist:
                add_index( table, 'scid', "('scid')")

    def load_old(self):
        """
        Loads a pickled cache dict from file
        """
        try:
            pdict = pickle.load(open(self.filename,'r'))

        except:
            log(['Error loading old cache file: %s (possibly corrupt)\n' % self.filename, traceback.format_exc()])
            return

        dnow = datetime.date.today()
        log(['Converting the old pickle cache to sqlite.\n', 'This may take some time!\n'])
        pcount = 0
        for p in pdict.values():
            if 'stop-time'  in p and 'name'  in p and \
                    p['stop-time'].date() >= dnow and \
                    type(p['name']) == unicode and \
                    p['name'].lower() != 'onbekend':

                self.add(p)
                pcount += 1

        log('Added %s program records to the database.\n' % pcount)

    def query(self, table='pid', item=None):
        """
        Updates/gets/whatever.
        """
        pcursor = self.pconn.cursor()
        if table == 'pid':
            pcursor.execute(u"SELECT * FROM programs WHERE pid = ?", (item,))
            r = pcursor.fetchone()
            if r == None:
                return

            program = xml_output.channelsource[0].checkout_program_dict()
            for item in r.keys():
                if item == 'pid':
                    continue

                elif item in xml_output.channelsource[0].video_values:
                    program['video'][item] = r[item]

                elif item in self.ID_list.keys():
                    program['prog_ID'][self.ID_list[item]] = r[item]

                elif item in self.url_list.keys():
                    program['detail_url'][self.url_list[item]] = r[item]

                else:
                    program[item] = r[item]

            pcursor.execute(u"SELECT * FROM credits WHERE pid = ?", (item,))
            for r in pcursor.fetchall():
                if not r[str('title')] in program['credits'].keys():
                    program['credits'][r[str('title')]] = []

                program['credits'][r[str('title')]].append(r[str('name')])

            program = xml_output.channelsource[0].checkout_program_dict(program)
            return program

        elif table == 'ttvdb':
            pcursor.execute(u"SELECT * FROM ttvdb WHERE tid = ?", (item,))
            r = pcursor.fetchone()
            if r == None:
                return

            serie = {}
            serie['tid'] = r[str('tid')]
            serie['title'] = r[str('title')]
            serie['tdate'] = r[str('tdate')]
            return serie

        elif table == 'ttvdb_aliasses':
            pcursor.execute(u"SELECT alias FROM ttvdb_alias WHERE lower(title) = ?", (item.lower(), ))
            r = pcursor.fetchall()
            aliasses = []
            if r != None:
                for a in r:
                    aliasses.append( a[0])

            return aliasses

        elif table == 'ttvdb_langs':
            pcursor.execute(u"SELECT langs FROM ttvdb WHERE tid = ?", (item['tid'],))
            r = pcursor.fetchone()
            aliasses = []
            if r == None:
                return r[0]

            else:
                return []

        elif table == 'ep_by_id':
            qstring = u"SELECT * FROM episodes WHERE tid = ?"
            qlist = [item['tid']]
            if item['sid'] > 0:
                qstring += u" and sid = ?"
                qlist.append(item['sid'])

            if item['eid'] > 0:
                qstring += u" and eid = ?"
                qlist.append(item['eid'])

            if 'lang' in item:
                qstring += u" and lang = ?"
                qlist.append(item['lang'])

            pcursor.execute(qstring, tuple(qlist))

            r = pcursor.fetchall()
            series = []
            for s in r:
                series.append({'tid': int(s[str('tid')]),
                                          'sid': int(s[str('sid')]),
                                          'eid': int(s[str('eid')]),
                                          'title': s[str('title')],
                                          'airdate': s[str('airdate')],
                                          'lang': s[str('lang')],
                                          'description': s[str('description')]})
            return series

        elif table == 'ep_by_title':
            pcursor.execute(u"SELECT * FROM episodes WHERE tid = ? and lower(title) = ?", (item['tid'], item['title'].lower(), ))
            r = pcursor.fetchone()
            if r == None:
                return

            serie = {}
            serie['tid'] = int(r[str('tid')])
            serie['sid'] = int(r[str('sid')])
            serie['eid'] = int(r[str('eid')])
            serie['title'] = r[str('title')]
            serie['airdate'] = r[str('airdate')]
            serie['lang'] = r[str('lang')]
            serie['description'] = r[str('description')]
            return serie
        elif table == 'icon':
            if item == None:
                pcursor.execute(u"SELECT chanid, sourceid, icon FROM iconsource")
                r = pcursor.fetchall()
                icons = {}
                if r != None:
                    for g in r:
                        if not g[0] in icons:
                            icons[g[0]] ={}

                        icons[g[0]][g[1]] = g[2]

                return icons

            else:
                pcursor.execute(u"SELECT icon FROM iconsource WHERE chanid = ? and sourceid = ?", (item['chanid'], item['sourceid']))
                r = pcursor.fetchone()
                if r == None:
                    return

                return {'sourceid':  item['sourceid'], 'icon': r[0]}

        elif table == 'chan_group':
            if item == None:
                pcursor.execute(u"SELECT chanid, cgroup, name FROM channels")
                r = pcursor.fetchall()
                changroups = {}
                if r != None:
                    for g in r:
                        changroups[g[0]] = {'name': g[2],'cgroup': int(g[1])}

                return changroups

            else:
                pcursor.execute(u"SELECT cgroup, name FROM channels WHERE chanid = ?", (item['chanid'],))
                r = pcursor.fetchone()
                if r == None:
                    return

                return {'cgroup':r[0], 'name': r[1]}

        elif table == 'chan_scid':
            if item == None:
                pcursor.execute(u"SELECT chanid, sourceid, scid, name, hd FROM channelsource")
                r = pcursor.fetchall()
                scids = {}
                if r != None:
                    for g in r:
                        if not g[0] in scids:
                            scids[g[0]] ={}

                        scids[g[0]][g[1]] = {'scid': g[2],'name': g[3], 'hd': g[4]}

                return scids

            elif 'chanid' in item and 'sourceid' in item:
                pcursor.execute(u"SELECT scid FROM channelsource WHERE chanid = ? and sourceid = ?", (item['chanid'], item['sourceid']))
                r = pcursor.fetchone()
                if r == None:
                    return

                return scid

            elif 'sourceid' in item:
                pcursor.execute(u"SELECT scid, chanid, name FROM channelsource WHERE sourceid = ?", (item['sourceid']))
                r = pcursor.fetchall()
                scids = {}
                if r != None:
                    for g in r:
                        if not g[0] in scids:
                            scids[g[0]] ={}

                        scids[g[0]] = {'chanid': g[1],'name': g[2]}

                return scids

    def query_id(self, table='program', item=None):
        """
        Check which ID is used
        """
        pcursor = self.pconn.cursor()
        if table == 'program':
            ID_list = [item['ID']]
            for key in xml_output.source_order:
                if item['prog_ID'][key] != '' and item['prog_ID'][key] != None:
                    ID_list.append(item['prog_ID'][key])

            for id in ID_list:
                pcursor.execute(u"SELECT pid FROM programs WHERE pid = ?", (id,))
                if pcursor.fetchone() != None:
                    return id

            return None

        elif table == 'ttvdb':
            pcursor.execute(u"SELECT ttvdb.tid, tdate, ttvdb.title, ttvdb.langs FROM ttvdb JOIN ttvdb_alias " + \
                    "ON lower(ttvdb.title) = lower(ttvdb_alias.title) WHERE lower(alias) = ?", \
                    (item['title'].lower(), ))
            r = pcursor.fetchone()
            if r == None:
                pcursor.execute(u"SELECT tid, tdate, title, langs FROM ttvdb WHERE lower(title) = ?", (item['title'].lower(), ))
                r = pcursor.fetchone()
                if r == None:
                    return

            return {'tid': r[0], 'tdate': r[1], 'title': r[2], 'langs': r[3]}

        elif table == 'ttvdb_alias':
            pcursor.execute(u"SELECT title FROM ttvdb_alias WHERE lower(alias) = ?", (item['alias'].lower(), ))
            r = pcursor.fetchone()
            if r == None:
                if 'title' in item:
                    return False

                else:
                    return

            if 'title' in item:
                if item['title'].lower() == r[0].lower():
                    return True

                else:
                    return False

            else:
                return {'title': r[0]}

        elif table == 'tdate':
            pcursor.execute(u"SELECT tdate FROM ttvdb WHERE tid = ?", (item,))
            r = pcursor.fetchone()
            if r == None:
                return

            return r[0]

        elif table == 'chan_group':
            pcursor.execute(u"SELECT cgroup, name FROM channels WHERE chanid = ?", (item['chanid'],))
            r = pcursor.fetchone()
            if r == None:
                return

            return r[0]

    def add(self, table='program', item=None):
        """
        Adds a record
        """
        pcursor = self.pconn.cursor()
        rec = []
        rec_upd = []
        if table == 'program':
            cache_id = self.query_id('program', item)
            if cache_id != None:
                with self.pconn:
                    self.pconn.execute(u"DELETE FROM programs WHERE pid = ?", (cache_id,))
                    self.pconn.execute(u"DELETE FROM credits WHERE pid = ?", (cache_id,))

            if item['ID'] != '' and item['ID'] != None:
                id = item['ID']

            else:
                for key in xml_output.source_order:
                    if item['prog_ID'][key] != '' and item['prog_ID'][key] != None:
                        id = item['prog_ID'][key]
                        break

                else:
                    log('Error saving program %s to the cache.\n' %  item['name'])
                    return

            sql_flds = u"INSERT INTO programs ('pid'"
            sql_cnt = u"VALUES (?"
            sql_vals = [id]
            for f, v in item.items():
                if f in self.field_list:
                    sql_flds = u"%s, '%s'" % (sql_flds, f)
                    sql_cnt = u"%s, ?" % (sql_cnt)
                    sql_vals.append(v)

            for f, v in item['video'].items():
                sql_flds = u"%s, '%s'" % (sql_flds, f)
                sql_cnt = u"%s, ?" % (sql_cnt)
                sql_vals.append(v)

            for f, v in item['prog_ID'].items():
                sql_flds = u"%s, '%s'" % (sql_flds, xml_output.channelsource[f].detail_id)
                sql_cnt = u"%s, ?" % (sql_cnt)
                sql_vals.append(v)

            for f, v in item['detail_url'].items():
                sql_flds = u"%s, '%s'" % (sql_flds, xml_output.channelsource[f].detail_url)
                sql_cnt = u"%s, ?" % (sql_cnt)
                sql_vals.append(v)

            add_string = u"%s) %s)" % (sql_flds, sql_cnt)
            with self.pconn:
                self.pconn.execute(add_string, tuple(sql_vals))

            add_string = u"INSERT INTO credits (pid, title, name) VALUES (?, ?, ?)"
            for f, v in item['credits'].items():
                rec.append((id, f, v))

        elif table == 'channel':
            add_string = u"INSERT INTO channels ('chanid', 'cgroup', 'name') VALUES (?, ?, ?)"
            update_string = u"UPDATE channels SET `cgroup` = ?, `name` = ? WHERE chanid = ?"
            if isinstance(item, dict):
                item = [item]

            if isinstance(item, list):
                g = self.query('chan_group')

                for c in item:
                    if not c['chanid'] in g.keys():
                        rec.append((c['chanid'], c['cgroup'], c['name']))

                    elif g[c['chanid']]['name'].lower() != c['name'].lower() or g[c['chanid']]['cgroup'] != c['cgroup'] \
                      or (g[c['chanid']]['cgroup'] == 10 and c['cgroup'] not in (-1, 0, 10)):
                        rec_upd.append((c['cgroup'], c['name'] , c['chanid']))

        elif table == 'channelsource':
            add_string = u"INSERT INTO channelsource ('chanid', 'sourceid', 'scid', 'name', 'hd') VALUES (?, ?, ?, ?, ?)"
            update_string = u"UPDATE channelsource SET 'scid'= ?, 'name'= ?, 'hd'= ? WHERE chanid = ? and sourceid = ?"
            if isinstance(item, dict):
                item = [item]

            if isinstance(item, list):
                scids = self.query('chan_scid')
                for c in item:
                    if c['scid'] == '':
                        continue

                    if c['chanid'] in scids and c['sourceid'] in scids[c['chanid']]:
                        rec_upd.append((c['scid'], c['name'], c['hd'], c['chanid'], c['sourceid']))

                    else:
                        rec.append((c['chanid'], c['sourceid'], c['scid'], c['name'], c['hd']))

        elif table == 'icon':
            add_string = u"INSERT INTO iconsource ('chanid', 'sourceid', 'icon') VALUES (?, ?, ?)"
            update_string = u"UPDATE iconsource SET 'icon'= ? WHERE chanid = ? and sourceid = ?"
            if isinstance(item, dict):
                item = [item]

            if isinstance(item, list):
                icons = self.query('icon')
                for ic in item:
                    if ic['chanid'] in icons and ic['sourceid'] in icons[ic['chanid']] \
                      and icons[ic['chanid']][ic['sourceid']] != ic['icon']:
                        rec_upd.append((ic['icon'], ic['chanid'], ic['sourceid']))

                    else:
                        rec.append((ic['chanid'], ic['sourceid'], ic['icon']))

        elif table == 'ttvdb':
            add_string = u"INSERT INTO ttvdb ('tid', 'title', 'langs', 'tdate') VALUES (?, ?, ?, ?)"
            update_string = ''
            rec.append((int(item['tid']), item['title'], list(item['langs']), datetime.date.today()))

        elif table == 'ttvdb_lang':
            add_string = u"INSERT INTO ttvdb ('tid', 'title', 'tdate', 'langs') VALUES (?, ?, ?, ?)"
            update_string = u"UPDATE ttvdb SET langs = ?, tdate = ? WHERE tid = ?"
            g = self.query('ttvdb_langs', int(item['tid']))
            if len(g) == 0:
                rec.append((int(item['tid']), item['title'], datetime.date.today(), item['lang']))

            else:
                langs = g[0]
                if item['lang'] not in langs:
                    langs.append(item['lang'])
                    rec_upd.append((langs , datetime.date.today(), int(item['tid'])))

        elif table == 'ttvdb_alias':
            add_string = u"INSERT INTO ttvdb_alias ('title', 'alias') VALUES (?, ?)"
            aliasses = self.query('ttvdb_aliasses', item['title'])
            if isinstance(item['alias'], list) and len(item['alias']) > 0:
                for a in item['alias']:
                    if not a in aliasses:
                        rec.append((item['title'], a))

            else:
                if not item['alias'] in aliasses:
                    rec.append((item['title'], item['alias']))

        elif table == 'episode':
            add_string = u"INSERT INTO episodes ('tid', 'sid', 'eid', 'title', 'airdate', 'lang', 'description') " + \
                                  u"VALUES (?, ?, ?, ?, ?, ?, ?)"
            update_string = u"UPDATE episodes SET title = ?, airdate = ?, description = ? " + \
                                       u"WHERE tid = ? and sid = ? and eid = ? and lang = ?"
            if isinstance(item, dict):
                item = [item]

            if isinstance(item, list):
                rec = []
                rec_upd = []
                for e in item:
                    ep = self.query('ep_by_id', e)
                    if ep == None or len(ep) == 0:
                        rec.append((int(e['tid']), int(e['sid']), int(e['eid']), e['title'], e['airdate'], e['lang'], e['description']))

                    elif ep[0]['title'].lower() != e['title'].lower() or ep[0]['airdate'] != e['airdate']:
                        rec_upd.append((e['title'], e['airdate'], int(e['tid']), int(e['sid']), int(e['eid']), e['lang'], e['description']))

        if len(rec_upd) == 1:
            with self.pconn:
                self.pconn.execute(update_string, rec_upd[0])

        elif len(rec_upd) > 1:
            with self.pconn:
                self.pconn.executemany(update_string, rec_upd)

        if len(rec) == 1:
            with self.pconn:
                self.pconn.execute(add_string, rec[0])

        elif len(rec) > 1:
            with self.pconn:
                self.pconn.executemany(add_string, rec)

    def delete(self, table='ttvdb', item=None):
        if table == 'ttvdb':
            with self.pconn:
                self.pconn.execute(u"DELETE FROM ttvdb WHERE tid = ?",  (int(item['tid']), ))
                self.pconn.execute(u"DELETE FROM episodes WHERE tid = ?",  (int(item['tid']), ))

    def clear(self, table):
        """
        Clears the cache (i.e. empties it)
        """
        with self.pconn:
            self.pconn.execute(u"DROP TABLE IF EXISTS %s" % table)

        with self.pconn:
            self.pconn.execute(u"VACUUM")

        self.create_table(table)
        self.check_indexes(table)

    def clean(self):
        """
        Removes all cached programming before today.
        And ttvdb ids older then 30 days
        """
        dnow = int(time.mktime(datetime.date.today().timetuple())*1000)
        with self.pconn:
            self.pconn.execute(u"DELETE FROM programs WHERE `stop-time` < ?", (dnow,))

        with self.pconn:
            self.pconn.execute(u"DELETE FROM credits WHERE NOT EXISTS (SELECT * FROM programs WHERE programs.pid = credits.pid)")

        dnow = datetime.date.today().toordinal()
        with self.pconn:
            self.pconn.execute(u"DELETE FROM ttvdb WHERE tdate < ?", (dnow - 30,))

        with self.pconn:
            self.pconn.execute(u"VACUUM")

# end ProgramCache

class FetchURL(Thread):
    """
    A simple thread to fetch a url with a timeout
    """
    def __init__ (self, url, encoding = "default encoding", accept_header = None):
        Thread.__init__(self, name = 'fetching')
        self.url = url
        self.result = None
        self.encoding = encoding
        self.accept_header = accept_header

    def run(self):
        with xml_output.output_lock:
            xml_output.fetch_count += 1

        try:
            self.result = self.get_page_internal(self.url, self.encoding)

        except:
            log('An unexpected error "%s:%s" has occured while fetching page: %s\n' %  (sys.exc_info()[0], sys.exc_info()[1], self.url), 0)
            if config.write_info_files:
                infofiles.add_url_failure('%s,%s:\n  %s\n' % (sys.exc_info()[0], sys.exc_info()[1], self.url))

            return None

    def find_html_encoding(self, httphead, htmlhead, default_encoding="default encoding"):
        # look for the text '<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />'
        # in the first 600 bytes of the HTTP page
        m = re.search(r'<meta[^>]+\bcharset=["\']?([A-Za-z0-9\-]+)\b', htmlhead[:512].decode('ascii', 'ignore'))
        if m:
            return m.group(1)

        # Find a HTTP header: Content-Type: text/html; charset=UTF-8
        m = re.search(r'\bcharset=([A-Za-z0-9\-]+)\b', httphead.info().getheader('Content-Type'))
        if m:
            return m.group(1)

        elif default_encoding == "default encoding":
            return config.httpencoding

        else:
            return default_encoding # the default HTTP encoding.

    def get_page_internal(self, url, encoding = "default encoding"):
        """
        Retrieves the url and returns a string with the contents.
        Optionally, returns None if processing takes longer than
        the specified number of timeout seconds.
        """
        txtdata = None
        txtheaders = {'Keep-Alive' : '300',
                      'User-Agent' : config.user_agents[random.randint(0, len(config.user_agents)-1)] }

        if self.accept_header != None:
            txtheaders['Accept'] = self.accept_header

        try:
            rurl = urllib.Request(url, txtdata, txtheaders)
            fp = urllib.urlopen(rurl)
            bytes = fp.read()
            page = None

            encoding = self.find_html_encoding(fp, bytes, encoding)

            try:
                page = bytes.decode(encoding, 'replace')

            except:
                log('Cannot decode url %s as %s, trying Windows-1252\n' % (url, encoding))
                # 'Windows-1252'
                page = bytes.decode('Windows-1252', 'ignore') # At least gets it somewhat correct

            return page

        except (urllib.URLError) as e:
            log('Cannot open url %s: %s\n' % (url, e.reason), 1, 1)
            if config.write_info_files:
                infofiles.add_url_failure('URLError: %s: %s\n' % (e.reason,url))

            return None

        except (urllib.HTTPError) as e:
            log('Cannot parse url %s: code=%s\n' % (url, e.code), 1, 1)
            if config.write_info_files:
                infofiles.add_url_failure('HTTPError: %s: %s\n' % (e.code, url))

            return None

        except (httplib.IncompleteRead):
            log('Cannot retrieve full url %s: %s\n' % (url, sys.exc_info()[1]), 1, 1)
            if config.write_info_files:
                infofiles.add_url_failure('IncompleteRead: %s\n' % url)

            return None

# end FetchURL

class theTVDB(Thread):
    def __init__(self):
        Thread.__init__(self, name = 'source-thetvdb')
        self.quit = False
        self.ready = False
        self.state = 0
        self.active = True
        self.api_key = "0629B785CE550C8D"
        self.detail_request = Queue()
        self.cache_return = Queue()
        self.source_lock = Lock()
        self.fetch_count = 0
        self.fail_count = 0

    def run(self):
        self.lastrequest = None
        if config.opt_dict['disable_ttvdb']:
            return
        try:
            self.state = 2
            while True:
                if self.quit and self.detail_request.empty():
                    self.state = 0
                    break

                try:
                    crequest = self.detail_request.get(True, 5)
                    self.lastrequest = datetime.datetime.now()

                except Empty:
                    continue

                if (not isinstance(crequest, dict)) or (not 'task' in crequest):
                    continue

                if crequest['task'] == 'update_ep_info':
                    if not 'parent' in crequest:
                        continue

                    if 'tdict' in crequest:
                        qanswer = self.get_season_episode(crequest['parent'], crequest['tdict'])
                        qanswer = xml_output.channelsource[0].checkout_program_dict(qanswer)
                        if qanswer['ID'] != '':
                            xml_output.program_cache.cache_request.put({'task':'add', 'program': qanswer})

                        with crequest['parent'].channel_lock:
                            crequest['parent'].detailed_programs.append(qanswer)


                    crequest['parent'].update_counter('fetch', -1, False)
                    continue

                if crequest['task'] == 'last_one':
                    if not 'parent' in crequest:
                        continue

                    crequest['parent'].detail_data.set()

                if crequest['task'] == 'quit':
                    self.quit = True
                    continue

        except:
            log_list = ['\n', 'An unexpected error has occured in the ttvdb thread\n']
            log_list.extend([traceback.print_exc(), '\n', \
                'If you want assistence, please attach your configuration and log files!\n', \
                '     %s\n' % (config.config_file), '     %s\n' % (config.log_file)])

            log(log_list,0)

            self.ready = True
            self.state = 0
            for source in xml_output.channelsource.values():
                if source.is_alive():
                    source.cache_return.put('quit')
                    source.quit = True

            for channel in config.channels.values():
                if channel.is_alive():
                    channel.cache_return.put('quit')
                    channel.quit = True

            return(98)

    def query_ttvdb(self, type='seriesid', title=None, lang='nl'):
        base_url = "http://www.thetvdb.com"
        api_key = '0BB856A59C51D607'
        if isinstance(title, (int, str)):
            title = unicode(title)

        title = urllib.quote(title.encode("utf-8"))
        if type == 'seriesid':
            if not lang in ('all', 'cs', 'da', 'de', 'el', 'en', 'es', 'fi', 'fr', 'he', 'hr', 'hu', 'it',
                                'ja', 'ko', 'nl', 'no', 'pl', 'pt', 'ru', 'sl', 'sv', 'tr', 'zh'):
                lang = 'en'

            if title != None:
                data = self.get_page('%s/api/GetSeries.php?seriesname=%s&language=%s' % (base_url, title, lang))

        elif type == 'episodes':
            if not lang in ('cs', 'da', 'de', 'el', 'en', 'es', 'fi', 'fr', 'he', 'hr', 'hu', 'it',
                                'ja', 'ko', 'nl', 'no', 'pl', 'pt', 'ru', 'sl', 'sv', 'tr', 'zh'):
                lang = 'en'

            if title != None:
                data= self.get_page("%s/api/%s/series/%s/all/%s.xml" % (base_url, api_key, title, lang))

        elif type == 'seriesname':
            if title != None:
                data= self.get_page("%s/api/%s/series/%s/en.xml" % (base_url, api_key, title))

        else:
            data = None

        # be nice to the source site
        time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
        if data != None:
            return ET.fromstring(data.encode('utf-8'))

    def get_page(self, url):
        """
        Wrapper around get_page_internal to catch the
        timeout exception
        """
        try:
            fu = FetchURL(url, 'utf-8')
            config.max_fetches.acquire()
            fu.start()
            fu.join(config.opt_dict['global_timeout'])
            page = fu.result
            config.max_fetches.release()
            if (page == None) or (page.replace('[\n ]','') == ''):
                self.fail_count += 1
                with xml_output.output_lock:
                    xml_output.fail_count += 1
                    self.fail_count += 1

                return None

            else:
                self.fetch_count += 1
                return page

        except(urllib.URLError, socket.timeout):
            log('get_page timed out on (>%s s): %s\n' % (config.opt_dict['global_timeout'], url), 1, 1)
            self.fail_count += 1
            with xml_output.output_lock:
                xml_output.fail_count += 1

            config.max_fetches.release()
            return None

    def get_all_episodes(self, tid, lang='nl'):
        xml_output.program_cache.cache_request.put({'task':'query', 'parent': self, \
                'ep_by_id': {'tid': int(tid), 'sid': 0, 'eid': 0}})
        eps = self.cache_return.get(True)
        known_eps = {}
        for e in eps:
            if not (e['sid'],e['eid'],e['lang']) in known_eps.keys():
                known_eps[(e['sid'],e['eid'],e['lang'])] = []

            known_eps[(e['sid'],e['eid'],e['lang'])].append((e['title'],e['description']))

        try:
            eps = []
            langs = ('nl', 'en') if lang in ('nl', 'en') else (lang, 'nl', 'en')
            for l in langs:
                xmldata = self.query_ttvdb('episodes', tid, l)
                if xmldata == None:
                    # No data
                    continue

                for e in xmldata.findall('Episode'):
                    sid = e.findtext('SeasonNumber')
                    if sid == None or sid == '':
                        continue

                    eid = e.findtext('EpisodeNumber')
                    if eid == None or eid == '':
                        continue

                    title = e.findtext('EpisodeName')
                    if title == None or title == '':
                        title = 'Episode %s' % eid

                    airdate = e.findtext('FirstAired')

                    desc = e.findtext('Overview')
                    if desc == None:
                        desc == ''

                    if not (int(sid), int(eid), l) in known_eps.keys() or (title, desc) not in known_eps[(int(sid), int(eid), l)]:
                        eps.append({'tid': int(tid), 'sid': int(sid), 'eid': int(eid), 'title': title, 'airdate': airdate, 'lang': l, 'description': desc})

        except:
            log(['Error retreiving episodes from theTVDB.com\n', traceback.print_exc()])
            return

        xml_output.program_cache.cache_request.put({'task':'add', 'episode': eps})

    def get_ttvdb_id(self, title, lang='nl', search_db=True):
        get_id = False
        if search_db:
            xml_output.program_cache.cache_request.put({'task':'query_id', 'parent': self, 'ttvdb': {'title': title}})
            tid = self.cache_return.get(True)
            if tid != None:
                if ((datetime.date.today() - tid['tdate']).days > 30):
                    if (tid['tid'] == '' or int(tid['tid']) == 0):
                        # we try again to get an ID
                        get_id = True

                elif (tid['tid'] == '' or int(tid['tid']) == 0):
                    # Return failure
                    return 0

                else:
                    # We'll  use the episode info in the database
                    return tid

            else:
                # It's  not jet known
                get_id = True

        langs = ('nl', 'en') if lang in ('nl', 'en') else (lang, 'nl', 'en')
        if get_id or not search_db:
            # First we look for a known alias
            xml_output.program_cache.cache_request.put({'task':'query_id', 'parent': self, 'ttvdb_alias': {'alias': title}})
            alias = self.cache_return.get(True)
            series_name = title if alias == None else alias['title']
            try:
                xmldata = self.query_ttvdb('seriesid', series_name, lang)
                if xmldata == None:
                    # No data
                    xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb': {'tid': 0, 'title': series_name, 'langs': langs}})
                    return 0

                tid = xmldata.findtext('Series/seriesid')
                if tid == None:
                    # No data
                    xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb': {'tid': 0, 'title': series_name, 'langs': langs}})
                    return 0

                xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb': {'tid': int(tid), 'title': series_name, 'langs': langs}})
                #We look for aliasses
                xmldata = self.query_ttvdb('seriesid', series_name, 'all')
                if xmldata!= None:
                    alias_list = []
                    for s in xmldata.findall('Series'):
                        t = s.findtext('SeriesName')
                        if s.findtext('seriesid') == tid and t.strip().lower()  != series_name.strip().lower() and t not in alias_list:
                            alias_list.append(s.findtext('SeriesName'))

                    if len(alias_list) > 1:
                        xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb_alias': {'title':series_name, 'alias': alias_list}})

                    elif len(alias_list) == 1:
                        xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb_alias': {'title':series_name, 'alias': alias_list[0]}})

            except:
                log(['Error retreiving an ID from theTVdb.com', traceback.print_exc()])
                return 0

        # And we retreive the episodes
        self.get_all_episodes(tid, lang)
        return {'tid': int(tid), 'tdate': datetime.date.today(), 'title': series_name}

    def get_season_episode(self, parent = None, data = None):
        if config.opt_dict['disable_ttvdb'] or parent.opt_dict['disable_ttvdb']:
            return data

        if data == None:
            return

        if data['titel aflevering'][0:27].lower() == 'geen informatie beschikbaar':
            return data

        if parent != None and parent.group == 6:
            # We do not lookup for regional channels
            return data

        elif parent != None and parent.group == 4:
            tid = self.get_ttvdb_id(data['name'], 'de')

        elif parent != None and parent.group == 5:
            tid = self.get_ttvdb_id(data['name'], 'fr')

        else:
            tid = self.get_ttvdb_id(data['name'])

        if tid == None or tid == 0:
            if parent != None:
                parent.update_counter('ttvdb_fail')

            log("  No ttvdb id for '%s' on channel %s\n" % (data['name'], data['channel']), 128)
            return data

        # First we just look for a matching subtitle
        tid = tid['tid']
        xml_output.program_cache.cache_request.put({'task':'query', 'parent': self, \
                'ep_by_title': {'tid': tid, 'title': data['titel aflevering']}})
        eid = self.cache_return.get(True)
        if eid != None:
            if parent != None:
                parent.update_counter('ttvdb')

            data['season'] = eid['sid']
            data['episode'] = eid['eid']
            if isinstance(eid['airdate'], (datetime.date)):
                data['airdate'] = eid['airdate']

            log('ttvdb  lookup for %s: %s\n' % (data['name'], data['titel aflevering']), 24)
            return data

        # Now we get a list of episodes matching what we already know and compare with confusing characters removed
        xml_output.program_cache.cache_request.put({'task':'query', 'parent': self, \
                'ep_by_id': {'tid': tid, 'sid': data['season'], 'eid': data['episode']}})
        eps = self.cache_return.get(True)
        subt = re.sub('[-,. ]', '', xml_output.remove_accents(data['titel aflevering']).lower())
        ep_dict = {}
        ep_list = []
        for ep in eps:
            s = re.sub('[-,. ]', '', xml_output.remove_accents(ep['title']).lower())
            ep_list.append(s)
            ep_dict[s] = {'sid': ep['sid'], 'eid': ep['eid'], 'airdate': ep['airdate'], 'title': ep['title']}
            if s == subt:
                if parent != None:
                    parent.update_counter('ttvdb')

                data['titel aflevering'] = ep['title']
                data['season'] = ep['sid']
                data['episode'] = ep['eid']
                if isinstance(ep['airdate'], (datetime.date)):
                    data['airdate'] = ep['airdate']

                log('ttvdb  lookup for %s: %s\n' % (data['name'], data['titel aflevering']), 24)
                return data

        # And finally we try a difflib match
        match_list = difflib.get_close_matches(subt, ep_list, 1, 0.7)
        if len(match_list) > 0:
            if parent != None:
                parent.update_counter('ttvdb')

            ep = ep_dict[match_list[0]]
            data['titel aflevering'] = ep['title']
            data['season'] = ep['sid']
            data['episode'] = ep['eid']
            if isinstance(ep['airdate'], (datetime.date)):
                data['airdate'] = ep['airdate']

            log('ttvdb  lookup for %s: %s\n' % (data['name'], data['titel aflevering']), 24)
            return data

        if parent != None:
            parent.update_counter('ttvdb_fail')

        log("ttvdb failure for '%s': '%s' on channel %s\n" % (data['name'], data['titel aflevering'], data['channel']), 128)
        return data

    def check_ttvdb_title(self, series_name, lang='nl'):
        if config.opt_dict['disable_ttvdb']:
            return

        langs = ['nl', 'en', 'de', 'fr']
        if lang in ('cs', 'da', 'el', 'es', 'fi', 'he', 'hr', 'hu', 'it',
                                'ja', 'ko', 'no', 'pl', 'pt', 'ru', 'sl', 'sv', 'tr', 'zh'):
            langs.append(lang)

        # Check if a record exists
        xml_output.program_cache.cache_request.put({'task':'query_id', 'parent': self, 'ttvdb': {'title': series_name}})
        tid = self.cache_return.get(True)
        if tid != None:
            print('The series "%s" is already saved under ttvdbID: %s -> %s' % (series_name,  tid['tid'], tid['title']))
            print('    for the languages: %s\n' % tid['langs'])
            old_tid = int(tid['tid'])
            for l in tid['langs']:
                if l not in langs:
                    langs.append(lang)

        else:
            print('The series "%s" is not jet known!\n' % (series_name))
            old_tid = -1

        try:
            xmldata = self.query_ttvdb('seriesid', series_name, lang)
            if xmldata == None or xmldata.find('Series') == None:
                print('No match for %s is found on theTVDB.com' % series_name)
                return

            series_list = []
            for s in xmldata.findall('Series'):
                if not {'sid': s.findtext('seriesid'), 'name': s.findtext('SeriesName')} in series_list:
                    series_list.append({'sid': s.findtext('seriesid'), 'name': s.findtext('SeriesName')})

            print("theTVDB Search Results:")
            for index in range(len(series_list)):
                print("%3.0f -> %9.0f: %s" % (index+1, int(series_list[index]['sid']), series_list[index]['name']))

            # Ask to select the right one
            while True:
                try:
                    print("Enter choice (first number, q to abort):")
                    ans = raw_input()
                    selected_id = int(ans)-1
                    if 0 <= selected_id < len(series_list):
                        break

                except ValueError:
                    if ans.lower() == "q":
                        return

            tid = series_list[selected_id]
            # Get the English name
            xmldata = self.query_ttvdb('seriesname', tid['sid'])
            ename = xmldata.findtext('Series/SeriesName')
            if ename == None:
                ename = tid['name']

            if old_tid != int(tid['sid']):
                print('Removing old instance')
                xml_output.program_cache.cache_request.put({'task':'delete', 'ttvdb': {'tid': old_tid}})

            xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb': {'tid': int(tid['sid']), 'title': ename, 'langs': langs}})
            aliasses = []
            if ename.lower() != tid['name'].lower():
                aliasses.append(tid['name'])

            if ename.lower() != series_name.lower() and tid['name'].lower() != series_name.lower():
                aliasses.append(series_name)

            if len(aliasses) > 0:
                # Add an alias record
                xml_output.program_cache.cache_request.put({'task':'add', 'ttvdb_alias': {'tid': int(tid['sid']), 'title': ename, 'alias': aliasses}})
                if len(aliasses) == 2:
                    print('Adding "%s" under aliasses "%s" and "%s" as ttvdbID: %s to the database for lookups!' \
                                % (ename, aliasses[0], aliasses[1],  tid['sid']))

                else:
                    print('Adding "%s" under alias "%s" as ttvdbID: %s to the database for lookups!' \
                                % (ename, aliasses[0],  tid['sid']))

            else:
                print('Adding "%s" ttvdbID: %s to the database for lookups!' % (ename,  tid['sid']))

        except:
            traceback.print_exc()
            return

        self.get_all_episodes(int(tid['sid']), langs)

# end theTVDB

class FetchData(Thread):
    """
    Generic Class to fetch the data

    The output is a list of programming in order where each row
    contains a dictionary with program information.

    It runs as a separate thread for every source
    """
    current_date = datetime.datetime.now(CET_CEST).toordinal()

    def __init__(self, proc_id, source, detail_id, detail_url = '', isjson = False, detail_check = '', detail_processor = False):
        tname = ('source-%s'% source).encode('utf-8', 'replace')
        Thread.__init__(self, name = tname)
        # Flag to stop the thread
        self.quit = False
        self.ready = False
        self.state = 0
        self.active = True
        self.isjson = isjson
        # The ID of the source
        self.proc_id = proc_id
        # The Name of the source
        self.source = source
        # The dict name of the details etc.
        self.detail_id = detail_id
        self.detail_url = detail_url
        self.detail_check = detail_check
        self.detail_processor = detail_processor
        self.detail_request = Queue()
        self.cache_return = Queue()
        self.source_lock = Lock()

        self.all_channels = {}
        self.channels = {}
        self.chanids = {}
        self.all_chanids = {}
        self.channel_loaded = {}
        self.day_loaded = {}
        self.program_data = {}
        self.program_by_id = {}
        self.chan_count = 0
        self.base_count = 0
        self.detail_count = 0
        self.fail_count = 0
        self.fetch_string_parts = re.compile("(.*?[.?!:]+ |.*?\Z)")

    def run(self):
        """The grabing thread"""
        self.testlist = ((1, 0), (9, 0,), (1, 9))
        self.lastrequest = None
        def check_queue():
            # If the queue is empty
            if self.detail_request.empty():
                time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                # and if we are not tvgids.nl we wait for followup requests from other failures failures
                for q_no in self.testlist:
                    if (self.proc_id == q_no[0]) and xml_output.channelsource[q_no[1]].is_alive():
                        return 0

                # Check if all channels are ready
                for channel in config.channels.values():
                    if channel.is_alive() and not channel.detail_data.is_set():
                        break

                # All channels are ready, so if there is nothing in the queue
                else:
                    self.ready = True
                    return -1

                # OK we have been sitting idle for 30 minutes, So we tell all channels they won get anything more!
                if (datetime.datetime.now() - self.lastrequest).total_seconds() > idle_timeout:
                    if self.proc_id == 1:
                        for channel in config.channels.values():
                            if channel.is_alive() and not channel.detail_data.is_set():
                                channel.detail_data.set()
                                log('Channel %s seems to be waiting for %s lost detail requests from %s.\nSetting it to ready\n' % \
                                    (channel.chan_name, channel.counters['fetch'][1], self.source))

                    self.ready = True
                    return -1

                else:
                    return 0

            self.lastrequest = datetime.datetime.now()
            try:
                return self.detail_request.get()

            except Empty:
                return 0

        def check_ttvdb(tdict, parent):
            if not (config.opt_dict['disable_ttvdb'] or parent.opt_dict['disable_ttvdb']) and \
              tdict['genre'].lower() == u'serie/soap' and tdict['titel aflevering'] != '' and tdict['season'] == 0:
                # We do a ttvdb lookup
                parent.update_counter('fetch', -1)
                xml_output.ttvdb.detail_request.put({'tdict':tdict, 'parent': parent, 'task': 'update_ep_info'})

            else:
                with parent.channel_lock:
                    parent.detailed_programs.append(tdict)

        def check_other_sources(tdict, cache_id, logstring, parent):
            cached_program = None
            if (self.proc_id in (0, 9)) and (cache_id != None):
                # Check the cache again
                xml_output.program_cache.cache_request.put({'task':'query', 'parent': self, 'pid': cache_id})
                cached_program = self.cache_return.get(True)
                if cached_program == 'quit':
                    self.ready = True
                    return -1

            for q_no in self.testlist:
                if cached_program != None and self.proc_id == q_no[1] and \
                  cached_program[xml_output.channelsource[q_no[0]].detail_check]:
                    log(u'      [cached] %s:(%3.0f%%) %s\n' % (parent.chan_name, parent.get_counter(), logstring), 8, 1)
                    tdict= parent.use_cache(tdict, cached_program)
                    parent.update_counter('cache')
                    parent.update_counter('fetch', self.proc_id, False)
                    check_ttvdb(tdict, parent)
                    return 0

                # If there is an url we'll try tvgids.tv
                elif self.proc_id == q_no[1] and xml_output.channelsource[q_no[0]].detail_processor and \
                  q_no[0] not in parent.opt_dict['disable_detail_source'] and \
                  tdict['detail_url'][q_no[0]] != '':
                    xml_output.channelsource[q_no[0]].detail_request.put({'tdict':tdict, 'cache_id': cache_id, 'logstring': logstring, 'parent': parent, 'last_one': False})
                    parent.update_counter('fetch', q_no[0])
                    parent.update_counter('fetch', self.proc_id, False)
                    return 0

        # First some generic initiation that couldn't be done earlier in __init__
        # Specifics can be done in init_channels and init_json which are called here
        tdict = self.checkout_program_dict()
        idle_timeout = 1800
        try:
            # Check if the source is not deactivated and if so set them all loaded
            if self.proc_id in config.opt_dict['disable_source']:
                self.set_ready()
                self.ready = True

            else:
                self.state = 1
                self.day_loaded[0] = {}
                for day in range( config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days'])):
                    self.day_loaded[0][day] = False

                for c in config.channels.values():
                    channelid = c.get_source_id(self.proc_id)
                    self.channel_loaded[channelid] = False
                    self.day_loaded[channelid] ={}
                    for day in range( config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days'])):
                        self.day_loaded[channelid][day] = False

                self.init_channels()
                self.init_json()
                # Load and proccess al the program pages
                try:
                    self.load_pages()

                except:
                    self.fail_count += 1
                    log(['Fatal Error processing the basepages from %s\n' % (self.source), \
                        'Setting them all to being loaded, to let the other sources finish the job\n', traceback.print_exc()], 0)
                    self.set_ready()

                if config.write_info_files:
                    infofiles.check_new_channels(self)

            if self.detail_processor and  not self.proc_id in config.opt_dict['disable_detail_source']:
                # We process detail requests, so we loop till we are finished
                self.state = 2
                self.cookyblock = False
                self.lastrequest = datetime.datetime.now()
                while True:
                    if self.quit:
                        self.ready = True
                        break

                    queue_val = check_queue()
                    if queue_val == -1:
                        break

                    if queue_val == 0 or not isinstance(queue_val, dict):
                        continue

                    tdict = queue_val
                    parent = tdict['parent']
                    # Is this the closing item for the channel?
                    if ('last_one' in tdict) and tdict['last_one']:
                        if self.proc_id == 0 and parent.counters['fetch'][9] > 0:
                            xml_output.channelsource[9].detail_request.put(tdict)

                        elif self.proc_id == 9 and parent.counters['fetch'][1] > 0:
                            xml_output.channelsource[1].detail_request.put(tdict)

                        elif parent.counters['fetch'][-1] > 0 and not (config.opt_dict['disable_ttvdb'] or parent.opt_dict['disable_ttvdb']):
                            xml_output.ttvdb.detail_request.put({'task': 'last_one', 'parent': parent})

                        else:
                            parent.detail_data.set()

                        continue

                    cache_id = tdict['cache_id']
                    logstring = tdict['logstring']
                    tdict = tdict['tdict']
                    # be nice to the source site
                    time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                    # First if the cookyblock is not encountered try the html detail page (only tvgids.nl, the others only have html)
                    if not self.cookyblock:
                        try:
                            detailed_program = self.load_detailpage(tdict)
                            if detailed_program == None:
                                self.fail_count += 1

                        except:
                            detailed_program = None
                            self.fail_count += 1
                            log(['Error processing the detailpage: %s\n' % (tdict['detail_url'][self.proc_id]), traceback.print_exc()], 1)

                    else:
                        detailed_program = None

                    # It failed! If this is tvgids.nl we check the json page
                    if detailed_program == None and (self.proc_id == 0):
                        try:
                            detailed_program = self.load_json_detailpage(tdict)
                            if detailed_program == None:
                                self.fail_count += 1

                        except:
                            detailed_program = None
                            self.fail_count += 1
                            log(['Error processing the json detailpage: http://www.tvgids.nl/json/lists/program.php?id=%s\n' \
                                % tdict['prog_ID'][self.proc_id][3:], traceback.print_exc()], 1)

                    # It failed!
                    if detailed_program == None:
                        # If this is tvgids.nl and there is an url we'll try tvgids.tv, but first check the cache again
                        if self.proc_id == 1:
                            log(u'[fetch failed or timed out] %s:(%3.0f%%) %s\n' % (parent.chan_name, parent.get_counter(), logstring), 8, 1)
                            parent.update_counter('fail')
                            parent.update_counter('fetch', self.proc_id, False)
                            check_ttvdb(tdict, parent)
                            continue

                        else:
                            ret_val = check_other_sources(tdict, cache_id, logstring, parent)
                            if ret_val == -1:
                                break

                            else:
                                continue

                    # Success
                    else:
                        # If this is the prefered description source for this channel, set its value
                        if config.channels[detailed_program['channelid']].opt_dict['prefered_description'] == self.proc_id:
                            detailed_program['prefered description'] = detailed_program['description']

                        detailed_program[xml_output.channelsource[self.proc_id].detail_check] = True
                        detailed_program['ID'] = detailed_program['prog_ID'][self.proc_id]
                        check_ttvdb(detailed_program, parent)
                        if self.proc_id == 0:
                            log(u'[normal fetch] %s:(%3.0f%%) %s\n' % (parent.chan_name, parent.get_counter(), logstring), 8, 1)

                        elif self.proc_id == 1:
                            log(u'   [.tv fetch] %s:(%3.0f%%) %s\n' % (parent.chan_name, parent.get_counter(), logstring), 8, 1)

                        elif self.proc_id == 9:
                            log(u' [primo fetch] %s:(%3.0f%%) %s\n' % (parent.chan_name, parent.get_counter(), logstring), 8, 1)

                        parent.update_counter('fetched', self.proc_id)
                        parent.update_counter('fetch', self.proc_id, False)
                        self.detail_count += 1

                        # do not cache programming that is unknown at the time of fetching.
                        if tdict['name'].lower() != 'onbekend':
                            xml_output.program_cache.cache_request.put({'task':'add', 'program': xml_output.channelsource[0].checkout_program_dict(detailed_program)})

            else:
                self.ready = True
                self.state = 0

        except:
            log_list = ['\n', 'An unexpected error has occured in the %s thread\n' %  (self.source)]
            if tdict['detail_url'][self.proc_id] == '':
                log_list.append('While fetching the base pages\n')

            else:
                log_list.append('The current detail url is: %s\n' % (tdict['detail_url'][self.proc_id]))

            log_list.extend([traceback.print_exc(), '\n', \
                'If you want assistence, please attach your configuration and log files!\n', \
                '     %s\n' % (config.config_file), '     %s\n' % (config.log_file)])

            log(log_list,0)

            self.ready = True
            self.state = 0
            for source in xml_output.channelsource.values():
                if source.is_alive():
                    source.cache_return.put('quit')
                    source.quit = True

            for channel in config.channels.values():
                if channel.is_alive():
                    channel.cache_return.put('quit')
                    channel.quit = True

            return(98)

    # Dummys to be filled in by the sub-Classes
    def init_channels(self):
        """The specific initiation code before starting with grabbing"""
        self.init_channel_source_ids()

    def init_json(self):
        """The specific initiation code if the source is json before starting with grabbing"""
        if not self.isjson:
            return

        # Define here the json structure if it's not a flat list of program dicts
        # self.jsondata = {<name>: ['listname':<list>,'keyname':<key>,'valuename':<vname>}
        # self.jsondict[<list>][<key-in-json_by_id[id][tdict['keyname']] >][<vname>] = value
        self.json_by_id = {}
        self.jsondata = {}
        self.jsondict = {}

    def get_url(self):
        """return the several url's for ordinairy, detail and channel info"""
        pass

    def get_channels(self):
        """The code for the retreiving a list of suppoted channels"""
        pass

    def load_pages(self):
        """The code for the actual Grabbing and dataprocessing"""
        if len(self.channels) == 0 :
            return

        else:
            self.set_ready()

    def load_detailpage(self, tdict):
        """The code for retreiving and processing a detail page"""
        return tdict

    # Helper functions
    def init_channel_source_ids(self):
        self.source_regexes = {}
        if self.proc_id in config.source_regexes.keys() and isinstance(config.source_regexes[self.proc_id], list):
            for i in range(len(config.source_regexes[self.proc_id])):
                sre = config.source_regexes[self.proc_id][i]
                if isinstance(sre, (str, unicode)) and len(sre) > 0:
                    self.source_regexes[i] = re.compile(sre)

                elif isinstance(sre, list) and len(sre) > 0 and isinstance(sre[0], (str, unicode)) and len(sre[0]) > 0:
                    if len(sre) > 1 and sre[1] == 1:
                        self.source_regexes[i] = re.compile(sre[0],re.DOTALL)

                    else:
                        self.source_regexes[i] = re.compile(sre[0])

        self.text_replace = config.text_replace[self.proc_id] if self.proc_id in config.text_replace.keys() else []
        self.unquote_html = config.unquote_html[self.proc_id] if self.proc_id in config.unquote_html.keys() else []
        for chanid, channel in config.channels.iteritems():
            # Is the channel active and this source for the channel not disabled
            if channel.active and not self.proc_id in channel.opt_dict['disable_source']:
                # Is there a sourceid for this channel
                if channel.get_source_id(self.proc_id) != '':
                    channelid = channel.get_source_id(self.proc_id)
                    self.program_data[channelid] = []
                    # Unless it is in empty channels we add it else set it ready
                    if channelid in config.empty_channels[self.proc_id]:
                        self.channel_loaded[channelid] = True
                        channel.source_data[self.proc_id].set()

                    else:
                        self.channels[chanid] = channelid
                        if not channelid in self.all_chanids.keys():
                            self.all_chanids[channelid] = [chanid]

                        elif not chanid in self.all_chanids[channelid]:
                            self.all_chanids[channelid].append(chanid)

                # Does the channel have child channels
                if chanid in config.combined_channels.keys():
                    # Then see if any of the childs has a sourceid for this source and does not have this source disabled
                    for c in config.combined_channels[chanid]:
                        if c['chanid'] in config.channels.keys():
                            vchannel = config.channels[c['chanid']]
                            if vchannel.get_source_id(self.proc_id) != '':
                                channelid = vchannel.get_source_id(self.proc_id)
                                self.program_data[channelid] = []
                                # Unless it is in empty channels we add and mark it as a child else set it ready
                                if channelid in config.empty_channels[self.proc_id] or self.proc_id in vchannel.opt_dict['disable_source']:
                                    self.channel_loaded[channelid] = True
                                    vchannel.source_data[self.proc_id].set()

                                else:
                                    vchannel.is_child = True
                                    self.channels[c['chanid']] = channelid
                                    if not channelid in self.all_chanids.keys():
                                        self.all_chanids[channelid] = [c['chanid']]

                                    elif not c['chanid'] in self.all_chanids[channelid]:
                                        self.all_chanids[channelid].append(c['chanid'])

        for channelid, chanidlist in self.all_chanids.items():
            if len(chanidlist) == 1:
                self.chanids[channelid] = chanidlist[0]

            else:
                for chanid in chanidlist:
                    if not config.channels[chanid].is_virtual_sub:
                        self.chanids[channelid] = chanid
                        break

                else:
                    self.chanids[channelid] = chanidlist[0]

    def checkout_program_dict(self, tdict = None):
        """
        Checkout a given dict for invalid values or
        returnsa default empty dict for storing program info
        """
        # ID generation
        # u'nl-%s' % (item['db_id'])
        # u'tv-%s' % tdict['detail_url'][self.proc_id].split('/')[5]
        # u'be-%s' % tdict['detail_url'][self.proc_id].split('/')[5]
        # u'%s-%s' % (channel,  item['unixtime'])

        # source                        tvgids.nl    tvgids.tv   rtl.nl     teveblad.be
        # dagen                              4                 14        14              7
        # channelid                    j jd hd
        # channel                                            h hd         j             h hd
        # offset                          j                     h
        # unixtime                                                           j
        # datum                           jd hd              hd                        h hd
        # start-time                    j jd                h hd                        h hd
        # stop-time                     j jd hd
        # clumpidx
        # tvgids-fetched

        # name                           j jd hd           h hd         j              h hd
        # titel aflevering                                                  j              h hd
        # season                                                              j              h hd
        # episode                                                             j              h hd
        # description                   jd hd               hd         j              h hd
        # rerun                                hd                            j              h hd
        # jaar van premiere             hd              hd                        h hd
        # airdate
        # country                                                                           h hd
        # originaltitle                                                                     h hd

        # genre                          j jd hd               hd                       h hd
        # subgenre                     j

        # ID                               j jd hd            h hd          j             h hd
        # url                              j                      h hd                        h hd
        # official website                                    hd
        # kijkwijzer/nicam        j jd hd                hd         j

        # credits
        #       presentatie             jd hd                hd
        #       acteurs                   jd hd                hd                          hd
        #       regisseur                jd hd                hd
        #       scenario                     hd
        #       componist                  hd

        # stereo                              hd
        # dolby                                                                              h hd
        # teletekst                          hd
        # video
        #       HD                            hd                                            h hd
        #       breedbeeld                hd
        #       blackwhite                hd

        self.text_values = ('channelid', 'source', 'channel', 'unixtime', 'prefered description', \
              'clumpidx', 'name', 'titel aflevering', 'description', 'jaar van premiere', \
              'originaltitle', 'subgenre', 'ID', 'merge-source', 'infourl', 'audio', 'star-rating', \
              'country', 'omroep')
        self.datetime_values = ('start-time', 'stop-time')
        self.date_values = ('airdate', )
        self.bool_values = ('tvgids-fetched', 'tvgidstv-fetched', 'primo-fetched', 'rerun', 'teletekst', \
              'new', 'last-chance', 'premiere')
        self.num_values = ('season', 'episode', 'offset')
        self.dict_values = ('credits', 'video')
        self.source_values = ('prog_ID', 'detail_url')
        self.list_values = ('kijkwijzer', )
        self.video_values = ('HD', 'breedbeeld', 'blackwhite')

        if tdict == None:
            tdict = {}

        for key in self.text_values:
            if not key in tdict.keys() or tdict[key] == None:
                tdict[key] = u''

            try:
                if type(tdict[key]) != unicode:
                    tdict[key] = unicode(tdict[key])

            except UnicodeError:
                tdict[key] = u''

        for key in self.date_values:
            if not key in tdict.keys() or tdict[key] == None:
                tdict[key] = u''

        for key in self.datetime_values:
            if not key in tdict.keys() or tdict[key] == None:
                tdict[key] = u''

        if not 'genre' in tdict.keys() or tdict['genre'] == None or tdict['genre'] == '':
            tdict['genre'] = u'overige'

        for key in self.bool_values:
            if not key in tdict.keys() or tdict[key] != True:
                tdict[key] = False

        for key in self.num_values:
            if not key in tdict.keys() or tdict[key] == None or tdict[key] == '':
                tdict[key] = 0

        for key in self.dict_values:
            if not key in tdict.keys() or not isinstance(tdict[key], dict):
                tdict[key] = {}

        for key in self.source_values:
            if not key in tdict.keys() or not isinstance(tdict[key], dict):
                tdict[key] = {}
                for s in  xml_output.source_order:
                    if not s in tdict[key] or tdict[key][s] == None:
                        tdict[key][s] = u''

                    try:
                        if type(tdict[key][s]) != unicode:
                            tdict[key][s] = unicode(tdict[key][s])

                    except UnicodeError:
                        tdict[key][s] = u''

        for key in self.list_values:
            if not key in tdict.keys() or not isinstance(tdict[key], list):
                tdict[key] = []

        for subkey in tdict['credits'].keys():
            if  tdict['credits'][subkey] == None:
                tdict['credits'][subkey] = []

            for i, item in enumerate(tdict['credits'][subkey]):
                try:
                    if type(item) != unicode:
                        tdict['credits'][subkey][i] = unicode(item)

                except UnicodeError:
                    tdict['credits'][subkey][i] = u''

        for subkey in self.video_values:
            if not subkey in tdict['video'].keys() or  tdict['video'][subkey] != True:
                tdict['video'][subkey] = False

        return tdict

    def unescape(self, text):
        # Removes HTML or XML character references and entities from a text string.
        # source: http://effbot.org/zone/re-sub.htm#unescape-html
        #
        # @param text The HTML (or XML) source text.
        # @return The plain text, as a Unicode string

        def fixup(m):
            text = m.group(0)
            if text[:2] == "&#":
                # character reference
                try:
                    if text[:3] == "&#x":
                        return unichr(int(text[3:-1], 16))

                    else:
                        return unichr(int(text[2:-1]))

                except ValueError:
                    pass

            else:
                # named entity
                try:
                    text = unichr(name2codepoint[text[1:-1]])

                except KeyError:
                    pass

            return text # leave as is

        text = re.sub("…", "...", text)
        text = re.sub("‘", "'", text)
        text = re.sub("’", "'", text)
        return re.sub("&#?\w+;", fixup, text)

    def clean_html(self, data):
        """Process characters that interfere with ElementTree processing"""
        if data == None:
            return

        data = re.sub('&quot;', ' emprsant quot;', data)
        data = re.sub('&lt;', ' emprsant lt;', data)
        data = re.sub('&gt;', ' emprsant gt;', data)
        data = self.unescape(data)
        data = re.sub('&raquo<', '<', data)
        data = re.sub('&', ' emprsant ', data)
        return data

    def empersant(self, data):
        if data == None:
            return u''

        data = re.sub(' emprsant ', '&', data)
        data = re.sub('emprsant ', '&', data)
        data = re.sub(' emprsant', '&', data)
        data = re.sub('emprsant', '&', data)
        data = re.sub('&quot;', '"', data)
        data = re.sub('&lt;', '<', data)
        data = re.sub('&gt;', '>', data)
        if type(data) != unicode:
            return unicode(data)

        return data

    def add_endtimes(self, channelid, date_switch = 6):
        """
        For the sites that only give start times, add the next starttime as endtime
        date_switch is the time we asume the last program will end if started before that time
        else  we assume next midnight
        """
        if len(self.program_data[channelid]) > 0:
            for i, tdict in enumerate(self.program_data[channelid]):
                if i > 0 and type(tdict['start-time']) == datetime.datetime:
                    try:
                        if not type(self.program_data[channelid][i-1]['stop-time']) == datetime.datetime:
                            self.program_data[channelid][i-1]['stop-time'] =  tdict['start-time']

                    except:
                        pass

            # And one for the last program
            prog_date = datetime.date.fromordinal(self.current_date + self.program_data[channelid][-1]['offset'])
            if not type(self.program_data[channelid][-1]['stop-time']) == datetime.datetime:
                if int(self.program_data[channelid][-1]['start-time'].strftime('%H')) < date_switch:
                    self.program_data[channelid][-1]['stop-time'] = datetime.datetime.combine(prog_date, datetime.time(date_switch, 0,0 ,0 ,CET_CEST))

                else:
                    self.program_data[channelid][-1]['stop-time'] = datetime.datetime.combine(prog_date, datetime.time(23, 59,0 ,0 ,CET_CEST))

            # remove programs that end when they start
            for tdict in self.program_data[channelid][:]:
                if tdict['start-time'] == tdict['stop-time']:
                    self.program_data[channelid].remove(tdict)

    def get_datestamp(self, offset=0):
        tsnu = (int(time.time()/86400)) * 86400
        day =  datetime.datetime.fromtimestamp(tsnu)
        datenu = int(tsnu - CET_CEST.utcoffset(day).total_seconds())
        if time.time() -  datenu > 86400:
            datenu += 86400

        return datenu + offset * 86400

    def get_offset(self, date):
        """Return the offset from today"""
        return int(date.toordinal() -  self.current_date)

    def check_title_name(self, program):
        """
        Process Title names on Grouping issues and apply the rename table
        Return the updated Progam dict
        """
        ptitle = program['name']
        psubtitle = program['titel aflevering']
        if  ptitle == None or ptitle == '':
            return program

        if re.sub('[-,. ]', '', ptitle) == re.sub('[-,. ]', '', psubtitle):
            program['titel aflevering'] = ''
            psubtitle = ''

        # Remove a groupname if in the list
        for group in config.groupnameremove:
            if (len(ptitle) > len(group) + 3) and (ptitle[0:len(group)].lower() == group):
                p = ptitle.split(':')
                if len(p) >1:
                    log('Removing \"%s\" from \"%s\"\n' %  (group, ptitle), 64)
                    if config.write_info_files:
                        infofiles.addto_detail_list(unicode('Group removing = \"%s\" from \"%s\"' %  (group, ptitle)))

                    ptitle = "".join(p[1:]).strip()

        # Fixing subtitle both named and added to the title
        if ptitle.lower() == psubtitle.lower() and program['genre'] != 'serie/soap':
            psubtitle = ''
        if  (psubtitle != '') and (len(ptitle) > len(psubtitle)):
            lentitle = len(ptitle) - len(psubtitle)
            if psubtitle.lower().strip() == ptitle[lentitle:].lower().strip():
                ptitle = ptitle[0:lentitle].strip()
                if (ptitle[-1] == ':') or (ptitle[-1] == '-'):
                    ptitle = ptitle[0:(len(ptitle) - 1)].strip()

        # And the other way around
        elif  (psubtitle != '') and (len(ptitle) < len(psubtitle)):
            lentitle = len(ptitle.strip())
            if ptitle.lower().strip() == psubtitle[0:lentitle].lower().strip():
                psubtitle = psubtitle[lentitle:].strip()
                if (psubtitle[0:1] == ':') or (psubtitle[0:1] == '-'):
                    psubtitle = psubtitle[1:].strip()

        # Check the Title rename list
        if ptitle.lower() in config.titlerename:
            log('Renaming %s to %s\n' % (ptitle, config.titlerename[ptitle.lower()]), 64)
            if config.write_info_files:
                infofiles.addto_detail_list(unicode('Title renaming %s to %s\n' % (ptitle, config.titlerename[ptitle.lower()])))

            ptitle = config.titlerename[ptitle.lower()]

        program['name'] = ptitle
        program['titel aflevering'] = psubtitle
        return program

    def get_string_parts(self, sstring, header_items = None):
        if not isinstance(header_items, (list, tuple)):
            header_items = []

        test_items = []
        for hi in header_items:
            if isinstance(hi, (str, unicode)):
                test_items.append((hi.lower(), hi))

            elif isinstance(hi, (list, tuple)):
                if len(hi) > 0 and isinstance(hi[0], (str, unicode)):
                    hi0 = hi[0].lower()
                    if len(hi) > 1 and isinstance(hi[1], (str, unicode)):
                        hi1 = hi[1]

                    else:
                        hi1 = hi[0]

                    test_items.append((hi0, hi1))

        string_parts = self.fetch_string_parts.findall(sstring)
        string_items = {}
        act_item = 'start'
        string_items[act_item] = []
        for dp in string_parts:
            if dp.strip() == '':
                continue

            if dp.strip()[-1] == ':':
                act_item = dp.strip()[0:-1].lower()
                string_items[act_item] = []

            else:
                for ti in test_items:
                    if dp.strip().lower()[0:len(ti[0])] == ti[0]:
                        act_item = ti[1]
                        string_items[act_item] = []
                        string_items[act_item].append(dp[len(ti[0]):].strip())
                        break

                else:
                    string_items[act_item].append(dp.strip())

        return string_items

    def filter_description(self,ETitem, ETfind, tdict):
        """
        Filter the description as found on the detailpages for relevant info
        and return the adapted program dict
        """
        alinea = []
        atype = []
        aheader = []

        def format_text(text):
            newtext = self.empersant(text.strip())
            newtext = re.sub('\n','', newtext)
            newtext = re.sub(' +?',' ', newtext)
            return newtext

        pcount = 0
        # We scan every alinea of the description
        for p in ETitem.findall(ETfind):
            aheader.append('')
            atype.append('')
            # Check if it has a class like 'summary'
            if p.get('class') == None:
                atype[pcount] = u''

            else:
                atype[pcount] = self.empersant(p.get('class')).strip()
                if config.write_info_files:
                    infofiles.addto_detail_list(u'%s descriptionattribute => class: %s' % (self.source, p.get('class').strip()))

            content = ''
            # Add the alinea text
            if (p.text != None) and (p.text != ''):
                content = format_text(p.text) + u' '

            # Check for further tags like <i>talic and their following text
            for d in list(p.iter()):
                if d.tag == 'span' and atype[pcount] == 'summary':
                    # On tvgids.nl, this is the genre
                    pass

                elif d.tag in ('br', 'img'):
                    # Linebreaks don't contain text and images we ignore and don't count
                    # But we want the tail text
                    pass

                elif (d.tag == 'p') or (d.text != None and 'gesponsorde link' in d.text.lower()):
                    # We don't want those
                    continue

                elif (d.text != None) and (d.text != ''):
                    if d.tag == 'strong':
                        # The first is an alineaheader
                        # or if it's the first alinea the subgenre or something like it
                        if content.strip() == '':
                            aheader[pcount] = format_text(d.text)
                        else:
                            aheader[pcount] = u''
                            content = content + format_text(d.text) + u' '

                    elif d.tag in ('i', 'em', 'a', 'b'):
                        content = content + format_text(d.text) + u' '

                    else:
                        # Unknown tag we just check for text
                        content = content + format_text(d.text) + u' '
                        if config.write_info_files:
                            infofiles.addto_detail_list(unicode('new '+ self.source+' descriptiontag => ' + \
                                                    unicode(d.tag.strip()) + ': ' + unicode(d.text.strip())))

                # and we add the text inbetween the tags
                if (d.tail != None) and d.tail != '' :
                    content = content + format_text(d.tail) + u' '

            content = content.strip()

            if re.search('geen detailgegevens be(?:kend|schikbaar)', content.lower()) \
              or (content.lower() == '') or (content.lower() == 'none'):
                # No text so unless it's the first alinea, we ignore it
                if pcount == 0:
                    alinea.append('')
                    pcount +=1
                else:
                    continue

            else:
                alinea.append(content)
                pcount +=1

        # Now we decide what to return
        if len(alinea) > 0:
            for i, v in enumerate(atype):
                if v == 'summary' and alinea[i] != '':
                    # We just go for the summary
                    description = alinea[i]
                    break

            else:
                if len(alinea) ==1:
                    # Only ony alinea
                    description = alinea[0]

                elif len(alinea) == 2 and alinea[0] == '':
                    # we go for the second alinea
                    description = alinea[1]

                # Now it gets tricky for most of the time one is general and the other is specific
                # We check if one is contained in the other
                elif len(alinea) == 2 and alinea[1] in alinea[0] :
                     description = alinea[0]

                elif len(alinea) == 2 and alinea[0] in alinea[1] :
                     description = alinea[1]

                # So we return everything
                else:
                    content = ''
                    for p in alinea:
                        if p != '':
                            content = '%s%s ' % (content, p)
                    description = content.strip()

                    if config.write_info_files:
                        strdesc = ''
                        for p in alinea:
                            strdesc = strdesc + '    <p>%s</p>\n' % p

                        strdesc = '  <div start="' + tdict['start-time'].strftime('%d %b %H:%M') + \
                                                    '" name="' + tdict['name'] + '">\n' + strdesc + '  </div>'
                        if config.write_info_files:
                            infofiles.addto_raw_string(strdesc)

            # We check to not ovrwrite an already present longer description
            if description > tdict['description']:
                tdict['description'] = description

            # use the first header as subgenre, if not already present
            if tdict['subgenre'] == '' and aheader[0] != '':
                tdict['subgenre']  = aheader[0]

        return tdict

    def check_text_subs(self, page):
        def unquote(matchobj):
            rval = matchobj.group(0)
            try:
                for mg in matchobj.groups():
                    if mg == None:
                        continue

                    tt = mg
                    for s in (('"', '&quot;'), ('<', '&lt;'), ('>', '&gt;')):
                        if s[0] in tt:
                            tt = re.sub(s[0], s[1], tt)

                    rval = re.sub(re.escape(mg), tt, rval)
                return rval

            except:
                return rval

        if not isinstance(page, (str, unicode)):
            return page

        if isinstance(self.text_replace, list):
            for subset in self.text_replace:
                if not isinstance(subset, list) or len(subset) < 2:
                    continue

                page = re.sub(subset[0], subset[1], page, 0, re.DOTALL)

        if isinstance(self.unquote_html, list):
            for ut in self.unquote_html:
                if not isinstance(ut, (str, unicode)):
                    continue

                page = re.sub(ut, unquote, page, 0, re.DOTALL)

        return page

    def get_source_regex(self, data, sr_id, sr_type = 1, sr_group = None):
        if not sr_id in self.source_regexes.keys() or self.source_regexes[sr_id] == None:
            if sr_type == 2:
                return []

            else:
                return None

        if sr_type == 0:
            return self.source_regexes[sr_id].match(data)

        if sr_type == 1:
            sre = self.source_regexes[sr_id].search(data)
            if sr_group == None or sre == None:
                return sre

            try:
                return sre.group(sr_group)

            except:
                return None

        if sr_type == 2:
            return self.source_regexes[sr_id].findall(data)

    def set_ready(self, channelid = None):
        if channelid == None:
            # All channels
            clist = self.chanids.keys()

        elif channelid in self.chanids.keys():
            # Or one
            clist = [channelid]

        else:
            return

        with self.source_lock:
            for channelid in clist:
                self.channel_loaded[channelid] = True
                for chanid in self.all_chanids[channelid]:
                    config.channels[chanid].source_data[self.proc_id].set()

    # Selectie functions
    def get_json_data(self, id, item):
        """Return the requested json item or None if not found"""
        if not self.isjson:
            return None

        if not id in self.json_by_id.keys():
            return None

        if item in self.json_by_id[id].keys():
            return self.unescape(self.json_by_id[id][item])

        if item in self.jsondata.keys():
            tdict = self.jsondata[item]
            if  tdict['keyname'] in self.json_by_id[id]:
                key =self.json_by_id[id][tdict['keyname']]
                if key in self.jsondict[tdict['listname']] and \
                  tdict['valuename'] in self.jsondict[tdict['listname']][key]:
                    return self.unescape(self.jsondict[tdict['listname']][key][tdict['valuename']])

    # Filter/merge processes
    def restrict_times(self, programs, start_time, end_time):
        if not isinstance(start_time, datetime.time) and isinstance(end_time, datetime.time):
            return programs

        restricted_programs = []
        for tdict in programs[:]:
            pstart = tdict['start-time']
            pstop = tdict['stop-time']
            tstart = datetime.datetime.combine(pstart.date(), start_time)
            tstop = datetime.datetime.combine(pstop.date(), end_time)
            if pstart.date() != pstop.date() and tstop - tstart > datetime.timedelta(days=1):
                tstart = datetime.datetime.combine(pstop.date(), start_time)
                tstop = datetime.datetime.combine(pstart.date(), end_time)

            if (tstart > tstop and tstop <= pstart <= tstart and tstop <= pstop <= tstart) or \
                (tstart < tstop and ((pstart <= tstart and pstop <= tstart) or (pstart >= tstop and pstop >= tstop))):
                    continue

            if pstart < tstart and pstop >= tstart:
                tdict['start-time'] = tstart

            if pstart <= tstop and pstop > tstop:
                tdict['stop-time'] = tstop

            restricted_programs.append(tdict)

        return restricted_programs

    def parse_programs(self, chanid, mode = 0, overlap_strategy = None):
        """
        Parse a list of programs as generated by parser and
        adjust begin and end times to avoid gaps and overlap.
        Depending on the mode either:
        it's own data 'self.program_data[channelid]' (mode = 0) or
        the finally joined data 'config.channels[chanid].all_programs' (mode = 1) is parsed.
        Not setting the overlap_strategy will use the configured default.
        For inbetween parsing you best set it to 'None'
        """

        channel = config.channels[chanid]
        channelid = channel.get_source_id(self.proc_id)
        if mode == 0:
            with self.source_lock:
                programs = self.program_data[channelid][:]

        elif mode == 1:
            programs = channel.all_programs[:]

        else:
            return

        for item in programs[:]:
            if item == None:
                programs.remove(item)

        if len(programs) == 0:
            return

        # good programs
        good_programs = []
        fill_programs = []

        # sort all programs by startdate, enddate
        programs.sort(key=lambda program: (program['start-time'],program['stop-time']))
        if overlap_strategy == None:
            overlap_strategy = channel.get_opt('overlap_strategy')

        # next, correct for missing end time and copy over all good programming to the
        # good_programs list
        for i in range(len(programs)):

            # Try to correct missing end time by taking start time from next program on schedule
            if (programs[i]['stop-time'] == None and i < len(programs)-1):
                log('Oops, "%s" has no end time. Trying to fix...\n' % programs[i]['name'], 64)
                programs[i]['stop-time'] = programs[i+1]['start-time']

            # The common case: start and end times are present and are not
            # equal to each other (yes, this can happen)
            if programs[i]['start-time'] != None \
                and programs[i]['stop-time']  != None \
                and programs[i]['start-time'] < programs[i]['stop-time']:
                    good_programs.append(programs[i])

        # Try to exclude programs that only identify a group or broadcaster and have overlapping start/end times with
        # the actual programs
        for i in range(len(good_programs)-2,-1,-1):

            if good_programs[i]['start-time'] == good_programs[i+1]['start-time'] \
                and good_programs[i]['stop-time']  == good_programs[i+1]['stop-time'] \
                and good_programs[i]['name']  == good_programs[i+1]['name']:
                    log('Deleting duplicate: %s\n' % good_programs[i]['name'], 64)
                    del good_programs[i]
                    continue

            if good_programs[i]['start-time'] <= good_programs[i+1]['start-time'] \
                and good_programs[i]['stop-time']  >= good_programs[i+1]['stop-time']:
                    log('Deleting grouping/broadcaster: %s\n' % good_programs[i]['name'], 64)
                    del good_programs[i]

        # Fix overlaps/gaps
        if overlap_strategy in ['average', 'stop', 'start', 'fill']:
            for i in range(len(good_programs)-1):

                # PdB: Fix tvgids start-before-end x minute interval overlap.  An overlap (positive or
                # negative) is halved and each half is assigned to the adjacent programmes. The maximum
                # overlap length between programming is set by the global variable 'max_overlap' and is
                # default 10 minutes. Examples:
                #
                # Positive overlap (= overlap in programming):
                #   10:55 - 12:00 Lala
                #   11:55 - 12:20 Wawa
                # is transformed in:
                #   10:55 - 11.57 Lala
                #   11:57 - 12:20 Wawa
                #
                # Negative overlap (= gap in programming):
                #   10:55 - 11:50 Lala
                #   12:00 - 12:20 Wawa
                # is transformed in:
                #   10:55 - 11.55 Lala
                #   11:55 - 12:20 Wawa

                stop  = good_programs[i]['stop-time']
                start = good_programs[i+1]['start-time']
                dt    = stop-start
                avg   = start + dt // 2
                overlap = dt.total_seconds()

                # check for the size of the overlap
                if 0 < abs(overlap) <= channel.get_opt('max_overlap')*60:
                    if overlap > 0:
                        log('"%s" and "%s" overlap %s minutes. Adjusting times.\n' % \
                            (good_programs[i]['name'],good_programs[i+1]['name'],overlap // 60), 64)
                    else:
                        log('"%s" and "%s" have gap of %s minutes. Adjusting times.\n' % \
                            (good_programs[i]['name'],good_programs[i+1]['name'],abs(overlap) // 60), 64)

                    # stop-time of previous program wins
                    if overlap_strategy == 'stop':
                       good_programs[i+1]['start-time'] = good_programs[i]['stop-time']

                    # start-time of next program wins
                    elif overlap_strategy == 'start':
                       good_programs[i]['stop-time'] = good_programs[i+1]['start-time']

                    # average the difference
                    elif overlap_strategy == 'average':
                       good_programs[i]['stop-time']    = avg
                       good_programs[i+1]['start-time'] = avg

                    # We fill it with a programinfo/commercial block
                    elif overlap_strategy == 'fill' and overlap < 0:
                        tdict = self.checkout_program_dict()
                        tdict['source'] = good_programs[i]['source']
                        tdict['channelid'] = good_programs[i]['channelid']
                        tdict['channel'] = good_programs[i]['channel']
                        tdict['name'] = config.npo_fill
                        tdict['start-time'] = good_programs[i]['stop-time']
                        tdict['stop-time'] = good_programs[i+1]['start-time']
                        tdict['offset'] = good_programs[i+1]['offset']
                        tdict['genre'] = u'overige'
                        fill_programs.append(tdict)

                    # leave as is
                    else:
                       pass

                # For NPO we fill the night gap
                elif good_programs[i]['source'] == u'npo' and overlap_strategy == 'fill' and (0 < good_programs[i]['stop-time'].hour < 6):
                    if good_programs[i]['name'] == 'Tekst-TV':
                        good_programs[i]['stop-time'] = good_programs[i+1]['start-time']

                    elif good_programs[i+1]['name'] == 'Tekst-TV':
                        good_programs[i+1]['start-time'] = good_programs[i]['stop-time']

                    else:
                        tdict = self.checkout_program_dict()
                        tdict['source'] = good_programs[i]['source']
                        tdict['channelid'] = good_programs[i]['channelid']
                        tdict['channel'] = good_programs[i]['channel']
                        tdict['name'] = 'Tekst-TV'
                        tdict['start-time'] = good_programs[i]['stop-time']
                        tdict['stop-time'] = good_programs[i+1]['start-time']
                        tdict['offset'] = good_programs[i+1]['offset']
                        tdict['genre'] = u'nieuws/actualiteiten'
                        fill_programs.append(tdict)

        # Experimental strategy to make sure programming does not disappear. All programs that overlap more
        # than the maximum overlap length, but less than the shortest length of the two programs are
        # clumped.
        if config.do_clump:
            for i in range(len(good_programs)-1):

                stop  = good_programs[i]['stop-time']
                start = good_programs[i+1]['start-time']
                dt    = stop-start
                overlap = 24*60*60*dt.days + dt.seconds

                length0 = good_programs[i]['stop-time']   - good_programs[i]['start-time']
                length1 = good_programs[i+1]['stop-time'] - good_programs[i+1]['start-time']

                l0 = length0.days*24*60*60 + length0.seconds
                l1 = length1.days*24*60*60 + length0.seconds

                if abs(overlap) >= channel.get_opt('max_overlap')*60 <= min(l0,l1)*60 and \
                    'clumpidx' not in good_programs[i]   and \
                    'clumpidx' not in good_programs[i+1]:
                    good_programs[i]['clumpidx']   = '0/2'
                    good_programs[i+1]['clumpidx'] = '1/2'
                    good_programs[i]['stop-time'] = good_programs[i+1]['stop-time']
                    good_programs[i+1]['start-time'] = good_programs[i]['start-time']


        # done, nothing to see here, please move on
        if len(fill_programs) > 0:
            good_programs.extend(fill_programs)

        if mode == 0:
            with self.source_lock:
                self.program_data[self.channels[chanid]] = good_programs

        elif mode == 1:
            channel.all_programs = good_programs

    def merge_sources(self, chanid, prime_source, counter = 0, merge_channel = None):
        """
        Try to match the channel info from the sources into the prime source.  If No prime_source is set
        If available: rtl.nl is used for the rtl channels, npo.nl for the npo and regional channels and teveblad.be
        for the flemmish channels.
        Else the first available is used as set in xml_output.source_order
        """

        def write_fetchlist():
            try:
                infofiles.write_fetch_list(channel.all_programs[:], chanid, other_source_name, None, True)

            except:
                pass
        # end write_fetchlist():

        no_genric_matching = False
        channel = config.channels[chanid]
        channelid = channel.get_source_id(self.proc_id)
        if merge_channel == None:
            if channelid != '' and channelid in config.no_genric_matching[self.proc_id]:
                no_genric_matching = True

            source_merge = True
            prime_source_name = xml_output.channelsource[prime_source].source
            other_source_name = self.source
            with self.source_lock:
                if not channelid in self.program_data:
                    self.program_data[channelid] = []

                if len(self.program_data[channelid]) == 0:
                    # Nothing to merge
                    write_fetchlist()
                    return

                # This is the by this source collected data
                programs = deepcopy(self.program_data[channelid])
                if channel.is_virtual_sub:
                    # It's time restricted
                    programs = self.restrict_times(programs, channel.virtual_start, channel.virtual_end)

                if channel.opt_dict['prefered_description'] == self.proc_id:
                    # It supplies the description
                    for i in range(len(programs)):
                        programs[i]['prefered description'] = programs[i]['description']

                if len(channel.all_programs) == 0:
                    channel.all_programs = programs
                    # It's the first merge
                    write_fetchlist()
                    return

                # This is the already collected data to start with the prime source
                info = channel.all_programs[:]

        else:
            # This is a channel merge
            source_merge = False
            prime_source_name = channel.chan_name
            other_source_name = config.channels[merge_channel['chanid']].chan_name
            if len(config.channels[merge_channel['chanid']].child_programs) == 0:
                # Nothing to merge
                write_fetchlist()
                return

            programs = deepcopy(config.channels[merge_channel['chanid']].child_programs)
            if 'start' in merge_channel and 'end' in merge_channel:
                # This channel is limited to a timeslot
                no_genric_matching = True
                programs = self.restrict_times(programs, merge_channel['start'], merge_channel['end'])

            if len(channel.all_programs) == 0:
                channel.all_programs = programs
                # It's the first merge
                write_fetchlist()
                return

            # This is the already collected data to start with the prime source
            info = channel.all_programs[:]

        match_array = [   'Match details:\n']
        def matchlog(matchstr, other_prog, tvgids_prog = None, mode = 1):
            if not (mode & config.opt_dict['match_log_level']):
                return

            if mode == 4:
                match_array.extend([u'%s: %s - %s: %s.\n' % \
                        ((matchstr+other_source_name).rjust(25),  other_prog['start-time'].strftime('%d %b %H:%M'),  other_prog['stop-time'].strftime('%H:%M'), other_prog['name']), \
                        '%s: %s - %s: %s.\n' % \
                        (('to '+ prime_source_name).rjust(25), tvgids_prog['start-time'].strftime('%d %b %H:%M'), tvgids_prog['stop-time'].strftime('%H:%M'), tvgids_prog['name'])])
            elif tvgids_prog == None:
                match_array.append(u'%s: %s - %s: %s Genre: %s.\n' % \
                        ((matchstr+other_source_name).rjust(25), other_prog['start-time'].strftime('%d %b %H:%M'),  other_prog['stop-time'].strftime('%H:%M'), \
                        other_prog['name'], other_prog['genre']))
            elif other_prog == None:
                match_array.append(u'%s: %s - %s: %s Genre: %s.\n' % \
                        (matchstr.rjust(25), tvgids_prog['start-time'].strftime('%d %b %H:%M'), tvgids_prog['stop-time'].strftime('%H:%M'), \
                        tvgids_prog['name'], tvgids_prog['genre']))
        # end matchlog()

        def general_renames(name):
            # Some renaming to cover diferences between the sources
            mname = name.lower()
            if chanid in ('0-1', '0-2', '0-3'):
                if mname == 'journaal':
                    return 'NOS Journaal'

                if mname in ('tekst-tv', 'nos tekst tv', 'nos tekst-tv'):
                    return 'Tekst TV'

            if chanid in ('0-1', '0-2'):
                if mname == 'nieuws':
                    return 'NOS Journaal'

            if chanid == '0-3':
                if mname == 'nieuws':
                    return 'NOS op 3'

            if chanid == '0-5':
                if mname == 'herhalingen':
                    return 'Journaallus'

            if chanid == '0-6':
                if mname == 'herhalingen':
                    return 'Canvaslus'

            if chanid in ('0-7', '0-8'):
                if mname == 'nieuws':
                    return 'BBC News'

                if mname == 'het weer':
                    return 'Regional News and Weather'

            if chanid == '0-9':
                if mname == 'nieuws':
                    return 'Tagesschau'

            if chanid == '0-10':
                if mname == 'nieuws':
                    return 'Heute'

            if self.source == 'horizon.tv':
                if chanid in ('0-1', '0-2', '0-3'):
                    if  'nos journaal' in mname:
                        return 'NOS Journaal'

                    if  'nos jeugdjournaal' in mname:
                        return 'Jeugdjournaal'

                    if  'studio sport' in mname:
                        return 'Studio sport'

                    if  'sportjournaal' in mname:
                        return 'Sportjournaal'

                    if mname == 'z@ppbios':
                        return 'Zappbios'

                    if mname == 'z@ppsport':
                        return 'ZappSport'

                if chanid in ('0-5', '0-6'):
                    if  'het journaal' in mname:
                        return 'Journaal'

                if chanid in ('0-4', '0-31', '0-46', '0-92'):
                    if 'rtl nieuws' in mname:
                        return 'RTL Nieuws'

            name = re.sub(' / ',' - ', name)
            return name
        # end general_renames()

        def checkrange(crange = 0):
            checktimes = []
            if crange == 0:
                checktimes.append(0)

            for i in range(1, 6):
                checktimes.append(crange + i)
                checktimes.append(-(crange + i))

            return checktimes
        # end checkrange()

        def match_name(other_title, tvgids_name, other_subtitle = ''):
            """
            Main process for name matching
            Returns 0 if matched on name = name
            Returns 1 if matched on name:episode = name
            Returns None if no match
            """
            def compare(nother, ntvgids, nsub = ''):
                if nother == ntvgids:
                    return 0

                if re.sub('[-,. ]', '', nother) == re.sub('[-,. ]', '', ntvgids):
                    return 0

                if len(ntvgids.split(':')) > 1 and nsub != '':
                    ntvsplit = ntvgids.split(':')[0]
                    if nother == ntvsplit:
                        return 1

                    if len(nother) < len(ntvsplit):
                        if nother == ntvsplit[len(ntvsplit) - len(nother):]:
                            return 1

                        if nother == ntvsplit[0:len(nother)]:
                            return 1

                    if len(nother) > len(ntvsplit):
                        if nother[len(nother) - len(ntvsplit):] == ntvsplit:
                            return 1

                        elif nother[0:len(ntvsplit)] == ntvsplit:
                            return 1

                if len(nother) < len(ntvgids):
                    if nother == ntvgids[len(ntvgids) - len(nother):]:
                        return 0

                    if nother == ntvgids[0:len(nother)]:
                        return 0

                if len(nother) > len(ntvgids):
                    if nother[len(nother) - len(ntvgids):] == ntvgids:
                        return 0

                    elif nother[0:len(ntvgids)] == ntvgids:
                        return 0

                return None
            # end compare()

            other_name = other_title.lower().strip()
            other_subname = other_subtitle.lower().strip()
            tvgids_name = tvgids_name.lower().strip()
            x = compare(xml_output.remove_accents(other_name), xml_output.remove_accents(tvgids_name), xml_output.remove_accents(other_subname))
            if x != None:
                return x

            matchobject = difflib.SequenceMatcher(isjunk=lambda x: x in " '\",.-/", autojunk=False)
            matchobject.set_seqs(xml_output.remove_accents(other_name), xml_output.remove_accents(tvgids_name))
            if matchobject.ratio() > .8:
                return 0

            name_split = False
            lother_name = other_name
            rother_name = other_name
            if other_name.find(':') != -1:
                name_split = True
                lother_name = other_name.split(':')[0].strip()
                rother_name = other_name.split(':')[1].strip()

            ltvgids_name = tvgids_name
            rtvgids_name = tvgids_name
            if tvgids_name.find(':') != -1:
                name_split = True
                ltvgids_name = tvgids_name.split(':')[0].strip()
                rtvgids_name = tvgids_name.split(':')[1].strip()

            if name_split:
                x = compare(xml_output.remove_accents(rother_name), xml_output.remove_accents(rtvgids_name))
                if x != None:
                    return x

                matchobject.set_seqs(xml_output.remove_accents(rother_name), xml_output.remove_accents(rtvgids_name))
                if matchobject.ratio() > .8:
                    return 0

                x = compare(xml_output.remove_accents(lother_name), xml_output.remove_accents(ltvgids_name))
                if x != None:
                    return x

                matchobject.set_seqs(xml_output.remove_accents(lother_name), xml_output.remove_accents(ltvgids_name))
                if matchobject.ratio() > .8:
                    return 0

            return None
        # end match_name()

        def match_genre(other_genre, tvgids_genre, tvgids_subgenre):
            """
            Process for Genre matching
            Returns True or False
            """
            tvgids_genre = tvgids_genre.lower().strip()
            tvgids_subgenre = tvgids_subgenre.lower().strip()
            other_genre = other_genre.lower().strip()
            if  (tvgids_genre == 'overige') or (other_genre == 'overige'):
                return False

            elif  (tvgids_genre != '') and (other_genre == tvgids_genre):
                return True

            elif (other_genre == 'amusement'):
                if (tvgids_genre == 'amusement') or (tvgids_genre == 'amusment') \
                  or (tvgids_genre == 'kunst en cultuur'):
                    return True

            elif (other_genre == 'kinderen') and (tvgids_genre == 'jeugd'):
                return True

            elif (other_genre == 'magazine') and (tvgids_genre == 'informatief, amusement'):
                return True

            elif (other_genre == 'nieuws') and (tvgids_genre == 'nieuws/actualiteiten'):
                return True

            elif (other_genre == 'serie') and (tvgids_genre == 'serie/soap'):
                return True

            elif (other_genre == 'serie') and (tvgids_genre == 'film'):
                return True

            elif (other_genre == 'reality'):
                if (tvgids_genre == 'informatief'):
                    if (tvgids_subgenre == 'realityprogramma') or (tvgids_subgenre == 'realityserie'):
                        return True

            elif (other_genre == 'documentaire'):
                if (tvgids_genre == 'informatief') and (tvgids_subgenre == 'documentaire'):
                    return True

                elif (tvgids_genre == 'info') and (tvgids_subgenre == 'documentary'):
                    return True

                elif (tvgids_genre == 'natuur') and (tvgids_subgenre == 'natuurdocumentaire, natuurprogramma'):
                    return True

            return False
        # end match_genre()

        def set_main_id(tdict):

            for s in xml_output.sourceid_order:
                if tdict['prog_ID'][s] != '':
                    tdict['ID'] = tdict['prog_ID'][s]
                    break

            return tdict
        # end set_main_id()

        def merge_programs(tdict, tvdict, reverse_match=False, use_other_title = 0, copy_ids = True):
            if use_other_title != 0:
                tdict['name']  = tvdict['name']

            if tdict['jaar van premiere'] == '':
                tdict['jaar van premiere'] = tvdict['jaar van premiere']

            if tdict['airdate'] == '':
                tdict['airdate'] = tvdict['airdate']

            if tvdict['rerun']:
                tdict['rerun'] = True

            if tdict['country'] == '':
                tdict['country'] = tvdict['country']

            if tdict['originaltitle'] == '':
                tdict['originaltitle'] = tvdict['originaltitle']

            if len(tvdict['description']) > len(tdict['description']):
                tdict['description']  = tvdict['description']

            if tdict['prefered description'] == '':
                tdict['prefered description'] = tvdict['prefered description']

            if tdict['omroep'] == '':
                tdict['omroep'] = tvdict['omroep']

            if tdict['star-rating'] == '':
                tdict['star-rating'] = tvdict['star-rating']

            if len(tvdict['kijkwijzer']) > 0:
                for item in tvdict['kijkwijzer']:
                    tdict['kijkwijzer'].append(item)

            if tvdict['video']['HD']:
                tdict['video']['HD']  = True

            if tvdict['video']['breedbeeld']:
                tdict['video']['breedbeeld']  = True

            if tvdict['video']['blackwhite']:
                tdict['video']['blackwhite']  = True

            if tvdict['teletekst']:
                tdict['teletekst']  = True

            if tdict['audio'] == '':
                tdict['audio'] = tvdict['audio']

            for role in tvdict['credits']:
                if not role in tdict['credits']:
                    tdict['credits'][role] = []

                for rp in tvdict['credits'][role]:
                    if not rp in tdict['credits'][role]:
                        tdict['credits'][role].append(rp)

            if copy_ids:
                for source in xml_output.source_order:
                    if tvdict['prog_ID'][source] != u'':
                        tdict['prog_ID'][source]  = tvdict['prog_ID'][source]

                    if tvdict['detail_url'][source] != u'':
                        tdict['detail_url'][source]  = tvdict['detail_url'][source]

            tdict = set_main_id(tdict)
            if reverse_match:
                if not self.proc_id in (2, 6, 5) and tdict['titel aflevering'] == '':
                    tdict['titel aflevering'] = tvdict['titel aflevering']

                if self.proc_id != 1:
                    tdict['genre'] = tvdict['genre']
                    tdict['subgenre'] = tvdict['subgenre']

                elif tdict['genre'] in ('', 'overige'):
                    tdict['genre'] = tvdict['genre']
                    if tdict['subgenre'] == '':
                        tdict['subgenre'] = tvdict['subgenre']

                tdict['merge-source'] = other_source_name
                matched_programs.append(tdict)
                if tdict in programs: programs.remove(tdict)
                if tdict['start-time'] in prog_starttimes: del prog_starttimes[tdict['start-time']]

            else:
                # We try to fill gaps in the prime source that are defined in the other
                for item in info_gaps:
                    if tdict['stop-time'] == item['start-time'] and item['start-time'] < tvdict['stop-time'] <= item['stop-time']:
                            tdict['stop-time'] = tvdict['stop-time']
                            break

                    if tdict['start-time'] == item['stop-time'] and item['start-time'] < tvdict['start-time'] <= item['stop-time']:
                            tdict['start-time'] = tvdict['start-time']
                            break

                if self.proc_id in (2, 6, 5) and (tvdict['titel aflevering'] != '' or tdict['titel aflevering'] == ''):
                    tdict['titel aflevering'] = tvdict['titel aflevering']

                if tdict['season'] == 0:
                    tdict['season'] = tvdict['season']

                if tdict['episode'] == 0:
                    tdict['episode'] = tvdict['episode']

                if self.proc_id == 1:
                    tdict['genre'] = tvdict['genre']
                    tdict['subgenre'] = tvdict['subgenre']

                elif tdict['genre'] in ('', 'overige'):
                    tdict['genre'] = tvdict['genre']
                    if tdict['subgenre'] == '':
                        tdict['subgenre'] = tvdict['subgenre']

                if tdict['merge-source'] == '':
                    tdict['merge-source'] = prime_source_name

                matched_programs.append(tdict)
                if tdict in info: info.remove(tdict)

        # merge_programs()

        # tdict is from info
        def check_match_to_info(tdict, pi, mstart, check_overlap = True, check_genre = True, auto_merge = True):
            if no_genric_matching:
                check_genre = False

            x = match_name(pi['name'], tdict['name'], pi['titel aflevering'])
            if x != None:
                matchlog('title match: ', pi, tdict, 4)
                retval = 1

            elif check_genre and match_genre(pi['genre'], tdict['genre'], pi['subgenre']):
                matchlog('genre match: ', pi, tdict, 4)
                x = 0
                retval = 2

            else:
                return 0

            if check_overlap and not no_genric_matching:
                try:
                    mduur = (tdict['stop-time'] - tdict['start-time']).total_seconds()
                    pduur = (pi['stop-time'] - pi['start-time']).total_seconds()
                    if pduur * 1.1 > mduur:
                        # We check for program merging in info
                        merge_match.append({'type': 1, 'tdict': tdict, 'prog': pi, 'match': x})
                        if tdict in info: info.remove(tdict)

                    elif mduur * 1.1 > pduur:
                        # We check for program merging in programs
                        merge_match.append({'type': 2, 'tdict': tdict, 'prog': pi, 'match': x})
                        if tdict in info: info.remove(tdict)

                    elif auto_merge:
                        merge_programs(tdict, pi, reverse_match=False, use_other_title = x)

                except:
                    if auto_merge:
                        merge_programs(tdict, pi, reverse_match=False, use_other_title = x)

            elif auto_merge:
                merge_programs(tdict, pi, reverse_match=False, use_other_title = x)

            if pi in programs: programs.remove(pi)
            if mstart in prog_starttimes: del prog_starttimes[mstart]
            return retval

        # end check_match_to_info()

        if merge_channel == None:
            log(['\n', 'Now merging %s (channel %s of %s):\n' % (channel.chan_name , counter, config.chan_count), \
                '  %s programs from %s into %s programs from %s\n' % \
                (len(programs), other_source_name, len(info), prime_source_name)], 2)
            log_array =['\n']
            log_array.append('Merg statistics for %s (channel %s of %s) from %s into %s\n' % \
                (channel.chan_name , counter, config.chan_count, other_source_name, prime_source_name))

        else:
            log(['\n', 'Now merging %s programs from %s into %s programs from %s\n' % \
                    (len(programs), other_source_name, len(info), prime_source_name), \
                    '    (channel %s of %s)' % (counter, config.chan_count)], 2)
            log_array =['\n']
            log_array.append('Merg statistics for %s (channel %s of %s) from %s\n' % \
                (prime_source_name , counter, config.chan_count, other_source_name))

        # Do some general renaming to match tvgids.nl naming
        for i in range(0, len(programs)):
            programs[i]['name'] = general_renames(programs[i]['name'])

        for i in range(0, len(info)):
            info[i]['name'] = general_renames(info[i]['name'])

        # Sort both lists on starttime and get their ranges
        info.sort(key=lambda program: (program['start-time'],program['stop-time']))
        infostarttime = info[0]['start-time'] + datetime.timedelta(seconds = 5)
        infoendtime = info[-1]['stop-time'] - datetime.timedelta(seconds = 5)

        programs.sort(key=lambda program: (program['start-time'],program['stop-time']))
        progstarttime = programs[0]['start-time'] + datetime.timedelta(seconds = 5)
        progendtime = programs[-1]['stop-time'] - datetime.timedelta(seconds = 5)

        log_array.append('%6.0f programs in %s for range: %s - %s, \n' % \
            (len(info), prime_source_name.ljust(11), infostarttime.strftime('%d-%b %H:%M'), infoendtime.strftime('%d-%b %H:%M')))
        log_array.append('%6.0f programs in %s for range: %s - %s\n' % \
            (len(programs), other_source_name.ljust(11), progstarttime.strftime('%d-%b %H:%M'), progendtime.strftime('%d-%b %H:%M')))
        log_array.append('\n')

        # move all programs outside the range of programs to matched_programs
        # count the info names, changing them to lowercase for matching
        # and organise them by name and start-time
        matched_programs = []
        info_gaps = []
        generic_match = []
        info_groups = []
        info_starttimes = {}
        info_names = {}
        prog_groups = []
        prog_names = {}
        prog_starttimes ={}
        ocount = 0

        # Get existing gaps in info larger then 'max_overlap'
        for index in range(1, len(info)):
            if (info[index]['start-time'] -  info[index -1]['stop-time']).total_seconds()  > channel.get_opt('max_overlap')*60:
                info_gaps.append({'start-time': info[index -1]['stop-time'] - datetime.timedelta(seconds = 5 ),
                                                'stop-time': info[index]['start-time'] + datetime.timedelta(seconds = 5 )})

        # And we create a list of starttimes and of names for matching
        for tdict in info[:]:
            if (tdict['name'].lower() in config.groupslot_names) \
              or (chanid in ('0-1', '0-2', '0-3') and  tdict['name'].lower() == 'kro kindertijd') \
              or (chanid in ('0-34','1-veronica', "0-311") and \
              (tdict['name'].lower() == 'disney xd' or tdict['name'].lower() == 'disney')):
                # These are group names. We move them aside to not get hit by merge_match
                info_groups.append(tdict)
                if tdict in info: info.remove(tdict)
                continue

            info_starttimes[tdict['start-time']] = tdict
            iname = tdict['name'].lower().strip()
            if not iname in info_names or (info_names[iname]['genre'] in ('', 'overige')):
                info_names[iname] = tdict

            # These do not overlap in time so they cannot be matched
            if (tdict['start-time'] >= progendtime) or (tdict['stop-time'] <= progstarttime):
                ocount += 1
                tdict = set_main_id(tdict)
                if tdict['merge-source'] == '':
                    tdict['merge-source'] = prime_source_name

                if tdict['genre'] in ('', 'overige'):
                    # We later try to match them generic to get a genre
                    generic_match.append(tdict)

                else:
                    matched_programs.append(tdict)

                matchlog('added from info', None, tdict, 1)
                if tdict in info: info.remove(tdict)

        # count the occurense of the rest and organise by name/start-time and stop-time
        for tdict in programs[:]:
            if (tdict['name'].lower() in config.groupslot_names) \
              or (chanid in ('0-1', '0-2', '0-3') and  tdict['name'].lower() == 'kro kindertijd') \
              or (chanid in ('0-34','1-veronica', "0-311") and \
              (tdict['name'].lower() == 'disney xd' or tdict['name'].lower() == 'disney')):
                # These are group names. We move them aside to not get hit by merge_match
                prog_groups.append(tdict)
                if tdict in programs: programs.remove(tdict)
                continue

            prog_starttimes[tdict['start-time']] = tdict
            prog_starttimes[tdict['start-time']]['matched'] = False
            rname = tdict['name'].lower().strip()
            if not (rname in prog_names):
                prog_names[rname] = {}
                prog_names[rname]['count'] = 0
                prog_names[rname]['genre'] = tdict['genre']
                prog_names[rname]['subgenre'] = tdict['subgenre']

            elif prog_names[rname]['genre'] in ('', 'overige'):
                prog_names[rname]['genre'] = tdict['genre']
                prog_names[rname]['subgenre'] = tdict['subgenre']

            prog_names[rname]['count'] += 1
            # These do not overlap in time so they cannot be matched
            if (tdict['start-time'] >= infoendtime) or (tdict['stop-time'] <= infostarttime):
                ocount += 1
                tdict = set_main_id(tdict)
                tdict['merge-source'] = other_source_name
                if tdict['genre'] in ('', 'overige'):
                    # We later try to match them generic to get a genre
                    generic_match.append(tdict)

                else:
                    matched_programs.append(tdict)

                matchlog('added from ', tdict, None, 1)
                if tdict in programs: programs.remove(tdict)
                if tdict['start-time'] in prog_starttimes: del prog_starttimes[tdict['start-time']]
                continue

            # These are missing in info so they cannot be matched
            for pgap in info_gaps[:]:
                if (tdict['start-time'] >= pgap['start-time']) and (tdict['stop-time'] <= pgap['stop-time']):
                    ocount += 1
                    tdict = set_main_id(tdict)
                    tdict['merge-source'] = other_source_name
                    if tdict['genre'] in ('', 'overige'):
                        # We later try to match them generic to get a genre
                        generic_match.append(tdict)

                    else:
                        matched_programs.append(tdict)

                    matchlog('added from ', tdict, None, 1)
                    if tdict in programs: programs.remove(tdict)
                    if tdict['start-time'] in prog_starttimes: del prog_starttimes[tdict['start-time']]
                    break

        log_array.append('%6.0f programs added outside common timerange\n' % ocount)
        log_array.append('%6.0f programs left in %s to match\n' % (len(info), prime_source_name))
        log_array.append('%6.0f programs left in %s to match\n' % (len(programs), other_source_name))
        log_array.append('\n')

        ncount = 0
        gcount = 0
        rcount = 0
        scount = 0
        # Try to match programs without genre to get genre
        for tdict in generic_match[:]:
            rname = tdict['name'].lower().strip()
            match_list = difflib.get_close_matches(rname, info_names.iterkeys(), 1, 0.9)
            if len(match_list) > 0 and not info_names[match_list[0]]['genre'] in ('', 'overige'):
                tdict['genre'] = info_names[match_list[0]]['genre']
                tdict['subgenre'] = info_names[match_list[0]]['subgenre']
                rcount += 1

            else:
                match_list = difflib.get_close_matches(rname, prog_names.iterkeys(), 1, 0.9)
                if len(match_list) > 0 and not prog_names[match_list[0]]['genre'] in ('', 'overige'):
                    tdict['genre'] = prog_names[match_list[0]]['genre']
                    tdict['subgenre'] = prog_names[match_list[0]]['subgenre']
                    rcount += 1

            tdict = set_main_id(tdict)
            matched_programs.append(tdict)
            if tdict in generic_match: generic_match.remove(tdict)

        # Parse twice to recheck after generic name matching
        for checkrun in (0, 1):
            # first look on matching starttime (+/- 5 min) and similar names or matching genre
            # extending the range by 5 min to 30
            merge_match =[]
            for check in range(0, 30, 5):
                if len(info) == 0:
                    break

                for tdict in info[:]:
                    for i in checkrange(check):
                        mstart = tdict['start-time'] + datetime.timedelta(0, 0, 0, 0, i)
                        if mstart in prog_starttimes:
                            pi = prog_starttimes[mstart]
                            x = check_match_to_info(tdict, pi, mstart, check_genre = (source_merge and (checkrun==1)))
                            if x == 1:
                                ncount += 1
                                break

                            if x == 2:
                                gcount += 1
                                break

            # Check for following twins that were merged in the other (teveblad shows following parts often separate)
            for item in merge_match:
                tdict = item['tdict']
                pi = item['prog']
                pset = []
                # pi (from programs) is the longer one (by 10%+)
                if item['type'] == 1:
                    pset.append(tdict)
                    for pp in info:
                        pduur = (pp['stop-time'] - pp['start-time']).total_seconds()
                        if (pi['start-time'] <= pp['start-time'] <= pi['stop-time']) \
                          and (pi['start-time'] <= pp['start-time'] <= pi['stop-time']):
                            # Full overlap
                            pset.append(pp)

                        elif (pi['start-time'] <= pp['start-time'] <= pi['stop-time']):
                            # Starttime overlap more than 50%
                            if (pi['stop-time'] - pp['start-time']).total_seconds() > (0.5 * pduur):
                                pset.append(pp)

                        elif (pi['start-time'] <= pp['stop-time'] <= pi['stop-time']):
                            # Stoptime overlap more than 50%
                            if (pp['stop-time'] - pi['start-time']).total_seconds() > (0.5 * pduur):
                                pset.append(pp)

                    if len(pset) > 1:
                        twin_ncount = 0
                        twin_gcount = 0
                        for pp in pset[:]:
                            if pp != tdict:
                                x = check_match_to_info(pp, pi, None, False, check_genre = source_merge, auto_merge = False)
                                if x == 0:
                                    # No match. Remove it
                                    pset.remove(pp)

                                elif x == 1:
                                    # It matches on name
                                    twin_ncount += 1

                                elif x == 2:
                                    # It matches on genre
                                    twin_gcount += 1

                    if len(pset) > 1:
                        if channel.get_opt('use_split_episodes'):
                            ncount += twin_ncount
                            gcount += twin_gcount
                            for pp in pset:
                                if pp == tdict:
                                    # The original match
                                    merge_programs(pp, pi)

                                else:
                                    merge_programs(pp, pi, copy_ids = False)

                        else:
                            # So we have to use the timings from programs
                            merge_programs(pi, tdict, reverse_match = True, use_other_title = item['match'])

                    else:
                        merge_programs(tdict, pi, use_other_title = item['match'])

                # tdict (from info) is the longer one (by 10%+)
                elif item['type'] == 2:
                    pset.append(pi)
                    for pp in prog_starttimes.values():
                        pduur = (pp['stop-time'] - pp['start-time']).total_seconds()
                        if (tdict['start-time'] <= pp['start-time'] <= tdict['stop-time']) \
                          and (tdict['start-time'] <= pp['start-time'] <= tdict['stop-time']):
                            # Full overlap
                            pset.append(pp)

                        elif (tdict['start-time'] <= pp['start-time'] <= tdict['stop-time']) and \
                          (tdict['stop-time'] - pp['start-time']).total_seconds() > (0.5 * pduur):
                            # Starttime overlap more than 50%
                                pset.append(pp)

                        elif (tdict['start-time'] <= pp['stop-time'] <= tdict['stop-time']) and \
                          (pp['stop-time'] - tdict['start-time']).total_seconds() > (0.5 * pduur):
                            # Stoptime overlap more than 50%
                                pset.append(pp)

                    if len(pset) > 1:
                        twin_ncount = 0
                        twin_gcount = 0
                        for pp in pset[:]:
                            if pp != pi:
                                x = check_match_to_info(tdict, pp, None, False, check_genre = source_merge, auto_merge = False)
                                if x == 0:
                                    # No match. Remove it
                                    pset.remove(pp)

                                elif x == 1:
                                    # It matches on name
                                    twin_ncount += 1

                                elif x == 2:
                                    # It matches on genre
                                    twin_gcount += 1

                    if len(pset) > 1 and channel.get_opt('use_split_episodes'):
                        ncount += twin_ncount
                        gcount += twin_gcount
                        # So we have to use the timings from programs
                        for pp in pset:
                            if pp == pi:
                                # The original match
                                merge_programs(pp, tdict, reverse_match = True)

                            else:
                                merge_programs(pp, tdict, reverse_match = True, copy_ids = False)

                    else:
                        merge_programs(tdict, pi)

            # next mainly for rtl match generic on name to get genre. But only the first run
            if checkrun > 0:
                break

            for tdict in info[:]:
                rname = tdict['name'].lower().strip()
                match_list = difflib.get_close_matches(rname, info_names.iterkeys(), 1, 0.9)
                if len(match_list) > 0 and not info_names[match_list[0]]['genre'] in ('', 'overige'):
                    tdict['genre'] = info_names[match_list[0]]['genre']
                    tdict['subgenre'] = info_names[match_list[0]]['subgenre']
                    rcount += 1

                else:
                    match_list = difflib.get_close_matches(rname, prog_names.iterkeys(), 1, 0.9)
                    if len(match_list) > 0 and not prog_names[match_list[0]]['genre'] in ('', 'overige'):
                        tdict['genre'] = prog_names[match_list[0]]['genre']
                        tdict['subgenre'] = prog_names[match_list[0]]['subgenre']
                        rcount += 1

            log_array.append('%6.0f programs generically matched on name to get genre\n' % rcount)
            if rcount == 0 or no_genric_matching:
                break

        # Passing over generic timeslots that maybe detailed in the other
        delta_10 =  datetime.timedelta(minutes = 10)
        info.extend(info_groups)
        for tdict in prog_groups[:]:
            pcount = 0
            for tvdict in info[:]:
                if (tvdict['start-time'] >= (tdict['start-time'] - delta_10)) and (tvdict['stop-time'] <= (tdict['stop-time'] + delta_10)):
                    scount += 1
                    pcount += 1
                    tvdict = set_main_id(tvdict)
                    if tvdict['merge-source'] == '':
                        tvdict['merge-source'] = prime_source_name

                    matched_programs.append(tvdict)
                    if pcount == 1:
                        matchlog('groupslot in ', tdict, None, 8)

                    matchlog('', None, tvdict, 8)
                    if tvdict in info: info.remove(tvdict)
                    if tvdict in info_groups: info_groups.remove(tvdict)

            if pcount == 0:
                programs.append(tdict)

            if tdict['start-time'] in prog_starttimes: del prog_starttimes[tdict['start-time']]

        for tdict in info_groups[:]:
            pcount = 0
            for tvdict in programs[:]:
                if (tvdict['start-time'] >= (tdict['start-time'] - delta_10)) and (tvdict['stop-time'] <= (tdict['stop-time'] + delta_10)):
                    scount += 1
                    pcount += 1
                    tvdict = set_main_id(tvdict)
                    tvdict['merge-source'] = other_source_name
                    matched_programs.append(tvdict)
                    if pcount == 1:
                        matchlog('groupslot in info', None, tdict, 8)

                    matchlog('', tvdict, None, 8)
                    if tvdict in programs: programs.remove(tvdict)
                    if tvdict['start-time'] in prog_starttimes: del prog_starttimes[tvdict['start-time']]

            if pcount == 0:
                tdict = set_main_id(tdict)
                if tdict['merge-source'] == '':
                    tdict['merge-source'] = prime_source_name

                matchlog('added from info', None, tdict, 1)
                matched_programs.append(tdict)

            if tdict in info: info.remove(tdict)

        log_array.append('%6.0f programs matched on time and name\n' % ncount)
        log_array.append('%6.0f programs matched on time and genre\n' % gcount)
        log_array.append('%6.0f details  added from group slots\n' % scount)
        log_array.append('%6.0f programs added unmatched from info\n' % len(info))

        # List unmatched items to the log
        for tdict in info[:]:
            matchlog('added from info', None, tdict, 1)
            tdict = set_main_id(tdict)
            if tdict['merge-source'] == '':
                tdict['merge-source'] = prime_source_name

            matched_programs.append(tdict)

        p = []
        for tdict in prog_starttimes.itervalues():
            if infostarttime < tdict['start-time'] < infoendtime:
                p.append(tdict)

        p.sort(key=lambda program: (program['start-time'],program['stop-time']))
        for tdict in p:
            matchlog('left over in ', tdict, None , 2)

        log_array.append('\n')
        log(log_array, 4, 3)
        log(match_array, 32, 3)

        channel.all_programs = matched_programs
        write_fetchlist()

# end FetchData()

class tvgids_JSON(FetchData):
    """
    Get all available days of programming for the requested channels
    from the tvgids.nl json pages. Based on FetchData
    """
    def init_channels(self):
        """ Detail Site layout oud
            <head>
            <body>
                <div id="container">
                    <div id="header">
                    <div id="content">
                        <div id="content-header">Title</div>
                        <div id="content-col-left">
                            <div id="prog-content">Description</div>
                        <div id="content-col-right">
                            <div id="prog-info">
                                <div id="prog-info-content">
                                    <ul id="prog-info-content-colleft">
                                        <li><strong>Titel:</strong>Nederland Waterland</li>
                                            ...
                                    <ul id="prog-info-content-colright">
                                        <li><strong>Jaar van premiere:</strong>2014</li>
                                            ...
                                        <li><strong>Bijzonderheden:</strong>Teletekst ondertiteld, Herhaling, HD 1080i</li>
                                <div id="prog-info-footer"></div>
                            </div>
                        </div>
                    </div>
                    <div class="clearer"></div>
                </div>
                <div id="footer-container">
            </body>
            Nieuw
            <head>
            <body>
                <input type="hidden" id="categoryClass" value="">
                    <input type="hidden" id="notAllowedClass" value="">
                        <input type="hidden" id="notAllowedTitles" value="">
                            <div class="container pagecontainer">
                                <div class="row">
                                    <div class="col-md-8">
                                        <div id="prog-content">
                                            <div id="prog-video">
                                            ...
                                            </div>
                                            <div class="programmering">
                                                <h1>Harry Potter and the Goblet of Fire<span><sup>(2005)</sup></span></h1>
                                                <div class="clear:both;"></div>
                                                <script type="text/javascript" src="http://tvgidsassets.nl/v43/js/nlziet.js"></script>
                                                <div class="programmering_details">
                                                    <ul>
                                                        <li class="datum_tijd"> 1 mei 2015, 22:45 - 23:55 uur</li>
                                                        <li class="zender"><img src="http://tvgidsassets.nl/img/channels/53x27/36.png">SBS 6</li>
                                                    </ul>
                                                </div>
                                                <div style="clear:both"></div>
                                            </div>
                                            <div class="clear"></div>
                                                ...
                                            <div class="clear"></div>
                                            <p class="summary">
                                                <span class="articleblock articleblock_color_fantasy">
                                            FANTASY
                                                </span>
                                                                    Harry Potter gaat zijn vierde schooljaar in op de magische school Zweinstein, waar dit jaar het belangrijke internationale Triwizard Tournament wordt gehouden. Deze competitie is alleen voor de oudere en ervaren tovenaarsstudenten, maar toch komt Harry's naam boven als een van de deelnemers. Harry weet niet hoe dit mogelijk is, maar wordt toch gedwongen om mee te doen. Terwijl Harry zich voorbereidt op de gevaarlijke wedstrijd, wordt duidelijk dat de boosaardige Voldemort en zijn aanhangers steeds sterker worden en het nog altijd op zijn leven hebben gemunt. Dit nieuws is niet het enige wat Harry de rillingen bezorgt, hij heeft ook nog geen afspraakje voor het gala.
                                            </p>
                                            <p></p>
                                            <br class="brclear" />
                                            <div class="programmering_info_socials">
                                                ...
                                            </div>
                                            <br class="clear" />
                                        </div>
                                    </div>
                                </div>
                            </div>
                        </div>
                    </div>
                </div>
            </body>
        """

        self.init_channel_source_ids()
        self.url_channels = ''
        self.cooky_cnt = 0

        for channel in self.chanids.keys():
            if self.url_channels == '':
                self.url_channels = channel

            else:
                self.url_channels  = '%s,%s' % (self.url_channels, channel)

    def get_url(self, type = 'channels', offset = 0, id = None):

        tvgids_json = config.source_base_url[self.proc_id] + 'json/lists/'

        if type == 'channels':
            return  u'%schannels.php' % (tvgids_json)

        elif type == 'day':
            return '%sprograms.php?channels=%s&day=%s' % (tvgids_json, self.url_channels, offset)

        elif (id == None) or id == '':
            return ''

        elif type == 'detail':
            return u'%sprogramma/%s/?cookieoptin=true' % (config.source_base_url[self.proc_id], id)

        elif type == 'json_detail':
            return u'%sprogram.php?id=%s/' % (tvgids_json, id)

    def match_to_date(self, timestring, time, program):
        match = self.get_source_regex(self.unescape(timestring), 0, 0)

        if match:
            return datetime.datetime(int(match.group(1)),int(match.group(2)),\
                    int(match.group(3)),int(match.group(4)),int(match.group(5)),
                    tzinfo=CET_CEST)
        else:
            log("Can not determine %s for %s\n" % (time,program))
            return None

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        # download the json feed
        total = config.get_page(self.get_url(), 'utf-8')
        if total == None:
            log("Unable to get channel info from %s\n" % self.source)
            return 69  # EX_UNAVAILABLE

        channel_list = json.loads(total)

        # and create a file with the channels
        self.all_channels ={}
        for channel in channel_list:
            # the json data has the channel names in XML entities.
            channelid = channel['id']
            self.all_channels[channelid] = {}
            self.all_channels[channelid]['name'] = self.unescape(channel['name']).strip()

    def load_pages(self):

        if config.opt_dict['offset'] > 4:
            self.set_ready()
            return

        if len(self.chanids) == 0 :
            return

        dl = {}
        dd = {}
        for channelid in self.chanids.keys():
            dl[channelid] =[]
            dd[channelid] =[]

        first_fetch = True

        for retry in (0, 1):
            for offset in range(config.opt_dict['offset'], min((config.opt_dict['offset'] + config.opt_dict['days']), 4)):
                if self.quit:
                    return

                # Check if it is already loaded
                if self.day_loaded[0][offset]:
                    continue

                log(['\n', 'Now fetching %s channels from tvgids.nl\n' % len(self.chanids), \
                    '    (day %s of %s).\n' % (offset, config.opt_dict['days'])], 2)

                channel_url = self.get_url('day', offset)

                if not first_fetch:
                    # be nice to tvgids.nl
                    time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                    first_fetch = false

                # get the raw programming for the day
                strdata = config.get_page(channel_url, 'utf-8')
                if strdata == None or strdata.replace('\n','') == '{}':
                    if retry == 0:
                        log("No data on tvgids.nl for day=%d. First attempt.\n" % (offset))
                    else:
                         log("No data on tvgids.nl for day=%d. Final attempt.\n" % (offset))

                    self.fail_count += 1
                    continue

                # Just let the json library parse it.
                self.base_count += 1
                strdata = self.check_text_subs(strdata)
                for channelid, v in json.loads(strdata).iteritems():
                    # Most channels provide a list of program dicts, some a numbered dict
                    try:
                        if isinstance(v, dict):
                            v=list(v.values())

                        elif not isinstance(v, (list,tuple)):
                            raise TypeError

                    except (TypeError, LookupError):
                        log("Unsubscriptable content from channel url: %r\n" % channel_url)
                        continue
                    # remove the overlap at daychange and seperate the channels
                    for p in v:
                        if not p in dl[channelid]:
                            dd[channelid].append(p)

                self.day_loaded[0][offset] = True
                for channelid in self.chanids.keys():
                    if len(dd) > 0:
                        self.day_loaded[channelid][offset] = True
                        dl[channelid].extend(dd[channelid])
                        dd[channelid] =[]

        for channelid, chanid in self.chanids.items():
            if len(dl[channelid]) == 0:
                self.set_ready(channelid)
                continue

            # item is a dict, like:
            # {
            #  u'db_id': u'12379780',
            #  u'titel': u'Der unauff\xe4llige Mr. Crane'
            #  u'genre': u'Film',
            #  u'soort': u'Zwarte komedie',
            #  u'kijkwijzer': u'',
            #  u'artikel_id': None,
            #  u'artikel_titel': None,
            #  u'artikel_tekst': None,
            #  u'artikel_foto': None,
            #  u'datum_start': u'2012-03-12 01:20:00',
            #  u'datum_end': u'2012-03-12 03:05:00',
            # }

            # parse the list to adjust to what we want
            for item in dl[channelid]:
                tdict = self.checkout_program_dict()
                if (item['db_id'] != '') and (item['db_id'] != None):
                    tdict['prog_ID'][self.proc_id] = u'nl-%s' % (item['db_id'])
                    self.json_by_id[tdict['prog_ID'][self.proc_id]] = item
                    tdict['ID'] = tdict['prog_ID'][self.proc_id]

                tdict['source'] = self.source
                tdict['channelid'] = chanid
                tdict['channel']  = config.channels[chanid].chan_name
                tdict['detail_url'][self.proc_id] = self.get_url(type= 'detail', id = item['db_id'])

                # The Title
                tdict['name'] = self.unescape(item['titel'])
                tdict = self.check_title_name(tdict)
                if  tdict['name'] == None or tdict['name'] == '':
                    log('Can not determine program title for "%s"\n' % tdict['detail_url'][self.proc_id])
                    continue

                # The timing
                tdict['start-time'] = self.match_to_date(item['datum_start'],"begintijd", tdict['name'])
                tdict['stop-time']  = self.match_to_date(item['datum_end'], "eindtijd", tdict['name'])
                if tdict['start-time'] == None or tdict['stop-time'] == None:
                    continue

                tdict['offset'] = self.get_offset(tdict['start-time'])

                tdict['genre'] = self.unescape(item['genre']) if ('genre' in item and item['genre'] != None) else ''
                tdict['subgenre'] = self.unescape(item['soort']) if ('soort' in item and item['soort'] != None) else ''
                if  ('kijkwijzer' in item and not (item['kijkwijzer'] == None or item['kijkwijzer'] == '')):
                    for k in item['kijkwijzer']:
                        if k in config.kijkwijzer.keys() and k not in tdict['kijkwijzer']:
                            tdict['kijkwijzer'].append(k)

                self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict
                with self.source_lock:
                    self.program_data[channelid].append(tdict)

                config.genre_list.append((tdict['genre'].lower(), tdict['subgenre'].lower()))

            self.program_data[channelid].sort(key=lambda program: (program['start-time'],program['stop-time']))
            self.parse_programs(chanid, 0, 'None')
            self.set_ready(channelid)
            try:
                infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

            except:
                pass

    def load_detailpage(self, tdict):

        try:
            strdata = config.get_page(tdict['detail_url'][self.proc_id])
            if strdata == None:
                log('Page %s returned no data\n' % (tdict['detail_url'][self.proc_id]), 1)
                return

            if re.search('<div class="cookie-backdrop">', strdata):
                self.cooky_cnt += 1
                if self.cooky_cnt > 2:
                    self.cookyblock = True
                    log('More then 2 sequential Cooky block pages encountered. Falling back to json\n', 1)

                else:
                    self.cooky_cnt = 0

                return

            strdata = self.get_source_regex(strdata, 1)
            if strdata == None:
                log('Page %s returned no data\n' % (tdict['detail_url'][self.proc_id]), 1)
                return

            strdata = '<div>\n' +  strdata.group(1)
            if re.search('[Gg]een detailgegevens be(?:kend|schikbaar)', strdata):
                strtitle = ''
                strdesc = ''

            else:
                # They sometimes forget to close a <p> tag
                strdata = re.sub('<p>', '</p>xxx<p>', strdata, flags = re.DOTALL)
                strtitle = self.get_source_regex(strdata, 2)
                if strtitle == None:
                    strtitle = ''

                else:
                    # There are titles containing '<' (eg. MTV<3) which interfere. Since whe don't need it we remove the title
                    strtitle = re.sub('<h1>.*?<span>', '<h1><span>', strtitle.group(0), flags = re.DOTALL)
                    strtitle = strtitle + '\n</div>\n'

                strdesc = ''
                for d in self.get_source_regex(strdata, 3, 2):
                    strdesc += '<p%s</p>\n' % d

                strdesc = '<div>\n' + strdesc + '\n</div>\n'

                d = self.get_source_regex(strdata, 4)
                if d != None:
                    d = re.sub('</p>xxx<p>', '<p>', d.group(0), flags = re.DOTALL)
                    strdesc += d + '\n'

            strdetails = self.get_source_regex(strdata, 5)
            if strdetails == None:
                strdetails = ''

            else:
                strdetails = strdetails.group(0)

            strdata = (self.clean_html('<root>\n' + strtitle + strdesc + strdetails + '\n</root>\n')).strip()
            strdata = self.check_text_subs(strdata)
            htmldata = ET.fromstring(strdata.encode('utf-8'))

        except:
            log(['Fetching page %s returned an error:\n' % (tdict['detail_url'][self.proc_id]), traceback.format_exc()])
            if config.write_info_files:
                infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                infofiles.write_raw_string('<root>\n' + strtitle + strdesc + strdetails + '\n</root>\n')

            # if we cannot find the description page,
            # go to next in the loop
            return None

        # We scan every alinea of the description
        try:
            tdict = self.filter_description(htmldata, 'div/p', tdict)
            if config.channels[tdict['channelid']].opt_dict['prefered_description'] == self.proc_id:
                tdict['prefered description'] = tdict['description']

        except:
            log(['Error processing the description from: %s\n' % (tdict['detail_url'][self.proc_id]), traceback.format_exc()])
            if config.write_info_files:
                infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                infofiles.write_raw_string('<root>\n' + strdesc + '\n</root>\n')

        try:
            if htmldata.find('div/h1/span/sup') != None:
                tmp = htmldata.find('div/h1/span/sup').text
                if tmp != None:
                    tmp = re.sub('\(', '', tmp)
                    tdict['jaar van premiere'] = re.sub('\)', '', tmp).strip()

        except:
            log(traceback.format_exc())
            if config.write_info_files:
                infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                infofiles.write_raw_string(strdata)

        # We scan all the details
        for d in htmldata.findall('div/ul/li'):
            try:
                ctype = self.empersant(d.find('span[@class="col-lg-3"]').text).strip().lower()
                if ctype[-1] == ':':
                    ctype = ctype[0:len(ctype)-1]

                if ctype == 'kijkwijzer':
                    content = ''
                    for k in d.find('span[@class="col-lg-9 programma_detail_info kijkwijzer_img"]'):
                        item = {'text':k.get('alt', '') ,'icon':k.get('src', '')}
                        if item['text'] != '' or item['icon'] != '':
                            for kk, kw in config.kijkwijzer.items():
                                if (kw['text'] == item['text'] or kw['icon'] == item['icon']) and kk not in tdict['kijkwijzer']:
                                    tdict['kijkwijzer'].append(kk)
                                    break

                else:
                    content = self.empersant(d.find('span[@class="col-lg-9 programma_detail_info"]').text).strip()

            except:
                log(traceback.format_exc())
                if config.write_info_files:
                    infofiles.write_raw_string('Error: %s at line %s\n%s\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno, d))
                    infofiles.write_raw_string(strdata)

                continue

            try:
                if content == '':
                    continue

                elif ctype == 'aflevering':
                    # This contains a subtitle, optionally preseded by an episode number and an episode count
                    txt = self.get_source_regex(content, 6)
                    if txt != None:
                        tdict['episode'] = 0 if txt.group(1) in ('', None) else int(txt.group(1))
                        tdict['titel aflevering'] = '' if txt.group(2) in ('', None) else txt.group(2).strip()

                elif ctype == 'seizoen':
                    try:
                        tdict['season'] = int(content)

                    except:
                        pass

                elif ctype == 'genre':
                    tdict['genre'] = content.title()

                # Parse persons and their roles for credit info
                elif ctype in config.roletrans:
                    if not config.roletrans[ctype] in tdict['credits']:
                        tdict['credits'][config.roletrans[ctype]] = []

                    content = re.sub(' en ', ' , ', content)
                    persons = content.split(',');
                    for name in persons:
                        if name.find(':') != -1:
                            name = name.split(':')[1]

                        if name.find('-') != -1:
                            name = name.split('-')[0]

                        if name.find('e.a') != -1:
                            name = name.split('e.a')[0]

                        if not self.unescape(name.strip()) in tdict['credits'][config.roletrans[ctype]]:
                            tdict['credits'][config.roletrans[ctype]].append(self.unescape(name.strip()))

                # Add extra properties, while at the same time checking if we do not uncheck already set properties
                elif ctype == 'kleur':
                    tdict['video']['blackwhite'] = (content.find('zwart/wit') != -1)

                elif ctype == 'bijzonderheden':
                    if config.write_info_files:
                        infofiles.addto_detail_list(unicode(ctype + ' = ' + content))

                    content = content.lower()
                    if tdict['video']['breedbeeld'] == False:
                        tdict['video']['breedbeeld'] = (content.find('breedbeeld') != -1)
                    if tdict['video']['HD'] == False:
                        tdict['video']['HD'] = (content.find('hd 1080i') != -1)
                    if tdict['video']['blackwhite'] == False:
                        tdict['video']['blackwhite'] = (content.find('zwart/wit') != -1)
                    if tdict['teletekst'] == False:
                        tdict['teletekst'] = (content.find('teletekst') != -1)
                    if content.find('stereo') != -1: tdict['audio'] = 'stereo'
                    if tdict['rerun'] == False:
                        tdict['rerun'] = (content.find('herhaling') != -1)

                elif ctype == 'nl-url':
                    tdict['infourl'] = content

                elif (ctype not in tdict) and (ctype.lower() not in ('zender', 'datum', 'uitzendtijd', 'titel', 'prijzen')):
                    # In unmatched cases, we still add the parsed type and content to the program details.
                    # Some of these will lead to xmltv output during the xmlefy_programs step
                    if config.write_info_files:
                        infofiles.addto_detail_list(unicode('new tvgids.nl detail => ' + ctype + ': ' + content))

                    tdict[ctype] = content

            except:
                log(traceback.format_exc())
                if config.write_info_files:
                    infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                    infofiles.write_raw_string(strdata)

        tdict['ID'] = tdict['prog_ID'][self.proc_id]
        tdict[self.detail_check] = True
        return tdict

    def load_json_detailpage(self, tdict):
        try:
            # We first get the json url
            url = 'http://www.tvgids.nl/json/lists/program.php?id=%s' % tdict['prog_ID'][self.proc_id][3:]
            strdata = config.get_page(url, 'utf-8')
            if strdata == None or strdata.replace('\n','') == '{}':
                return None

            strdata = self.check_text_subs(strdata)
            detail_data = json.loads(strdata)

        except:
            # if we cannot find the description page,
            # go to next in the loop
            return None

        for ctype, content in detail_data.items():
            if ctype in ('db_id', 'titel', 'datum', 'btijd', 'etijd', 'zender_id'):
                # We allready have these or we don use them
                continue

            if content == '':
                continue

            if ctype == 'genre':
                tdict['genre'] = content

            elif  ctype == 'kijkwijzer':
                for k in content:
                    if k in config.kijkwijzer.keys() and k not in tdict['kijkwijzer']:
                        tdict['kijkwijzer'].append(k)

            elif ctype == 'synop':
                content = re.sub('<p>', '', content)
                content = re.sub('</p>', '', content)
                content = re.sub('<br/>', '', content)
                content = re.sub('<strong>.*?</strong>', '', content)
                content = re.sub('<.*?>', '', content)
                content = re.sub('\\r\\n', '\\n', content)
                content = re.sub('\\n\\n\\n', '\\n', content)
                content = re.sub('\\n\\n', '\\n', content)
                if tdict['subgenre'].lower().strip() == content[0:len(tdict['subgenre'])].lower().strip():
                    content = content[len(tdict['subgenre'])+1:]

                if content > tdict['description']:
                    tdict['description'] = self.unescape(content)

                if config.channels[tdict['channelid']].opt_dict['prefered_description'] == self.proc_id:
                    tdict['prefered description'] = tdict['description']

            # Parse persons and their roles for credit info
            elif ctype in config.roletrans:
                if not config.roletrans[ctype] in tdict['credits']:
                    tdict['credits'][config.roletrans[ctype]] = []
                persons = content.split(',');
                for name in persons:
                    if name.find(':') != -1:
                        name = name.split(':')[1]

                    if name.find('-') != -1:
                        name = name.split('-')[0]

                    if name.find('e.a') != -1:
                        name = name.split('e.a')[0]

                    if not self.unescape(name.strip()) in tdict['credits'][config.roletrans[ctype]]:
                        tdict['credits'][config.roletrans[ctype]].append(self.unescape(name.strip()))

            else:
                if config.write_info_files:
                    infofiles.addto_detail_list(unicode('new tvgids.nl json detail => ' + ctype + ': ' + content))

        tdict['ID'] = tdict['prog_ID'][self.proc_id]
        tdict[self.detail_check] = True
        return tdict

# end tvgids_JSON

class tvgidstv_HTML(FetchData):
    """
    Get all available days of programming for the requested channels
    from the tvgids.tv page. Based on FetchData Class
    """
    def init_channels(self):
        """ General Site layout
            <head>
            <body><div id="wrap"><div class="container"><div class="row">
                            <div class="span16">
                            <div class="span47 offset1">
                                een of meer
                                <div class="section">
                                    ...
                            <div class="span30 offset1">
                <div id="footer">

        Channel listing:
            <div class="section-title">
                contains the grouping name (Nederlands, Vlaams, ...)
            </div>
            <div class="section-content"><div class="section-item channels"><div class="section-item-content">
                        each contain groupings of up to four channels
                        <a href="/zenders/nederland-1" title="TV Gids NPO 1" class="">
                            <div class="channel-icon sprite-channel-1"></div><br />
                           <div class="channel-name ellipsis">NPO 1</div>
                        </a>
            </div></div></div>

        Program listing:
            <div class="section-content">
                contains for each program
                <a href="/tv/hart-van-nederland" title="Hart van Nederland"></a>
                <a href="/tv/hart-van-nederland/12568262" title="Hart van Nederland" class="section-item posible-progress-bar" rel="nofollow">
                    <div class="content">
                        <div class="channel-icon sprite-channel-8"></div>
                        <span class="section-item-title">
                                                                05:25
                                                                Hart van Nederland
                        </span>
                        <div class="clearfix"></div>
                    </div>
                </a>
            </div>

        Detail layout
            <div class="section-title">
                <h1>Navy NCIS</h1>
                <a class="channel-icon sprite-channel-8 pull-right" href="/zenders/net-5" title="TV Gids NET 5"></a>
            </div>
            <div class="section-content">
                <div class="section-item gray">
                    <img class="pull-right large" src="http://images.cdn.tvgids.tv/programma/square_iphone_hd_TVGiDStv_navy-ncis.jpg" alt="Navy NCIS" title="Navy NCIS" />
                    <dl class="dl-horizontal program-details">
                        <dt>Datum</dt><dd>Ma 29 december 2014 </dd>
                        <dt>Tijd</dt><dd>19:35 tot 20:30</dd>
                        <dt>    Name    </dt><dd>    Content    </dd>
                                   ...
                    </dl>
                    <div class="program-details-social">
                        ...
                    </div>
                    <p>                description                     </p>
                </div>
            </div>
        """

        self.init_channel_source_ids()

    def get_url(self, channel = None, offset = 0, href = None):

        if href == None and channel == None:
            return u'%s/zenders/' % config.source_base_url[self.proc_id]

        if href == None:
            return u'%s/zenders/%s/%s' % (config.source_base_url[self.proc_id], channel, offset)

        if href == '':
            return ''

        else:
            return u'%s%s' % (config.source_base_url[self.proc_id], self.unescape(href))

    def check_date(self, page_data, channel, offset):

        # Check on the right offset for appending the date to the time. Their date switch is aroud 6:00
        dnow = datetime.datetime.now(CET_CEST).strftime('%d %b').split()
        dlast = datetime.date.fromordinal(self.current_date - 1).strftime('%d %b').split()

        if page_data == None:
            log("Skip channel=%s on tvgids.tv!, day=%d. No data\n" % (channel, offset))
            return None

        d = self.get_source_regex(page_data, 0)
        if d == None:
            log('Unable to veryfy the right offset on .\n' )
            return None

        try:
            d = d.group(1)
            d = self.clean_html(d)
            htmldata = ET.fromstring( ('<div>' + d).encode('utf-8'))

        except:
            log('Unable to veryfy the right offset on .\n' )
            return None

        dd = htmldata.find('div/a[@class="today "]/br')
        if dd == None:
            dd = htmldata.find('div/a[@class="today"]/br')

        if dd == None:
            dd = htmldata.find('div/a[@class="today active"]/br')

        if dd.tail == None:
            log('Unable to veryfy the right offset on .\n' )
            return None

        d = dd.tail.strip().split()
        if int(dnow[0]) == int(d[0]):
            return offset

        elif int(dlast[0]) == int(d[0]):
            return offset - 1

        else:
            log("Skip channel=%s, day=%d. Wrong date!\n" % (channel, offset))
            return None

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        try:
            strdata = config.get_page(self.get_url())
            if strdata == None:
                self.fail_count += 1
                return

            strdata = self.clean_html('<div>' + self.get_source_regex(strdata, 1, 1, 1))
            htmldata = ET.fromstring(strdata.encode('utf-8'))

        except:
            self.fail_count += 1
            log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
            return 69  # EX_UNAVAILABLE

        self.all_channels ={}
        for changroup in htmldata.findall('div[@class="section"]'):
            group_name = self.empersant(changroup.findtext('div[@class="section-title"]')).strip()
            for chan in changroup.findall('div[@class="section-content"]/div[@class="section-item channels"]/div[@class="section-item-content"]/a'):
                channelid = chan.get('href')
                if channelid == None:
                    continue

                channelid = re.split('/', channelid)[2]
                name = self.empersant(chan.findtext('div[@class="channel-name ellipsis"]'))
                self.all_channels[channelid] = {}
                self.all_channels[channelid]['name'] = name
                self.all_channels[channelid]['group'] = 99
                for id in config.group_order:
                    if group_name == config.chan_groups[id]:
                        self.all_channels[channelid]['group'] = id
                        break

    def match_genre(self, dtext, tdict):
        if len(dtext) > 20:
            tdict['genre'] = u'overige'
            return tdict

        dtext = dtext.strip().strip('.')
        if dtext.lower() in config.source_cattrans[self.proc_id].keys():
            tdict['genre'] = config.source_cattrans[self.proc_id][dtext.lower()].capitalize()
            tdict['subgenre'] = dtext

        # Now we try to match the genres not found in source_cattrans[self.proc_id]
        else:
            if 'jeugd' in dtext.lower():
                tdict['genre'] = u'Jeugd'

            elif 'muziek' in dtext.lower():
                tdict['genre'] = u'Muziek'

            elif 'sport' in dtext.lower():
                tdict['genre'] = u'Sport'

            elif 'nieuws' in dtext.lower():
                tdict['genre'] = u'Nieuws/Actualiteiten'

            elif 'natuur' in dtext.lower():
                tdict['genre'] = u'Natuur'

            elif 'cultuur' in dtext.lower():
                tdict['genre'] = u'Kunst en Cultuur'

            elif 'kunst' in dtext.lower():
                tdict['genre'] = u'Kunst en Cultuur'

            elif 'wetenschap' in dtext.lower():
                tdict['genre'] = u'Wetenschap'

            elif 'medisch' in dtext.lower():
                tdict['genre'] = u'Wetenschap'

            elif 'film' in dtext.lower():
                tdict['genre'] = u'Film'

            elif 'spel' in dtext.lower():
                tdict['genre'] = u'Amusement'

            elif 'show' in dtext.lower():
                tdict['genre'] = u'Amusement'

            elif 'quiz' in dtext.lower():
                tdict['genre'] = u'Amusement'

            elif 'praatprogramma' in dtext.lower():
                tdict['genre'] = u'Magazine'

            elif 'magazine' in dtext.lower():
                tdict['genre'] = u'Magazine'

            elif 'documentair' in dtext.lower():
                tdict['genre'] = u'Informatief'

            elif 'serie' in dtext.lower():
                tdict['genre'] = u'Serie/Soap'

            elif 'soap' in dtext.lower():
                tdict['genre'] = u'Serie/Soap'

            elif 'drama' in dtext.lower():
                tdict['genre'] = u'Serie/Soap'

            elif 'thriller' in dtext.lower():
                tdict['genre'] = u'Serie/Soap'

            elif 'komedie' in dtext.lower():
                tdict['genre'] = u'Serie/Soap'

            elif 'western' in dtext.lower():
                tdict['genre'] = u'Serie/Soap'

            else:
                tdict['genre'] = u'overige'
                if config.write_info_files and not tdict['channelid'] in ('29', '438',):
                    infofiles.addto_detail_list(unicode('unknown tvgids.tv genre => ' + dtext + ' on ' + tdict['channel']))

            if not tdict['channelid'] in ('29', '438',):
                tdict['subgenre'] = dtext
                # And add them to source_cattrans[self.proc_id] (and tv_grab_nl_py.set for later reference
                # But not for Discovery Channel or TLC as that is garbage
                if not tdict['genre'] == u'overige':
                    config.new_cattrans[self.proc_id].append((dtext.lower().strip(), tdict['genre']))

        return tdict

    def load_pages(self):
        failure_count = 0
        first_fetch = True
        try:
            for retry in (0, 1):
                channel_cnt = 0
                for channelid, chanid in self.chanids.items():
                    channel_cnt += 1
                    failure_count = 0
                    if self.quit:
                        return

                    if config.channels[chanid].source_data[self.proc_id].is_set():
                        continue

                    # Start from the offset but skip the days allready fetched by tvgids.nl
                    # Except when append_tvgidstv is False
                    nlid = config.channels[chanid].get_source_id(0)
                    if config.channels[chanid].opt_dict['append_tvgidstv']:
                        fetch_range = []
                        for i in range( config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days'])):
                            if not nlid in xml_output.channelsource[0].day_loaded or not xml_output.channelsource[0].day_loaded[nlid][i]:
                                fetch_range.append(i)

                    else:
                        fetch_range = range( config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days']))

                    if len(fetch_range) == 0:
                        self.set_ready(channelid)
                        continue

                    # Tvgids.tv shows programs per channel per day, so we loop over the number of days
                    # we are required to grab
                    for offset in fetch_range:
                        # Check if it is allready loaded
                        if self.day_loaded[channelid][offset] != False or \
                          (config.channels[chanid].opt_dict['append_tvgidstv'] and \
                          nlid in xml_output.channelsource[0].day_loaded and \
                          xml_output.channelsource[0].day_loaded[nlid][offset]):
                            continue

                        log(['\n', 'Now fetching %s(xmltvid=%s%s) from tvgids.tv\n' % \
                            (config.channels[chanid].chan_name, config.channels[chanid].xmltvid , config.channels[chanid].get_opt('compat')), \
                            '    (channel %s of %s) for day %s of %s.\n' % \
                            (channel_cnt, len(self.chanids), offset, config.opt_dict['days'])], 2)
                        if not first_fetch:
                            # be nice to tvgids.tv
                            time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                            first_fetch = false

                        # get the raw programming for the day
                        try:
                            channel_url = self.get_url(channelid, offset)
                            strdata = config.get_page(channel_url)

                            if strdata == None:
                                if retry == 0:
                                    log("Skip channel=%s on tvgids.tv, day=%d. No data! First attempt.\n" % (config.channels[chanid].chan_name, offset))
                                else:
                                    log("Skip channel=%s on tvgids.tv, day=%d. No data! Final attempt.\n" % (config.channels[chanid].chan_name, offset))

                                failure_count += 1
                                self.fail_count += 1
                                continue

                        except:
                            log('Error: "%s" reading the tvgids.tv basepage for channel=%s, day=%d.\n' %
                                (sys.exc_info()[1], config.channels[chanid].chan_name, offset))
                            failure_count += 1
                            self.fail_count += 1
                            continue


                        # Check on the right offset for appending the date to the time. Their date switch is aroud 6:00
                        x = self.check_date(strdata, config.channels[chanid].chan_name, offset)
                        if x == None:
                            log("Skip channel=%s on tvgids,tv, day=%d. Wrong date!\n" % (config.channels[chanid].chan_name, offset))
                            failure_count += 1
                            self.fail_count += 1
                            continue

                        date_offset = x
                        scan_date = datetime.date.fromordinal(self.current_date + date_offset)
                        last_program = datetime.datetime.combine(datetime.date.fromordinal(self.current_date + date_offset - 1), datetime.time(0, 0, 0 ,0 ,CET_CEST))

                        # and extract the ElementTree
                        try:
                            strdata = self.get_source_regex(strdata, 2, 1, 1)
                            strdata = self.clean_html(strdata)
                            strdata = self.check_text_subs(strdata)
                            htmldata = ET.fromstring( ('<div><div>' + strdata).encode('utf-8'))

                        except:
                            log(["Error extracting ElementTree for channel:%s day:%s on tvgids.tv\n" % \
                                (config.channels[chanid].chan_name, offset), \
                                "Possibly an incomplete pagefetch. Retry in the early morning after 4/5 o'clock.\n"])

                            if config.write_info_files:
                                infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                                infofiles.write_raw_string('<div><div>' + strdata + '\n')

                            failure_count += 1
                            self.fail_count += 1
                            self.day_loaded[channelid][offset] = None
                            continue

                        try:
                            if htmldata.find('div/a[@class]') == None:
                                log(["No Programming for channel=%s, day=%d on tvgids.tv!\n" % (config.channels[chanid].chan_name, offset), \
                                        "   We assume further pages to be empty!\n"])

                                for d in range((offset - 1), config.opt_dict['days']):
                                    self.day_loaded[channelid][d] = None

                                continue

                            for p in htmldata.findall('div/a[@class]'):
                                tdict = self.checkout_program_dict()
                                tdict['source'] = u'tvgidstv'
                                tdict['channelid'] = chanid
                                tdict['channel'] = config.channels[chanid].chan_name
                                tdict['detail_url'][self.proc_id] = self.get_url(href = p.get('href'))
                                tdict['prog_ID'][self.proc_id] = u'tv-%s' % tdict['detail_url'][self.proc_id].split('/')[5]  if (tdict['detail_url'][self.proc_id] != '') else ''

                                # The Title
                                tdict['name'] = self.empersant(p.get('title'))
                                tdict = self.check_title_name(tdict)
                                if  tdict['name'] == None or tdict['name'] == '':
                                    log('Can not determine program title for "%s"\n' % tdict['detail_url'][self.proc_id])
                                    continue

                                # Get the starttime and make sure the midnight date change is properly crossed
                                start = p.findtext('div[@class="content"]/span[@class="section-item-title"]').split()[0]
                                if start == None or start == '':
                                    log('Can not determine starttime for "%s"\n' % tdict['name'])
                                    continue

                                prog_time = datetime.time(int(start.split(':')[0]), int(start.split(':')[1]), 0 ,0 ,CET_CEST)
                                if datetime.datetime.combine(scan_date, prog_time) < last_program:
                                    date_offset = date_offset +1
                                    scan_date = datetime.date.fromordinal(self.current_date + date_offset)

                                tdict['offset'] = date_offset
                                tdict['start-time'] = datetime.datetime.combine(scan_date, prog_time)
                                last_program = tdict['start-time']

                                m = p.findtext('div[@class="content"]/span[@class="label"]')
                                # span = "IMDB * n.n"
                                if m != None:
                                    dd = unicode(m.split(':')[1])
                                    if dd != '':
                                        tdict['star-rating'] = dd

                                d = p.findtext('div[@class="content"]/p')
                                # p      = "dd/mm - IMDB * n.n - <genre>, beschrijving"
                                if d != None:
                                    dd = d.split(',')
                                    tdict['description'] = self.empersant(d[len(dd[0])+1:]).strip()
                                    dd = self.empersant(dd[0]).split('-')
                                    tdict = self.match_genre(self.empersant(unicode(dd[-1])), tdict)

                                    if tdict['star-rating'] == '' and len(dd) > 1:
                                        ddd = dd[-2].split('*')
                                        if ddd[0].strip() == 'IMDB':
                                            tdict['star-rating'] = unicode(ddd[1].strip())

                                # and append the program to the list of programs
                                with self.source_lock:
                                    self.program_data[channelid].append(tdict)

                        except:
                            log(['Error processing tvgids.tv data for channel:%s day:%s\n' % \
                                (config.channels[chanid].chan_name, offset), traceback.format_exc()])
                            self.fail_count += 1
                            continue

                        self.base_count += 1
                        self.day_loaded[channelid][offset] = True

                    if len(self.program_data[channelid]) == 0:
                        self.set_ready(channelid)
                        continue

                    # Add starttime of the next program as the endtime
                    with self.source_lock:
                        self.program_data[channelid].sort(key=lambda program: (program['start-time']))
                        self.add_endtimes(channelid, 6)

                        for tdict in self.program_data[channelid]:
                            self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

                    if failure_count == 0 or retry == 1:
                        self.parse_programs(chanid, 0, 'None')
                        self.set_ready(channelid)
                        try:
                            infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                        except:
                            pass

        except:
            log(['\n', 'An unexpected error has occured in the %s thread:\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()

    def load_detailpage(self, tdict):

        try:
            strdata = config.get_page(tdict['detail_url'][self.proc_id])
            if strdata == None:
                return

            strdata = self.clean_html('<root><div><div class="section-title">' + self.get_source_regex(strdata, 3, 1, 1) + '</root>')
            strdata = self.check_text_subs(strdata)

        except:
            log(['Error Fetching detailpage %s\n' % tdict['detail_url'][self.proc_id], traceback.format_exc()])
            return None

        try:
            htmldata = ET.fromstring(strdata.encode('utf-8'))

        except:
            log("Error extracting ElementTree from:%s on tvgids.tv\n" % (tdict['detail_url'][self.proc_id]))
            if config.write_info_files:
                infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                infofiles.write_raw_string(strdata + u'\n')

            return None

        # We scan every alinea of the description
        try:
            tdict = self.filter_description(htmldata, 'div/div/div/p', tdict)
            if config.channels[tdict['channelid']].opt_dict['prefered_description'] == self.proc_id:
                tdict['prefered description'] = tdict['description']

        except:
            log(['Error processing the description from: %s\n' % (tdict['detail_url'][self.proc_id]), traceback.format_exc()])

        data = htmldata.find('div/div[@class="section-content"]')
        datatype = u''
        try:
            for d in data.find('div/dl'):
                if d.tag == 'dt':
                    datatype = self.empersant(d.text.lower())

                elif d.tag == 'dd':
                    dtext = self.empersant(d.text).strip() if (d.text != None) else ''
                    if datatype in ('datum', 'tijd', 'uitzending gemist', 'officiële twitter', 'twitter hashtag', 'deel-url'):
                        continue

                    elif datatype == 'genre':
                        if dtext == '':
                            continue

                        tdict = self.match_genre(dtext, tdict)

                    elif datatype == 'jaar':
                        tdict['jaar van premiere'] = dtext

                    elif datatype in config.roletrans:
                        tdict['credits'][config.roletrans[datatype]] = []
                        persons = dtext.split(',');
                        for name in persons:
                            if name.find(':') != -1:
                                name = name.split(':')[1]

                            if name.find('-') != -1:
                                name = name.split('-')[0]

                            if name.find('e.a') != -1:
                                name = name.split('e.a')[0]

                            tdict['credits'][config.roletrans[datatype]].append(name.strip())

                    elif datatype == 'imdb':
                        dd = d.find('a')
                        if dd == None:
                            continue

                        durl = self.empersant(dd.get('href', ''))
                        if durl != '':
                            tdict['infourl'] = durl

                        stars = unicode(dd.text.strip())
                        if stars != '' and tdict['star-rating'] == '':
                            tdict['star-rating'] = stars

                    elif datatype== 'officiële website':
                        if d.find('a') == None:
                            continue

                        durl = self.empersant(d.find('a').get('href', ''))
                        if durl != '':
                            tdict['infourl'] = durl

                    elif datatype== 'kijkwijzer':
                        kw_val = d.find('div')
                        if kw_val != None:
                            kw_val = kw_val.get('class').strip()

                        if kw_val != None and len(kw_val) > 27:
                            kw_val = kw_val[27:]
                            if kw_val in config.tvkijkwijzer.keys():
                                if config.tvkijkwijzer[kw_val] not in tdict['kijkwijzer']:
                                    tdict['kijkwijzer'].append(config.tvkijkwijzer[kw_val])

                            elif config.write_info_files:
                                infofiles.addto_detail_list(unicode('new tvgids.tv kijkwijzer detail => ' + datatype + '=' + kw_val))

                    else:
                        if dtext != '':
                            if config.write_info_files:
                                infofiles.addto_detail_list(unicode('new tvgids.tv text detail => ' + datatype + '=' + dtext))

                            tdict[datatype] = dtext

                        elif d.find('div') != None and d.find('div').get('class') != None:
                            if config.write_info_files:
                                infofiles.addto_detail_list(unicode('new tvgids.tv div-class detail => ' + datatype + '=' + d.find('div').get('class')))

                            tdict[datatype] = unicode(d.find('div').get('class'))

                        elif d.find('a') != None and d.find('a').get('href') != None:
                            if config.write_info_files:
                                infofiles.addto_detail_list(unicode('new tvgids.tv a-href detail => ' + datatype + '=' + d.find('a').get('href')))

                            tdict[datatype] = unicode(d.find('a').get('href'))

                        elif config.write_info_files:
                            infofiles.addto_detail_list(unicode('new tvgids.tv empty detail => ' + datatype))

                elif config.write_info_files:
                    infofiles.addto_detail_list(unicode('new tvgids.d-tag => ' + d.tag))

        except:
            log(['Error processing tvgids.tv detailpage:%s\n' % (tdict['detail_url'][self.proc_id]), traceback.format_exc()])
            return

        tdict['ID'] = tdict['prog_ID'][self.proc_id]
        tdict[self.detail_check] = True

        return tdict

# end tvgidstv_HTML

class rtl_JSON(FetchData):
    """
    Get all available days of programming for the requested channels
    from the rtl.nl json page. Based on FetchData
    """
    def init_channels(self):
        """ json Layout
            {
            "schedule": [
                {"abstract_key":"       ","season_key":"        ","episode_key":"       ","station":"   ","rerun":false,"unixtime":1421278680}, ...
            ],
            "library": [{
                "abstracts": [
                    {"abstract_key":"   ","name":"Up All Night"}, ...
                ],
                "seasons": [
                    {"season_key":"273426","season_number":"1","name":"Seizoen 1"}, ...
                ],
                "episodes": [
                    {"episode_key":"    ","episode_number":"10","name":"Week off","nicam":"ALt","synopsis":"                    ."}, ...
                ]}
            ]}
        """

        self.page_loaded = False
        self.schedule = {}

        self.init_channel_source_ids()
        for sourceid in self.chanids.keys():
            self.schedule[sourceid] =[]

    def init_json(self):

        self.json_by_id = {}
        self.jsondata = {}
        self.jsondict = {}
        self.jsondict['abstracts'] = {}
        self.jsondict['seasons'] = {}
        self.jsondict['episodes'] = {}
        self.jsondata = {'abstract_name': {'listname':'abstracts','keyname':'abstract_key','valuename':'name'}, \
                                   'season':                {'listname':'seasons','keyname':'season_key','valuename':'season_number'}, \
                                   'season_name':      {'listname':'seasons','keyname':'season_key','valuename':'name'}, \
                                   'episode':              {'listname':'episodes','keyname':'episode_key','valuename':'episode_number'}, \
                                   'episode_name':    {'listname':'episodes','keyname':'episode_key','valuename':'name'}, \
                                   'description':      {'listname':'episodes','keyname':'episode_key','valuename':'synopsis'}, \
                                   'nicam':                  {'listname':'episodes','keyname':'episode_key','valuename':'nicam'}}

    def get_url(self, abstract = None, days = 0):

        rtl_general = config.source_base_url[self.proc_id] + 'guide_for_one_day.xml?output=json'
        rtl_abstract = config.source_base_url[self.proc_id] + 'guide_for_one_abstract.xml?output=json'

        if abstract == None:
            channels = ''
            for channelid in self.chanids.keys():
                if len(channels) == 0:
                    channels = channelid

                else:
                    channels = '%s,%s' % (channels, channelid)

            return '%s&days_ahead=%s&days_back=%s&station=%s' % \
                ( rtl_general, (config.opt_dict['offset'] + config.opt_dict['days'] -1), - config.opt_dict['offset'], channels)

        else:
            return '%s&abstract_key=%s&days_ahead=%s' % ( rtl_abstract, abstract, days)

    def get_channels(self):
        self.all_channels = config.rtl_channellist

    def load_pages(self):

        if len(self.chanids) == 0 :
            return

        log(['\n', 'Now fetching %s channels from rtl.nl for %s days.\n' %  (len(self.chanids), config.opt_dict['days'])], 2)

        channel_url = self.get_url()

        # be nice to rtl.nl
        time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))

        # get the raw programming for the day
        strdata = config.get_page(channel_url, 'utf-8')

        if strdata == None or strdata.replace('\n','') == '{}':
            # Wait a while and try again
            time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
            strdata = config.get_page(channel_url, 'utf-8')
            if strdata == None or strdata.replace('\n','') == '{}':
                log("Error loading rtl json data\n")
                self.fail_count += 1
                self.set_ready()
                return False

        # Just let the json library parse it.
        strdata = self.check_text_subs(strdata)
        total = json.loads(strdata)
        self.base_count += 1
        # and find relevant programming info
        schedules = total['schedule']
        for r in schedules:
            self.schedule[r['station']].append(r)

        library = total['library'][0]

        for i in library['abstracts']:
            self.jsondict['abstracts'][i['abstract_key']] = i

        for i in library['seasons']:
           self.jsondict['seasons'][i['season_key']] = i

        for i in library['episodes']:
            self.jsondict['episodes'][i['episode_key']] = i

        self.page_loaded = True

        for channelid, chanid in self.chanids.items():
            if len( self.schedule[channelid]) == 0:
                config.channels[chanid].source_id[self.proc_id] = ''
                continue

            for item in self.schedule[channelid]:
                tdict = self.checkout_program_dict()
                tdict['prog_ID'][self.proc_id] = u'%s-%s' % (channelid,  item['unixtime'])
                self.json_by_id[tdict['prog_ID'][self.proc_id]] = item
                tdict['source'] = 'rtl'
                tdict['channelid'] = chanid
                tdict['channel']  = config.channels[chanid].chan_name

                # The Title
                tdict['name'] = self.get_json_data(tdict['prog_ID'][self.proc_id],'abstract_name')
                if  tdict['name'] == None or tdict['name'] == '':
                    log('Can not determine program title\n')
                    continue

                # The timing
                tdict['unixtime']  =int( item['unixtime'])
                tdict['start-time']  = datetime.datetime.fromtimestamp(tdict['unixtime'], CET_CEST)
                tdict['offset'] = self.get_offset(tdict['start-time'])
                tdict['rerun']  = (item['rerun'] == 'true')

                # The Season Number
                season = self.get_json_data(tdict['prog_ID'][self.proc_id],'season')
                tdict['season'] = int(season) if (season != None) else 0

                # The Episode Number, SubTitle and Descriptionseason
                episode = self.get_json_data(tdict['prog_ID'][self.proc_id],'episode')
                tdict['episode'] = int(episode) if (episode != None) else 0

                subtitle = self.get_json_data(tdict['prog_ID'][self.proc_id],'episode_name')
                tdict['titel aflevering'] = subtitle if ((subtitle != None) and (subtitle != tdict['name'])) else ''
                tdict = self.check_title_name(tdict)

                description = self.get_json_data(tdict['prog_ID'][self.proc_id],'description')
                tdict['description'] = description if (description != None) else ''

                nicam = self.get_json_data(tdict['prog_ID'][self.proc_id],'nicam')
                if '16' in nicam:
                    tdict['kijkwijzer'].append('4')

                elif '12' in nicam:
                    tdict['kijkwijzer'].append('3')

                elif '9' in nicam:
                    tdict['kijkwijzer'].append('9')

                elif '6' in nicam:
                    tdict['kijkwijzer'].append('2')

                elif 'AL' in nicam:
                    tdict['kijkwijzer'].append('1')

                for k in ('g', 'a', 's', 't', 'h', 'd'):
                    if k in nicam:
                        tdict['kijkwijzer'].append(k)

                with self.source_lock:
                    self.program_data[channelid].append(tdict)

            # Add starttime of the next program as the endtime
            with self.source_lock:
                self.program_data[channelid].sort(key=lambda program: (program['start-time']))
                self.add_endtimes(channelid, 7)

                for tdict in self.program_data[channelid]:
                    self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

            self.parse_programs(chanid, 0, 'None')
            self.set_ready(channelid)
            for day in range( config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days'])):
                self.day_loaded[channelid][day] = True

            try:
                infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

            except:
                pass

# end rtl_JSON

class npo_HTML(FetchData):
    """
    Get all available days of programming for the requested channels
    from the npo.nl page. Based on FetchData Class
    """
    def init_channels(self):
        """ General Site layout
            <div class='guides-overlay overlay'></div>
            <div class='row-fluid'>
                <div class='span12'>
                    <div         class='vertical-guide'
                                    data-counter='?npo-nl.gids.verticaal.20150526'
                                    data-end='Wed, 27 May 2015 05:59:59 +0200'
                                    data-keyboard-input='true'
                                    data-scorecard='{"prefix":"npo","name":"gids.verticaal.26-05-2015"}'
                                    data-slide-increment='540'
                                    data-start='Tue, 26 May 2015 06:00:00 +0200'
                                    id='primary-guide'>
                        <div class='guide-scroller'>
                            <div class='vertical-guide-wrapper'>
                                <ul class='scroll-header'>
                                    <li>
                                        <a href="/live/npo-1" title="Bekijk live!">
                                            <div alt='Logo van NPO 1'
                                                    class='channel-logo'
                                                    style="background-image: url('//assets.www.npo.nl/uploads/tv_channel/263/logo/regular_logo-npo1.png')">
                                            </div>
                                                NPO 1
                                        </a>
                                    </li>
                                        ...
                                    <li class='ttv'>
                                        <div alt='Logo van NPO Nieuws'
                                                class='channel-logo'
                                                style="background-image: url('//assets.www.npo.nl/uploads/tv_channel/279/guide_label/regular_nponieuws-klein.png')">
                                        </div>
                                            NPO Nieuws
                                    </li>
                                        ...
                                    <li class='rtv'>
                                        <div alt='Logo van Regio TV Utrecht'
                                                class='channel-logo'
                                                style="background-image: url('//assets.www.npo.nl/uploads/tv_channel/273/logo/regular_rtvutrecht.png')">
                                        </div>
                                            Regio TV Utrecht
                                    </li>
                                </ul>
                                <table>
                                    <tr class='odd' data-hour='6'>          ('6' - '5')
                                        <td class='padder left'></td>
                                        <td class='red'>                             ('red', 'blue', 'green', 'ttv'..., 'rtv'...)
                                            <a           href="/nederland-in-beweging/25-05-2015/POW_00979881"
                                                            class="time-block inactive"
                                                            data-end-hour="06"
                                                            data-end-minutes="07"
                                                            data-genre="17"
                                                            data-start-hour="05"
                                                            data-start-minutes="53">
                                                <div class='time'>05:53</div>
                                                <div class='description'>
                                                    <i class='np'></i>
                                                    <div class='program-title'>Nederland in Beweging</div>
                                                </div>
                                            </a>
                                                ...
                                        </td>
                                                ...

                                        <td class='padder right'></td>
                                    <tr class='odd active' data-hour='1'>
                                        ...
                                    </tr>
                                </table>
                            </div>
                        </div>
                    </div>
                </div>
            </div>
        """

        self.init_channel_source_ids()

    def get_url(self, offset = 0, href = None, vertical = False):

        if href == None and vertical:
            scan_date = datetime.date.fromordinal(self.current_date + offset)
            return u'%s/gids/verticaal/%s/content' % (config.source_base_url[self.proc_id],  scan_date.strftime('%d-%m-%Y'))

        if href == None and not vertical:
            scan_date = datetime.date.fromordinal(self.current_date + offset)
            return u'%s/gids/horizontaal/%s/content' % (config.source_base_url[self.proc_id],  scan_date.strftime('%d-%m-%Y'))

        elif href == '':
            return ''

        else:
            return u'%s%s' % (config.source_base_url[self.proc_id],  href)

    def get_channels(self):
        try:
            strdata = config.get_page(self.get_url())
            strdata = self.clean_html(strdata)
            if strdata == None:
                self.fail_count += 1
                log(["Unable to get channel info from %s\n" % self.source])
                return 69  # EX_UNAVAILABLE

            strdata = self.check_text_subs(strdata)
            htmldata = ET.fromstring( (u'<root>\n' + strdata + u'\n</root>\n').encode('utf-8'))
            self.get_channel_lineup(htmldata)

        except:
            self.fail_count += 1
            log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
            return 69  # EX_UNAVAILABLE

    def get_channel_lineup(self, htmldata):
        chan_list = []
        channel_cnt = 0
        for c_grp in htmldata.findall('div'):
            if c_grp.find('div[@class="span12"]') == None:
                # The list of extra groups
                continue

            g_class = c_grp.get('class')
            g_id = c_grp.get('id')
            if  g_class == 'row-fluid':
                # The NPO base channels
                cgrp = 1

            elif g_id == 'themed-guide':
                # The NPO theme channels
                cgrp = 1

            elif g_id == 'regional-guide':
                # The Regional channels
                cgrp = 6

            elif g_id == 'radio-guide':
                # The Radio channels
                cgrp = 11

            else:
                # Unknown Group
                cgrp = 99

            for c in c_grp.findall('div[@class="span12"]/div/div[@class="guide-channel-icons"]/div[@class="channel-icon"]'):
                try:
                    if c.find('a') != None:
                        tag = c.find('a')
                        c = tag
                        if tag.get("alt") != None:
                            cname = self.empersant(tag.get("alt")[9:])

                        else:
                            cname = self.empersant(tag.get("href").split('/')[-1])

                    else:
                        cname = self.empersant(c.find('div').get("title"))

                    try:
                        cicon = c.find('div[@class="larger-image channel-icon-wrapper"]').get('style')

                    except:
                        cicon = c.find('div[@class="larger-image channel-icon-wrapper no-shadow"]').get('style')

                    cicon = cicon.split('/')
                    channelid = cicon[-3]
                    cicon = ('%s/%s/%s/%s' % (cicon[-4], cicon[-3], cicon[-2], cicon[-1]))[0:-2]
                    channel_cnt += 1
                    if channelid == '301':
                        #301: NPO Zapp = 265: NPO 3
                        channelid = '265'
                        cname = 'NPO 3'
                        cicon = 'tv-channel/265/logo/regular_logo-npo3.png'

                    self.all_channels[channelid] = {}
                    self.all_channels[channelid]['name'] = cname
                    self.all_channels[channelid]['group'] = cgrp
                    self.all_channels[channelid]['icongrp'] = 7
                    self.all_channels[channelid]['icon'] = cicon
                    chan_list.append(channelid)

                except:
                    log(['An error ocured while reading NPO channel info.', traceback.format_exc()])
                    continue

        return chan_list

    def load_pages(self):

        first_fetch = True
        max_offset = min((config.opt_dict['offset'] + config.opt_dict['days']), 7)
        def get_programs(xml, chanid, offset, omroep = True):
            try:
                channelid = self.channels[chanid]
                tdict = None
                day_offset = 0
                for p in xml.findall('a'):
                    if tdict != None:
                        with self.source_lock:
                            self.program_data[channelid].append(tdict)

                    ptext = p.find('i[@class="np"]')
                    if ptext == None:
                        # No title Found
                        continue

                    ptime = p.get('data-time')
                    if ptime == None:
                        # No start-stop time Found
                        continue

                    tdict = self.checkout_program_dict()
                    tdict['source'] = u'npo'
                    tdict['channelid'] = chanid
                    tdict['channel'] = config.channels[chanid].chan_name
                    tdict['detail_url'][self.proc_id] = self.get_url(href = p.get('href',''))
                    if tdict['detail_url'][self.proc_id] != '':
                        pid = tdict['detail_url'][self.proc_id].split('/')[-1]
                        tdict['prog_ID'][self.proc_id] = u'npo-%s' % pid.split('_')[-1]

                    # The Title
                    tdict['name'] = self.empersant(ptext.tail.strip())

                    ptime = ptime.split('-')
                    pstart = ptime[0].split(':')
                    prog_time = datetime.time(int(pstart[0]), int(pstart[1]), 0 ,0 ,CET_CEST)
                    if day_offset == 0 and int(pstart[0]) < 6:
                        day_offset = 1

                    tdict['offset'] = offset + day_offset

                    if day_offset == 1:
                        tdict['start-time'] = datetime.datetime.combine(nextdate, prog_time)

                    else:
                        tdict['start-time'] = datetime.datetime.combine(startdate, prog_time)

                    pstop = ptime[1].split(':')
                    prog_time = datetime.time(int(pstop[0]), int(pstop[1]), 0 ,0 ,CET_CEST)
                    if day_offset == 1 or int(pstop[0]) < 6:
                        tdict['stop-time'] = datetime.datetime.combine(nextdate, prog_time)

                    else:
                        tdict['stop-time'] = datetime.datetime.combine(startdate, prog_time)

                    if omroep:
                        tdict['omroep'] = p.findtext('span', '')

                    pgenre = p.get('data-genre','')
                    if pgenre != None and pgenre !=  '':
                        pgenre = pgenre.lower()
                        pg = pgenre.split(',', 1)
                        if len(pg) == 1:
                            pg = (pg[0].strip(), )

                        elif len(pg) == 2:
                            pg = (pg[0].strip(), pg[1].strip())

                        if pg in config.source_cattrans[self.proc_id].keys():
                            tdict['genre'] = config.source_cattrans[self.proc_id][pg][0].capitalize()
                            tdict['subgenre'] = config.source_cattrans[self.proc_id][pg][1].capitalize()

                        else:
                            if len(pg) > 1 and (pg[0].lower(), ) in config.source_cattrans[self.proc_id].keys():
                                tdict['genre'] = config.source_cattrans[self.proc_id][(pg[0].lower(), )][0].capitalize()
                                tdict['subgenre'] = config.source_cattrans[self.proc_id][(pg[0].lower(), )][1].capitalize()
                                config.new_cattrans[self.proc_id][(pg[0], pg[1])] = config.source_cattrans[self.proc_id][(pg[0].lower(), )]

                            else:
                                tdict['genre'] = u'overige'
                                if len(pg) == 2:
                                    tdict['subgenre'] = pg[1].capitalize()
                                    config.new_cattrans[self.proc_id][pg] = (u'Overige', pg[1])

                                else:
                                    config.new_cattrans[self.proc_id][pg] = (u'Overige', u'')

                            if config.write_info_files and pgenre != '':
                                infofiles.addto_detail_list(unicode('unknown npo.nl genre => ' + pgenre + ': ' + tdict['name']))

                    else:
                        tdict['genre'] = u'overige'

                    # and append the program to the list of programs
                    tdict = self.check_title_name(tdict)
                    if first_added[offset][chanid] == None:
                        # It's the first program of the day
                        first_added[offset][chanid] = tdict
                        tdict = None

                last_added[offset][chanid] = tdict

            except:
                log(traceback.format_exc())

        if config.opt_dict['offset'] >= 7:
            self.set_ready()
            return

        if len(self.chanids) == 0 :
            return

        first_added = {}
        last_added = {}
        month_names = ['dummy', 'jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']
        for retry in (0, 1):
            for offset in range(config.opt_dict['offset'], max_offset):
                if self.quit:
                    return

                # Check if it is already loaded
                if self.day_loaded[0][offset]:
                    continue

                first_added[offset] = {}
                last_added[offset] = {}
                for chanid in self.chanids.values():
                    first_added[offset][chanid] = None
                    last_added[offset][chanid] = None

                log(['\n', 'Now fetching %s channels from npo.nl\n' % (len(self.chanids)), \
                    '    (day %s of %s).\n' % (offset, max_offset - config.opt_dict['offset'])], 2)

                channel_url = self.get_url(offset)

                if not first_fetch:
                    # be nice to npo.nl
                    time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                    first_fetch = false

                # get the raw programming for the day
                strdata = config.get_page(channel_url)
                if strdata == None or 'We hebben deze pagina niet gevonden...' in strdata:
                    log("No data on npo.nl for day=%d\n" % (offset))
                    self.fail_count += 1
                    continue

                try:
                    strdata = self.clean_html(strdata)
                    strdata = self.check_text_subs(strdata)
                    htmldata = ET.fromstring( (u'<root>\n' + strdata + u'\n</root>\n').encode('utf-8'))

                except:
                    log('Error extracting ElementTree for day:%s on npo.nl\n' % (offset))
                    self.fail_count += 1
                    if config.write_info_files:
                        infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                        infofiles.write_raw_string('<root>\n' + strdata + '\n</root>\n')

                    continue

                # First we get the line-up and some date checks
                self.base_count += 1
                try:
                    startdate = htmldata.find('div[@class="row-fluid"]/div[@class="span12"]/div').get('data-start')
                    nextdate = htmldata.find('div[@class="row-fluid"]/div[@class="span12"]/div').get('data-end')
                    if startdate == None or nextdate == None:
                        log('Error validating page for day:%s on npo.nl\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno, offset))
                        continue

                    d = (startdate.split(',')[-1].strip()).split(' ')
                    for i in range(len(month_names)):
                        if d[1].lower() == month_names[i]:
                            startdate = datetime.datetime.strptime('%s %2.0f %s' % (d[0], i, d[2]),'%d %m %Y').date()
                            break

                    d = (nextdate.split(',')[-1].strip()).split(' ')
                    for i in range(len(month_names)):
                        if d[1].lower() == month_names[i]:
                            nextdate = datetime.datetime.strptime('%s %2.0f %s' % (d[0], i, d[2]),'%d %m %Y').date()
                            break

                    lineup = self.get_channel_lineup(htmldata)

                except:
                    log(traceback.format_exc())
                    continue

                try:
                    channel_cnt = 0
                    for c in htmldata.findall('div/div[@class="span12"]/div/div[@class="guide-scroller"]/div/div[@class="channels"]/div'):

                        channelid = lineup[channel_cnt]
                        channel_cnt += 1
                        if not channelid in self.chanids.keys():
                            continue

                        chanid = self.chanids[channelid]
                        get_programs(c, chanid, offset, self.all_channels[channelid]['group'] in (1, 7, 11))
                        self.day_loaded[channelid][offset] = True

                except:
                    log(traceback.format_exc())

                self.day_loaded[0][offset] = True

        for channelid, chanid in self.chanids.items():
            with self.source_lock:
                if first_added[config.opt_dict['offset']][chanid] != None:
                    self.program_data[channelid].append(first_added[config.opt_dict['offset']][chanid])

                if last_added[max_offset - 1][chanid] != None:
                    self.program_data[channelid].append(last_added[max_offset - 1][chanid])

                for offset in range(config.opt_dict['offset'] + 1, max_offset):
                    if first_added[offset][chanid] != None and last_added[offset - 1][chanid] != None \
                      and last_added[offset - 1][chanid]['name'].lower().strip() == first_added[offset][chanid]['name'].lower().strip():
                        # We merge them
                        first_added[offset][chanid]['start-time'] = last_added[offset - 1][chanid]['start-time']
                        self.program_data[channelid].append(first_added[offset][chanid])

                    else:
                        if first_added[offset][chanid] != None:
                            self.program_data[channelid].append(first_added[offset][chanid])

                        if last_added[offset - 1][chanid] != None:
                            self.program_data[channelid].append(last_added[offset - 1][chanid])

            if len(self.program_data[channelid]) == 0:
                self.set_ready(channelid)
                continue

            with self.source_lock:
                for tdict in self.program_data[channelid]:
                    self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

            self.parse_programs(chanid, 0, 'none')
            self.set_ready(channelid)
            try:
                infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

            except:
                pass

# end npo_HTML

class horizon_JSON(FetchData):
    """
    Get all available days of programming for the requested channels
    from the horizon.tv json pages. Based on FetchData
    """
    def init_channels(self):
        self.url_channels = ''
        self.init_channel_source_ids()

    def init_json(self):

        self.json_by_id = {}
        self.jsondata = {}
        self.jsondict = {}

    def get_url(self, type = 'channels', channel = 0, start = 0, end = 0, page = 1):

        if type == 'channels':
            return  u'%schannels/' % (config.source_base_url[self.proc_id])

        elif type == 'day':
            pend = page * 100
            pstart = pend - 99
            return '%slistings?byStationId=%s&byStartTime=%s~%s&sort=startTime&range=%s-%s' % \
                            (config.source_base_url[self.proc_id], channel, start, end, pstart, pend)

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        # download the json feed
        strdata = config.get_page(self.get_url(), 'utf-8')
        if strdata == None:
            self.fail_count += 1
            log("Unable to get channel info from %s\n" % self.source)
            return 69  # EX_UNAVAILABLE

        channel_list = json.loads(strdata)

        # and create a file with the channels
        self.all_channels ={}
        for channel in channel_list['channels']:
            for schedule in channel['stationSchedules']:
                channelid = schedule['station']['id']
                self.all_channels[channelid] = {}
                self.all_channels[channelid]['name'] = self.unescape(schedule['station']['title']).strip()
                if self.all_channels[channelid]['name'][-3:] == ' HD':
                    self.all_channels[channelid]['name'] = self.all_channels[channelid]['name'][:-3].strip()

                self.all_channels[channelid]['HD'] = schedule['station']['isHd']
                for icon in schedule['station']['images']:
                    if icon['assetType'] == 'station-logo-large' and icon['url'] != '':
                        icon = re.split('/', icon['url'])
                        self.all_channels[channelid]['icon'] = icon[-1].split('?')[0]
                        break

    def load_pages(self):
        first_fetch = True
        if config.opt_dict['offset'] > 7:
            self.set_ready()
            return

        if len(self.chanids) == 0 :
            return

        try:
            channel_cnt = 0
            for channelid, chanid in self.chanids.items():
                channel_cnt += 1
                page_count = 0
                if self.quit:
                    return

                # Maximum 100 programs are returned. So we try to get all and reset start to the endtime of the last
                start = int(time.mktime(datetime.date.fromordinal(self.current_date + config.opt_dict['offset']).timetuple()))*1000
                end = start + (86400000 * config.opt_dict['days'])
                page_fail = 0
                while True:
                    if self.quit:
                        return

                    page_count += 1
                    log(['\n', 'Now fetching %s(xmltvid=%s%s) from horizon.tv\n' % \
                        (config.channels[chanid].chan_name, config.channels[chanid].xmltvid, config.channels[chanid].get_opt('compat')), \
                        '    (channel %s of %s) for %s days, page %s.\n' % \
                        ( channel_cnt, len(self.chanids), config.opt_dict['days'], page_count)], 2)

                    channel_url = self.get_url('day', channelid, start, end, page_count)
                    if not first_fetch:
                        # be nice to horizon.tv
                        time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                        first_fetch = false

                    # get the raw programming for the day
                    strdata = config.get_page(channel_url, 'utf-8')
                    if strdata == None or strdata.replace('\n','') == '{}':
                        log("No data on horizon.tv channel %s page=%d\n" % (config.channels[chanid].chan_name, page_count))
                        self.fail_count += 1
                        page_fail += 1
                        if page_fail == 3:
                            break

                        page_count -= 1
                        continue

                    # Just let the json library parse it.
                    strdata = self.check_text_subs(strdata)
                    program_list = json.loads(strdata)
                    self.base_count += 1
                    for item in program_list['listings']:
                        if not 'program' in item:
                            continue

                        if item['program']['title'] == 'Zender verstrekt geen informatie':
                            break

                        if item['stationId'] != channelid:
                            # Wrong channel
                            continue

                        tdict = self.checkout_program_dict()
                        if (item['imi'] != '') and (item['imi'] != None):
                            tdict['prog_ID'][self.proc_id] = u'ho-%s' % (item['imi'][4:])
                            self.json_by_id[tdict['prog_ID'][self.proc_id]] = item
                            tdict['ID'] = tdict['prog_ID'][self.proc_id]

                        tdict['source'] = self.source
                        tdict['channelid'] = chanid
                        tdict['channel']  = config.channels[chanid].chan_name

                        # The Title
                        tdict['name'] = self.unescape(item['program']['title'])
                        if  tdict['name'] == None or tdict['name'] == '':
                            log('Can not determine program title for "%s"\n' % tdict['detail_url'][self.proc_id])
                            continue

                        # The timing
                        tdict['start-time'] = datetime.datetime.fromtimestamp(int(item['startTime'])/1000, CET_CEST)
                        tdict['stop-time']  = datetime.datetime.fromtimestamp(int(item['endTime'])/1000, CET_CEST)
                        if tdict['start-time'] == None or tdict['stop-time'] == None:
                            continue

                        tdict['offset'] = self.get_offset(tdict['start-time'])
                        #~ start = item['endTime']
                        if 'secondaryTitle' in item['program'] \
                          and item['program']['secondaryTitle'][:27].lower() != 'geen informatie beschikbaar' \
                          and item['program']['secondaryTitle'] not in (item['program']['title']) \
                          and len(item['program']['secondaryTitle']) < 50:
                            tdict['titel aflevering'] = self.unescape(item['program']['secondaryTitle'])

                        ep = int(item['program']['seriesEpisodeNumber']) if 'seriesEpisodeNumber' in item['program'] else 0
                        tdict['episode'] =  0 if ep > 1000 else str(ep)

                        shortdesc = self.unescape(item['program']['shortDescription']) if 'shortDescription' in item['program'] else ''
                        tdict['description'] = self.unescape(item['program']['description']) if 'description' in item['program'] else shortdesc
                        tdict['airdate'] = datetime.datetime.fromtimestamp(int(item['program']['airdate'])/1000, CET_CEST) if 'airdate' in item['program'] else ''
                        tdict['jaar van premiere'] = item['program']['year'] if 'year' in item['program'] else ''
                        tdict['rerun'] = ('latestBroadcastStartTime' in item['program'] and item['startTime'] != item['program']['latestBroadcastStartTime'])
                        if 'IMDb rating:' in tdict['description']:
                            d = re.split('IMDb rating:', tdict['description'])
                            tdict['description'] = d[0].strip()
                            tdict['star-rating'] = re.split('/', d[1])[0].strip()

                        if 'cast' in item['program'] and item['program']['cast'] != []:
                            tdict['credits']['actor'] = item['program']['cast']

                        if 'directors' in item['program'] and item['program']['directors'] != []:
                            tdict['credits']['director'] = item['program']['directors']

                        cats = item['program']['categories']
                        if 'mediaType' in item['program'] and item['program']['mediaType'] == 'FeatureFilm':
                            tdict['genre'] = 'film'

                            if len(cats) > 0:
                                tdict['subgenre'] = cats[-1]['title'].capitalize()

                        elif len(cats) == 0:
                            tdict['genre'] = 'overige'

                        elif len(cats) == 1 and (cats[0]['id'], ) in config.source_cattrans[self.proc_id].keys():
                            tdict['genre'] = config.source_cattrans[self.proc_id][(cats[0]['id'], )][0]
                            tdict['subgenre'] = config.source_cattrans[self.proc_id][(cats[0]['id'], )][1]

                        elif len(cats) == 2 and (cats[0]['id'], cats[1]['id']) in config.source_cattrans[self.proc_id].keys():
                            tdict['genre'] = config.source_cattrans[self.proc_id][(cats[0]['id'], cats[1]['id'])][0]
                            tdict['subgenre'] = config.source_cattrans[self.proc_id][(cats[0]['id'],cats[1]['id'])][1]

                        elif len(cats) == 2 and (cats[0]['id'], ) in config.source_cattrans[self.proc_id].keys():
                            tdict['genre'] = config.source_cattrans[self.proc_id][(cats[0]['id'], )][0]
                            if config.source_cattrans[self.proc_id][(cats[0]['id'], )][1] == '':
                                tdict['subgenre'] = cats[1]['title'].capitalize()
                                config.new_cattrans[self.proc_id][(cats[0]['id'], cats[1]['id'])] = (config.source_cattrans[self.proc_id][(cats[0]['id'], )][0], cats[1]['title'].capitalize())
                                if config.write_info_files:
                                    ids ="("
                                    titles = "("
                                    for c in cats:
                                        ids = "%s'%s', " % (ids, c['id'])
                                        titles = "%s'%s', " % (titles, c['title'].capitalize())
                                    ids = ids[:-2] + ")"
                                    titles = titles[:-2] + ")"
                                    infofiles.addto_detail_list(unicode('new horizon subcategorie => ' + ids + ': ' + titles + ', '))

                            else:
                                tdict['subgenre'] = config.source_cattrans[self.proc_id][(cats[0]['id'], )][1]

                        else:
                            tdict['genre'] = cats[0]['title'].capitalize()
                            if len(cats) == 2:
                                tdict['subgenre'] = cats[1]['title'].capitalize()
                                config.new_cattrans[self.proc_id][(cats[0]['id'], cats[1]['id'])] = (cats[0]['title'].capitalize(), cats[1]['title'].capitalize())

                            else:
                                config.new_cattrans[self.proc_id][(cats[0]['id'],)] = (cats[0]['title'].capitalize(), u'')

                            if config.write_info_files:
                                ids ="("
                                titles = "("
                                for c in cats:
                                    ids = "%s'%s', " % (ids, c['id'])
                                    titles = "%s'%s', " % (titles, c['title'].capitalize())
                                ids = ids[:-2] + ")"
                                titles = titles[:-2] + ")"
                                infofiles.addto_detail_list(unicode('new horizon categorie => ' + ids + ': ' + titles + ', '))

                        if config.write_info_files:
                            for cat in cats:
                                infofiles.addto_detail_list(u'horizon categorie: %s => %s' %(cat['id'], cat['title'].capitalize()))

                        self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict
                        tdict = self.check_title_name(tdict)
                        with self.source_lock:
                            self.program_data[channelid].append(tdict)

                    if int(program_list['entryCount']) < 100:
                        break

                with self.source_lock:
                    for tdict in self.program_data[channelid]:
                        self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict


                # If all went well or it's the last try we set them loaded
                self.parse_programs(chanid, 0, 'None')
                self.set_ready(channelid)
                try:
                    infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                except:
                    pass

        except:
            log(['\n', 'An unexpected error has occured in the %s thread:\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()
            return None

# end horizon_JSON

class humo_JSON(FetchData):
    """
    Get all available days of programming for the requested channels
    from the tvgids.nl json pages. Based on FetchData
    """
    def init_channels(self):

        self.init_channel_source_ids()

    def init_json(self):

        self.json_by_id = {}
        self.jsondata = {}
        self.jsondict = {}

    def get_url(self, channels = 'channels', offset = 0):

        scan_day = datetime.date.fromordinal(self.current_date + offset).strftime("%Y-%m-%d")

        if channels == 'channels':
            return  u'%s/channels' % (config.source_base_url[self.proc_id])

        elif channels == 'main':
            return '%s/schedule/main/%s/full' % (config.source_base_url[self.proc_id], scan_day)

        elif channels == 'rest':
            return '%s/schedule/rest/%s/full' % (config.source_base_url[self.proc_id], scan_day)

        else:
            return '%s/schedule/%s/%s/full' % (config.source_base_url[self.proc_id], channels, scan_day)

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        # download the json feed
        strdata = config.get_page(self.get_url(), 'utf-8')
        if strdata == None:
            self.fail_count += 1
            log("Unable to get channel info from %s\n" % self.source)
            return 69  # EX_UNAVAILABLE

        channel_list = json.loads(strdata)

        # and create a file with the channels
        self.all_channels ={}
        for chan_grp in channel_list['groups']:
            grp_name = chan_grp['name']
            grp_code = chan_grp['code']
            for channel in chan_grp['broadcasters']:
                channelid = unicode(channel['id'])
                icon = channel['media'][0]['resized_urls']['small']
                icon = icon.split('/')
                self.all_channels[channelid] = {}
                self.all_channels[channelid]['name'] = channel['display_name']
                self.all_channels[channelid]['icon'] = icon[-1]
                self.all_channels[channelid]['fetch_grp'] = grp_code

    def load_pages(self):

        if config.opt_dict['offset'] > 7:
            self.set_ready()
            return

        if len(self.chanids) == 0 :
            return

        first_fetch = True
        try:
            for offset in range(config.opt_dict['offset'], min((config.opt_dict['offset'] + config.opt_dict['days']), 8)):
                rest_channels = self.chanids.keys()
                for retry in (('main', 1), ('rest', 1), ('main', 2), ('rest', 2)):
                    if self.quit:
                        return

                    # Check if it is already loaded
                    channel_url = self.get_url(retry[0], offset)
                    if len(rest_channels) == 0:
                        continue

                    log(['\n', 'Now fetching %s channels from humo.be\n' % retry[0], \
                        '    (day %s of %s).\n' % (offset, config.opt_dict['days'])], 2)

                    if not first_fetch:
                        # be nice to humo.be
                        time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                        first_fetch = False

                    # get the raw programming for the day
                    strdata = config.get_page(channel_url, 'utf-8')
                    if strdata == None or strdata.replace('\n','') == '{}':
                        log("No data on humo.be %s-page for day=%d attempt %s\n" % (retry[0], offset, retry[1]))
                        self.fail_count += 1
                        continue

                    # Just let the json library parse it.
                    self.base_count += 1
                    self.day_loaded[0][offset] = True
                    strdata = self.check_text_subs(strdata)
                    jsondata = json.loads(strdata)
                    for channel in jsondata["broadcasters"]:
                        channelid = unicode(channel['id'])
                        if channelid in rest_channels:
                            chanid = self.chanids[channelid]
                            rest_channels.remove(channelid)

                        else:
                            continue

                        for item in channel['events']:
                            tdict = self.checkout_program_dict()
                            if (item['id'] != '') and (item['id'] != None):
                                tdict['prog_ID'][self.proc_id] = u'humo-%s' % (item['id'])
                                self.json_by_id[tdict['prog_ID'][self.proc_id]] = item
                                tdict['ID'] = tdict['prog_ID'][self.proc_id]

                            tdict['source'] = self.source
                            tdict['channelid'] = chanid
                            tdict['channel']  = config.channels[chanid].chan_name
                            tdict['detail_url'][self.proc_id] = item['url']

                            # The Title
                            tdict['name'] = self.unescape(item['program']['title'])
                            if  tdict['name'] == None or tdict['name'] == '':
                                log('Can not determine program title for "%s"\n' % tdict['detail_url'][self.proc_id])
                                continue

                            # The timing
                            tdict['start-time'] = datetime.datetime.fromtimestamp(item['starttime'], CET_CEST)
                            tdict['stop-time']  = datetime.datetime.fromtimestamp(item['endtime'], CET_CEST)
                            if tdict['start-time'] == None or tdict['stop-time'] == None:
                                continue

                            tdict['offset'] = self.get_offset(tdict['start-time'])
                            if 'content_long' in item['program'].keys():
                                tdict['description'] = item['program']['content_long']

                            elif 'content_short' in item['program'].keys():
                                tdict['description'] = item['program']['content_short']

                            elif 'description' in item['program'].keys():
                                tdict['description'] = item['program']['description']

                            if 'episodetitle' in item['program'].keys():
                                tdict['titel aflevering'] = item['program']['episodetitle']

                            if 'episodenumber' in item['program'].keys():
                                tdict['episode'] = item['program']['episodenumber']

                            if 'episodeseason' in item['program'].keys():
                                tdict['season'] = item['program']['episodeseason']

                            if 'year' in item['program'].keys():
                                tdict['jaar van premiere'] = item['program']['year']

                            if 'countries' in item['program'].keys():
                                #~ tdict['country'] = item['program']['countries']
                                for cstr in item['program']['countries']:
                                    cstr = cstr.upper().strip()
                                    if '(' in cstr:
                                        cstr = cstr.split('(')[1][:-1]

                                    if cstr in config.coutrytrans.values():
                                        tdict['country'] = cstr
                                        break

                                    elif cstr in config.coutrytrans.keys():
                                        tdict['country'] = config.coutrytrans[cstr]
                                        break

                                    elif config.write_info_files:
                                        infofiles.addto_detail_list(u'new country => %s' % (cstr))

                            if 'credits' in item['program'].keys():
                                for role in item['program']['credits']:
                                    if not role['role'] in tdict['credits']:
                                        tdict['credits'][role['role']] = []

                                    if not self.unescape(role['name']) in tdict['credits'][role['role']]:
                                        tdict['credits'][role['role']].append(self.unescape(role['name']))

                            if 'genres' in item['program'].keys():
                                if item['program']['genres'][0] in config.source_cattrans[self.proc_id].keys() and \
                                  config.source_cattrans[self.proc_id][item['program']['genres'][0]] != [u'Overige', u'']:
                                    tdict['genre'] = config.source_cattrans[self.proc_id][item['program']['genres'][0]][0]
                                    tdict['subgenre'] = config.source_cattrans[self.proc_id][item['program']['genres'][0]][1]

                                else:
                                    for g in  config.source_cattrans[self.proc_id].keys():
                                        if g == item['program']['genres'][0][0:len(g)]:
                                            tdict['genre'] = g
                                            sub = '' if len(item['program']['genres'][0]) <= len(g)+1 else item['program']['genres'][0][len(g)+1:]
                                            tdict['subgenre'] = sub
                                            config.new_cattrans[self.proc_id][item['program']['genres'][0]] = (g, sub)
                                            break

                                    else:
                                        tdict['genre'] = 'Overige'
                                        config.new_cattrans[self.proc_id][item['program']['genres'][0]] = (u'Overige', u'')
                                        if config.write_info_files:
                                            for gstr in item['program']['genres']:
                                                infofiles.addto_detail_list('new humo genre => ' + gstr)

                            else:
                                tdict['genre'] = 'Overige'

                            if 'teletext' in item['properties'].keys() and item['properties']['teletext'] == 1:
                                tdict['teletekst']  = True

                            if 'dolby' in item['properties'].keys() and item['properties']['dolby'] == 1:
                                tdict['audio']  = 'dolby'

                            if 'prop_16_9' in item['properties'].keys() and item['properties']['prop_16_9'] == 1:
                                tdict['video']['breedbeeld']  = True

                            if 'hd' in item['properties'].keys() and item['properties']['hd'] == 1:
                                tdict['video']['HD'] = True

                            if 'repeat' in item['properties'].keys() and item['properties']['repeat'] == 1:
                                tdict['rerun']  = True

                            if 'final' in item['properties'].keys() and item['properties']['final'] == 1:
                                tdict['last-chance']  = True

                            if 'new' in item['properties'].keys() and item['properties']['new'] == 1:
                                tdict['new']  = True

                            if config.write_info_files:
                                for key in item['properties'].keys():
                                    if not key in ('live', 'repeat', 'final', 'new', 'hd', 'prop_16_9', 'teletext', 'issub', 'dolby', \
                                      'part_of_series', 'series_id', 'maintitle', 'pdc', 'eventduration', 'selection'):
                                        infofiles.addto_detail_list('new humo property => %s=%s'  % (key, item['properties'][key]))

                                for key in item['program'].keys():
                                    if not key in ('id', 'external_id', 'title', 'media', 'twitterhashtag', 'youtubeid', 'website', \
                                      'programduration', 'episodetitle', 'episodenumber', 'episodeseason', 'episodetotal', \
                                      'description', 'content_short', 'content_long', 'year', 'countries', 'credits', 'genres', \
                                      'opinion'):
                                      #~ 'opinion', 'appreciation'):
                                        infofiles.addto_detail_list('new humo programitem => %s=%s' % (key, item['program'][key]))

                            tdict = self.check_title_name(tdict)
                            self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict
                            with self.source_lock:
                                self.program_data[channelid].append(tdict)

            for channelid, chanid in self.chanids.items():
                self.program_data[channelid].sort(key=lambda program: (program['start-time'],program['stop-time']))
                self.parse_programs(chanid, 0, 'None')
                self.set_ready(channelid)
                try:
                    infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                except:
                    pass

        except:
            log(['\n', 'An unexpected error has occured in the %s thread:\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()
            return None

# end humo_JSON

class vpro_HTML(FetchData):
    """
    Get all available days of programming for the requested channels
    from the vpro.nl page. Based on FetchData Class
    """
    def init_channels(self):
        """ General Site layout
        """

        self.init_channel_source_ids()
        self.availabe_days = []

    def get_url(self, offset = None):

        if offset == None:
            return u'%s.html' % (config.source_base_url[self.proc_id])

        elif isinstance(offset, int):
            scan_date = datetime.date.fromordinal(self.current_date + offset)
            return u'%s/content/0.html?day=%s' % (config.source_base_url[self.proc_id],  scan_date.strftime('%Y-%m-%d'))

    def get_channels(self):

        try:
            strdata = config.get_page(self.get_url())
            strdata = self.clean_html(strdata)
            strdata = self.check_text_subs(strdata)
            if strdata == None:
                self.fail_count += 1
                log(["Unable to get channel info from %s\n" % self.source])
                return 69  # EX_UNAVAILABLE

            self.get_available_days(strdata)
            self.get_channel_lineup(strdata)

        except:
            self.fail_count += 1
            log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
            return 69  # EX_UNAVAILABLE

    def get_channel_lineup(self, htmldata):
        chan_list = []
        channel_cnt = 0
        strdata = self.get_source_regex(htmldata, 1, 1, 1)
        htmldata = ET.fromstring( (u'<root>\n' + strdata + u'\n</root>\n').encode('utf-8'))
        for c in htmldata.findall('li'):
            channel_cnt+=1
            name = self.empersant(c.text)
            channelid =re.sub('[ /]', '_', name.lower())
            channelid =re.sub('é', 'e', channelid)
            channelid =re.sub('[!(),]', '', channelid)
            if c.attrib['class'] == "epg-source-radio epg-channel-name":
                grp = 11

            else:
                grp = 99

            self.all_channels[channelid] = {}
            self.all_channels[channelid]['name'] = name
            self.all_channels[channelid]['group'] = grp
            chan_list.append(channelid)

        return chan_list

    def get_available_days(self, htmldata):
        self.availabe_days = []
        htmldata = self.get_source_regex(htmldata, 0, 1, 1)
        htmldata = ET.fromstring( (htmldata).encode('utf-8'))
        for c in htmldata.findall('li/a'):
            d = re.split('-', c.attrib['rel'])
            self.availabe_days.append(datetime.date(int(d[0]), int(d[1]), int(d[2])).toordinal() - self.current_date)

    def load_pages(self):
        # The vpro description has all kind of inden info like: year episode, cast, presentation.
        def filter_desc(tdict):
            if tdict['description'] == '':
                return

            desc_items = self.get_string_parts(tdict['description'], ('met oa',))
            for di, dt in desc_items.items():
                if len(dt) == 0:
                    continue

                # Get subgenre, and possibly jaar van premiere, regisseur, country
                if di == 'start':
                    subg = self.get_source_regex(dt[0], 4)
                    subg2 = self.get_source_regex(dt[0], 5)
                    subg3 = self.get_source_regex(dt[0], 6)
                    subg4 = self.get_source_regex(dt[0], 7)
                    if subg != None:
                        tdict['subgenre'] = subg.group(1)
                        tdict['jaar van premiere'] = subg.group(2)
                        direct = re.sub(' en ',  ' , ', subg.group(3))
                        direct = re.split(',', direct)
                        if not 'director' in tdict['credits']:
                            tdict['credits']['director'] = []

                        for d in direct:
                            tdict['credits']['director'].append(d)

                    elif subg2 != None:
                        cstr = re.split('/', subg2.group(1))
                        for c in cstr:
                            if c in config.coutrytrans.values():
                                tdict['country'] = c
                                break

                            elif c in config.coutrytrans.keys():
                                tdict['country'] = config.coutrytrans[c]
                                break

                            elif config.write_info_files:
                                infofiles.addto_detail_list(u'new country => %s' % (c))

                        tdict['jaar van premiere'] = subg2.group(2)
                        tdict['subgenre'] = subg2.group(3)
                        direct = re.sub(' en ',  ' , ', subg2.group(4))
                        direct = re.split(',', direct)
                        if not 'director' in tdict['credits']:
                            tdict['credits']['director'] = []

                        for d in direct:
                            dtest = re.split(' ', d)
                            if dtest[0] in ('gebaseerd', 'naar', ) or len(dtest) > 5:
                                continue

                            tdict['credits']['director'].append(d)

                    elif subg3 != None:
                        tdict['subgenre'] = subg3.group(1)
                        tdict['jaar van premiere'] = subg3.group(2)

                    elif dt[0][0:3] == 'Afl' or dt[0][0:7] == 'Overige':
                        pass

                    elif subg4 != None:
                        subg5 = re.split(' ', subg4.group(1))
                        if len(subg5) <= 4:
                            tdict['subgenre'] = subg4.group(1)

                    else:
                        subg6 = re.split(' ', dt[0])
                        if len(subg6) <= 4:
                            tdict['subgenre'] = dt[0]


                # Get any roles
                elif di in config.roletrans.keys():
                    role = config.roletrans[di]
                    if not role in tdict['credits']:
                        tdict['credits'][role] = []

                    cast = re.sub('e\.a\.',  '', dt[0])
                    cast = re.sub(' en ',  ' , ', cast)
                    cast = re.split(',', cast)
                    for cn in cast:
                        tdict['credits'][role].append(cn.split('(')[0].strip())

                # Get the episode Number
                elif di == 'aflevering':
                    ep = re.search('[0-9]+', dt[0])
                    if ep != None:
                        tdict['episode'] = int(ep.group(0))

                # Get the subtitle and possibly an episode number
                elif di[0:3] == 'afl':
                    tdict['titel aflevering'] = dt[0].strip()
                    ep = re.search('[0-9]+', di)
                    if ep != None and tdict['episode'] == 0:
                        tdict['episode'] = int(ep.group(0))

                elif di[0:9] == 'vertaling':
                    pass

                elif di not in ('met oa', ) and config.write_info_files:
                    infofiles.addto_detail_list(u'new vpro descr item => %s' % (di))

            # Alternative Acters list
            if 'met oa' in desc_items and not 'met'in desc_items:
                try:
                    role = config.roletrans['met']
                    if not role in tdict['credits']:
                        tdict['credits'][role] = []

                    cast = re.sub(' en ',  ' , ', desc_items['met oa'][0])
                    cast = re.split(',', cast)
                    for cn in cast:
                        tdict['credits'][role].append(cn.split('(')[0].strip())

                except:
                    pass

        def get_programs(xml, chanid):
            try:
                channelid = self.channels[chanid]
                tdict = None
                day_offset = 0
                for p in xml.findall('li'):
                    ptext = p.get('data-title')
                    if ptext == None:
                        # No title Found
                        continue

                    ptime = p.findtext('div[@class="content"]/div[@class="program-data"]/div[@class="meta"]/span[@class="epg-start-time"]')
                    if ptime == None:
                        # No start-stop time Found
                        continue

                    tdict = self.checkout_program_dict()
                    tdict['source'] = u'vpro'
                    tdict['channelid'] = chanid
                    tdict['channel'] = config.channels[chanid].chan_name
                    tdict['detail_url'][self.proc_id] = p.get('data-read-more-url', '')
                    #~ if tdict['detail_url'][self.proc_id] != '':
                        #~ pid = tdict['detail_url'][self.proc_id].split('/')[-1]
                        #~ tdict['prog_ID'][self.proc_id] = u'npo-%s' % pid.split('_')[-1]

                    # The Title
                    tdict['name'] = self.empersant(ptext.strip())

                    pstart = re.sub('vpro', '', ptime).strip()
                    pstart = pstart.split(':')
                    prog_time = datetime.time(int(pstart[0]), int(pstart[1]), 0 ,0 ,CET_CEST)

                    if day_offset == 0 and int(pstart[0]) < 6:
                        day_offset = 1

                    tdict['offset'] = offset + day_offset

                    if day_offset == 1:
                        tdict['start-time'] = datetime.datetime.combine(nextdate, prog_time)

                    else:
                        tdict['start-time'] = datetime.datetime.combine(startdate, prog_time)

                    tdict['description'] = self.empersant(p.get('data-description','').strip())
                    eptitle = self.empersant(p.get('data-episode-title','').strip())
                    if eptitle != tdict['name']:
                        tdict['episode title'] = eptitle

                    omroep = p.findtext('div[@class="content"]/h6[@class="title"]/span[@class="broadcaster"]')
                    if not omroep in ('', None):
                        tdict['omroep'] = self.empersant(omroep)

                    pgenre = p.get('class','')
                    pg = self.get_source_regex(pgenre, 3, 2)
                    if len(pg) > 0:
                        for i in range(len(pg)):
                            pg[i] = re.sub('_', '', pg[i])

                        pg = tuple(pg)
                        if len(pg) > 2:
                            pg = pg[0:2]

                        if pg in config.source_cattrans[self.proc_id].keys():
                            tdict['genre'] = config.source_cattrans[self.proc_id][pg][0].capitalize()
                            tdict['subgenre'] = config.source_cattrans[self.proc_id][pg][1].capitalize()

                        else:
                            if len(pg) > 1 and (pg[0].lower(), ) in config.source_cattrans[self.proc_id].keys():
                                tdict['genre'] = config.source_cattrans[self.proc_id][(pg[0].lower(), )][0].capitalize()
                                tdict['subgenre'] = config.source_cattrans[self.proc_id][(pg[0].lower(), )][1].capitalize()
                                config.new_cattrans[self.proc_id][(pg[0], pg[1])] = config.source_cattrans[self.proc_id][(pg[0].lower(), )]

                            else:
                                tdict['genre'] = u'overige'
                                if len(pg) == 2:
                                    tdict['subgenre'] = pg[1].capitalize()
                                    config.new_cattrans[self.proc_id][pg] = (u'Overige', pg[1])

                                elif not pgenre in ('', 'gvpro'):
                                    config.new_cattrans[self.proc_id][pg] = (u'Overige', u'')

                            if config.write_info_files and not pgenre in ('', 'gvpro'):
                                infofiles.addto_detail_list(unicode('unknown vpro.nl genre => ' + pgenre + ': ' + tdict['name']))

                    else:
                        tdict['genre'] = u'overige'

                    filter_desc(tdict)
                    # and append the program to the list of programs
                    tdict = self.check_title_name(tdict)
                    with self.source_lock:
                        self.program_data[channelid].append(tdict)

            except:
                log(traceback.format_exc())

        if config.opt_dict['offset'] > 5:
            self.set_ready()
            return

        if len(self.chanids) == 0 :
            return

        for retry in (0, 1):
            for offset in range(config.opt_dict['offset'], min((config.opt_dict['offset'] + config.opt_dict['days']), 5)):
                if self.quit:
                    return

                # Check if it is already loaded
                if self.day_loaded[0][offset]:
                    continue

                if len(self.availabe_days) > 0 and not offset in self.availabe_days:
                    continue

                log(['\n', 'Now fetching %s channels from vpro.nl\n' % (len(self.chanids)), \
                    '    (day %s of %s).\n' % (offset, config.opt_dict['days'])], 2)

                channel_url = self.get_url(offset)

                # get the raw programming for the day
                strdata = config.get_page(channel_url)
                if strdata == None or 'We hebben deze pagina niet gevonden...' in strdata:
                    if retry == 0:
                        log("No data on vpro.nl for day=%d. First attempt.\n" % (offset))
                    else:
                        log("No data on vpro.nl for day=%d. Final attempt.\n" % (offset))

                    self.fail_count += 1
                    continue

                try:
                    strdata = self.clean_html(strdata)
                    strdata = self.check_text_subs(strdata)
                    if len(self.availabe_days) == 0:
                        self.get_available_days(strdata)
                        lineup = self.get_channel_lineup(strdata)

                    strdata = self.get_source_regex(strdata, 2, 1, 0)
                    htmldata = ET.fromstring(strdata.encode('utf-8'))

                except:
                    log('Error extracting ElementTree for day:%s on vpro.nl\n' % (offset))
                    self.fail_count += 1
                    if config.write_info_files:
                        infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                        infofiles.write_raw_string(strdata)

                    continue

                # First we get the line-up and some date checks
                self.base_count += 1
                try:
                    startdate = htmldata.find('div[@class="grid"]/div/div').get('data-selected-guide-date')
                    if startdate == None:
                        log('Error validating page for day:%s on vpro.nl\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno, offset))
                        continue

                    d = startdate.split('-')
                    startdate = datetime.date(int(d[0]), int(d[1]), int(d[2]))
                    nextdate = startdate + datetime.timedelta(days=1)
                    if startdate.toordinal() - self.current_date != offset:
                        log('Error validating page for day:%s on vpro.nl\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno, offset))
                        continue

                except:
                    log(traceback.format_exc())
                    continue

                try:
                    channel_cnt = 0
                    for c in htmldata.findall('div[@class="grid"]/div/div/div/div/div/div[@class="epg-channels-container"]/ol'):
                        channelid = lineup[channel_cnt]
                        channel_cnt += 1
                        if not channelid in self.chanids.keys():
                            continue

                        chanid = self.chanids[channelid]
                        get_programs(c, chanid)
                        if channel_cnt == 2:
                            self.day_loaded[channelid][offset] = True

                except:
                    log(traceback.format_exc())

                # be nice to vppo.nl
                self.day_loaded[0][offset] = True
                time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))

        for channelid, chanid in self.chanids.items():
            if len(self.program_data[channelid]) == 0:
                self.set_ready(channelid)
                continue

            # Add starttime of the next program as the endtime
            with self.source_lock:
                self.program_data[channelid].sort(key=lambda program: (program['start-time']))
                self.add_endtimes(channelid, 6)

                for tdict in self.program_data[channelid]:
                    self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

            self.parse_programs(chanid, 0, 'none')
            self.set_ready(channelid)
            try:
                infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

            except:
                pass

# end vpro_HTML

class nieuwsblad_HTML(FetchData):
    """
    Get all available days of programming for the requested channels
    from the nieuwsblad.be page. Based on FetchData Class
    """
    def init_channels(self):
        """ General Site layout
            <html class="no-js " dir="ltr" lang="nl-BE">
                <head>
                <body class="theme-default" itemscope itemtype="http://schema.org/WebPage">
                    <div class="body-wrapper">
                        <div class="site-container">
                            <div class="site-container__inner">
                                <main role="main">
                                    <!-- start zone Zone_0 -->
                                    <section class="l-zone">
                                        <div class="grid">
                                            # Datedefinition
                                            <div class="grid__col">
                                                <div class="grid__col__inner">
                                                <!-- start block 'tvgids-top' -->
                                                    <div data-mht-block="zone_0__tvgids-top">
                                                        <div class="grid__col size-2-3--bp4">
                                                            <div class="grid__col__inner">
                                                                <h1>
                                                                    TV-Gids vandaag</h1>
                                                            </div>
                                                        </div>
                                                        <div class="grid__col size-1-3--bp4">
                                                            <div class="grid__col__inner">
                                                                <p>
                                                                    dinsdag, 01 september 2015</p>
                                                            </div>
                                                        </div>
                                                    </div>
                                                <!-- end block 'tvgids-top' -->
                                                </div>
                                            </div>
                                            # Program list
                                            <div class="grid__col size-4-5">
                                                <div class="grid__col__inner">
                                                <!-- start block 'tvgids-left-center' -->
                                                    <div data-mht-block="zone_0__tvgids-left-center">
                                                        <div class="grid channel-block">
                                                            <div class="grid__col size-1-3--bp4">
                                                                <div class="grid__col__inner">
                                                                    <div class="tv-guide__channel">
                                                                        <h6>
                                                                            <img src="http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/een.png" class="channelLogo" alt="EEN"
                                                                                  ><a href="http://www.nieuwsblad.be/tv-gids/een/gisteren">EEN</a>
                                                                        </h6>
                                                                    </div>
                                                                    <div class="program">
                                                                        <div class="time">09:00</div>
                                                                        <div class="title"><a href="http://www.nieuwsblad.be/tv-gids/een/gisteren/zomerbeelden">Zomerbeelden</a></div>
                                                                    </div>
                                                                        ...
                                                                </div>
                                                            </div>
                                                        </div>
                                                    </div>
                                                <!-- end block 'tvgids-left-center' -->
                                                </div>
                                            </div>
                                            # Channel list
                                            <div class="grid__col size-1-5--bp4">
                                                <div class="grid__col__inner">
                                                <!-- start block 'tvgids-right-center' -->
                                                    <div data-mht-block="zone_0__tvgids-right-center">
                                                        <h3 class="heading">
                                                            Alle zenders</h3>
                                                        <div id="accordion" class="accordion" data-accordion data-jq-plugin="accordion">
                                                            <div class="accordion__header">
                                                                Vlaams</div>
                                                            <div class="accordion__content">
                                                                <a href="http://www.nieuwsblad.be/tv-gids/vandaag/0"><div class="channel-row">

                                                                    <img class="tv-icon" data-slug="EEN" src="http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/een.png" onerror="this.onerror=null;this.src='http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/dummy-channel.png';" data-lang="Vlaams" title="EEN">

                                                                    <img class="tv-icon" data-slug="VTM" src="http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/vtm.png" onerror="this.onerror=null;this.src='http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/dummy-channel.png';" data-lang="Vlaams" title="VTM">

                                                                    <img class="tv-icon" data-slug="VIER" src="http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/vier.png" onerror="this.onerror=null;this.src='http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/dummy-channel.png';" data-lang="Vlaams" title="VIER">
                                                                    </div>
                                                                </a>
                                                                    ...
                                                            </div>
                                                        </div>
                                                    </div>
                                                <!-- end block 'tvgids-right-center' -->
                                                </div>
                                            </div>
                                        </div>
                                    </section>
                                    <!-- end zone Zone_0 -->
                                </main>
                            </div>
                        </div>
                    </div>
                </body>
            </html>
        """

        self.init_channel_source_ids()

        # These regexes fetch the relevant data out of the nieuwsblad.be pages, which then will be parsed to the ElementTree
        self.relativedays = ['vandaag', 'morgen', 'overmorgen']

    def get_url(self, channel = None, offset = 0, chan_group = 0):

        if offset in range(len(self.relativedays)):
            scan_day = self.relativedays[offset]

        else:
            scan_day = config.weekdagen[int(datetime.date.fromordinal(self.current_date + offset).strftime("%w"))]

        if channel == 'base':
            return config.source_base_url[self.proc_id]

        elif channel == 'zenders':
            return '%s/zenders' % config.source_base_url[self.proc_id]

        elif channel != None:
            return '%s/%s/%s' % (config.source_base_url[self.proc_id], channel,  scan_day)

        else:
            return u'%s/%s/%s' % (config.source_base_url[self.proc_id],  scan_day, chan_group)

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        try:
            strdata = config.get_page(self.get_url('base'))
            if self.get_channel_lineup(strdata) == 69:
                self.fail_count += 1
                log(["Unable to get channel info from %s\n" % self.source])
                return 69  # EX_UNAVAILABLE

        except:
            self.fail_count += 1
            log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
            return 69  # EX_UNAVAILABLE

    def get_channel_lineup(self, chandata):

        chan_groups = {'Vlaams': 2,
                                    'Nederlands': 1,
                                    'Frans': 5,
                                    'Duits': 4,
                                    'Engels': 3,
                                    'Overige': 99}

        self.chan_names = {}
        self.page_strings = {}
        try:
            strdata = config.get_page(self.get_url('zenders'))
            if strdata == None:
                self.fail_count += 1

            else:
                strdata = self.get_source_regex(strdata, 0, 1, 1)
                strdata = re.sub('<img (.*?)"\s*>', '<img \g<1>"/>', strdata)
                strdata = self.clean_html('<div><div>' + strdata)
                htmldata = ET.fromstring(strdata.encode('utf-8'))

                for item in htmldata.findall('div/div[@class]/div[@class="grid__col__inner"]/a[@href]'):
                    url = item.get('href', '')
                    if url != '':
                        channelid = url.split('/')[-2].strip()
                        if channelid in self.all_channels:
                            continue

                        name = self.empersant(item.findtext('div[@class="grid"]/div[@class="grid__col"]/div[@class]/p')).strip()
                        icon = item.find('div[@class="grid"]/div[@class="grid__col size-1-3"]/div[@class]/img').get('src', '')
                        if icon != '':
                            icon = icon.split('/')
                            icon = '%s/%s' % (icon[-2], icon[-1])

                        self.all_channels[channelid] = {}
                        self.all_channels[channelid]['name'] = name
                        self.all_channels[channelid]['icon'] = icon
                        self.all_channels[channelid]['icongrp'] = 8
                        self.chan_names[name] = channelid

                for item in htmldata.findall('div/div[@class]/div[@class="grid__col__inner"]/div[@class="grid"]/a[@href]'):
                    url = item.get('href', '')
                    if url != '':
                        channelid = url.split('/')[-2]
                        if channelid in self.all_channels:
                            continue

                        name = self.empersant(item.findtext('div[@class]/div[@class="grid__col__inner"]/p')).strip()
                        icon = item.find('div[@class]/div[@class="grid__col__inner"]/img').get('src', '')
                        if icon != '':
                            icon = icon.split('/')
                            icon = '%s/%s' % (icon[-2], icon[-1])

                        self.all_channels[channelid] = {}
                        self.all_channels[channelid]['name'] = name
                        self.all_channels[channelid]['icon'] = icon
                        self.all_channels[channelid]['icongrp'] = 8
                        self.chan_names[name] = channelid

                for item in htmldata.findall('div/div[@class]/div[@class="grid__col__inner"]/ul/li/a[@href]'):
                    url = item.get('href', '')
                    if url != '':
                        channelid = url.split('/')[-2]
                        if channelid in self.all_channels or channelid == 'bbc1':
                            continue

                        name = self.empersant(item.text).strip()
                        icon = ''

                        self.all_channels[channelid] = {}
                        self.all_channels[channelid]['name'] = name
                        self.chan_names[name] = channelid

        except:
            self.fail_count += 1
            log( traceback.format_exc())

        changroup = 99
        try:
            if not isinstance(chandata, (str, unicode)):
                chandata = config.get_page(self.get_url('base'))

            if chandata == None:
                return 69  # EX_UNAVAILABLE

            strdata = self.get_source_regex(chandata, 2, 1, 1)
            strdata = re.sub('<img (.*?)"\s*>', '<img \g<1>"/>', strdata)
            strdata = self.clean_html('<div><div>' + strdata)
            htmldata = ET.fromstring(strdata.encode('utf-8'))
            for item in htmldata.findall('div/div[@class]'):
                if item.get('class') == 'accordion__header':
                    group =  self.empersant(item.text).strip()
                    if group in chan_groups:
                        changroup = chan_groups[group]

                    else:
                        changroup = 99

                elif item.get('class') == 'accordion__content':
                    for g in item.findall('a[@href]'):
                        pagegrp = g.get('href').split('/')[-1]
                        self.page_strings[pagegrp] = []
                        for c in g.findall('div/img'):
                            cname = c.get('title').strip()
                            icon = c.get('src', '')
                            if icon != '':
                                icon = re.split('/', icon)[-1]
                                channelid = re.split('\.', icon)[0]
                                if (changroup == 99 and channelid in ('npo1', 'npo2', 'npo3')) or channelid in ('tv5', 'bbc1'):
                                    continue

                            if not channelid in self.all_channels.keys():
                                if cname in self.chan_names.keys():
                                    channelid = self.chan_names[cname]

                                elif channelid != '':
                                    self.all_channels[channelid] = {}

                                else:
                                    continue

                            if not 'name' in self.all_channels[channelid] or self.all_channels[channelid]['name'] == '':
                                self.all_channels[channelid]['name'] = cname

                            if not 'icon' in self.all_channels[channelid] or self.all_channels[channelid]['icon'] == '':
                                self.all_channels[channelid]['icon'] = icon
                                self.all_channels[channelid]['icongrp'] = 8

                            self.all_channels[channelid]['pagegrp'] = pagegrp
                            self.page_strings[pagegrp].append(channelid)
                            self.all_channels[channelid]['group'] = changroup

        except:
            self.fail_count += 1
            log(traceback.format_exc())


    def load_pages(self):
        failure_count = 0
        if config.opt_dict['offset'] > 6:
            self.set_ready()
            return

        if len(self.channels) == 0:
            return

        dayoffset = config.create_dayoffset(self.current_date)

        try:
            for retry in (0, 1):
                channel_cnt = 0
                for channelid, chanid in self.chanids.items():
                    channel_cnt += 1
                    failure_count = 0
                    if self.quit:
                        return

                    if config.channels[chanid].source_data[self.proc_id].is_set():
                        continue

                    # Nieuwsblad.be either returns 6 days per channel or 3 channels per day for 7 days including today
                    start = config.opt_dict['offset']
                    # Check if it is allready loaded
                    if self.day_loaded[channelid][start] != False:
                        continue

                    log(['\n', 'Now fetching %s(xmltvid=%s%s) from nieuwsblad.be\n' % \
                        (config.channels[chanid].chan_name, config.channels[chanid].xmltvid , \
                        config.channels[chanid].get_opt('compat')), \
                        '    (channel %s of %s) for 6 days.\n' % \
                        (channel_cnt, len(self.channels))], 2)

                    # get the raw programming for the day
                    try:
                        channel_url = self.get_url(channelid, start)
                        strdata = config.get_page(channel_url)

                        if strdata == None:
                            if retry == 0:
                                log("Skip channel=%s on nieuwsblad.be. No data! First attempt.\n" % (config.channels[chanid].chan_name))
                            else:
                                log("Skip channel=%s on nieuwsblad.be. No data! Final attempt.\n" % (config.channels[chanid].chan_name))

                            failure_count += 1
                            self.fail_count += 1
                            continue

                        if self.all_channels == {}:
                            self.get_channel_lineup(strdata)

                    except:
                        log('Error: "%s" reading the nieuwsblad.be basepage for channel=%s.\n' % \
                            (sys.exc_info()[1], config.channels[chanid].chan_name))
                        failure_count += 1
                        self.fail_count += 1
                        continue

                    try:
                        strdata =self.get_source_regex(strdata, 1, 1, 1)
                        #~ strdata = re.sub('<img (.*?)"\s*>', '<img \g<1>"/>', strdata)
                        strdata = self.clean_html(strdata)
                        strdata = self.check_text_subs(strdata)
                        htmldata = ET.fromstring(strdata.encode('utf-8'))

                    except:
                        log(["Error extracting ElementTree for channel:%s on nieuwsblad.be\n" % \
                            (config.channels[chanid].chan_name)])

                        if config.write_info_files:
                            infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                            infofiles.write_raw_string(strdata)

                        failure_count += 1
                        self.fail_count += 1
                        self.day_loaded[channelid][start] = None
                        continue

                    for d in htmldata.findall('div[@class="grid channel-block"]/div[@class="grid__col size-1-3--bp4"]'):
                        weekday = d.findtext('div/div[@class="tv-guide__channel"]/h6/a').strip()
                        offset = dayoffset[weekday]
                        if offset >= config.opt_dict['offset'] + config.opt_dict['days']:
                            break

                        date_offset = offset
                        scan_date = datetime.date.fromordinal(self.current_date + date_offset)
                        last_program = datetime.datetime.combine(datetime.date.fromordinal(self.current_date + date_offset - 1), \
                                                                                                datetime.time(0, 0, 0 ,0 ,CET_CEST))
                        for p in d.findall('div/div[@class="program"]'):
                            #~ start = p.findtext('div[@class="time"]')
                            #~ title = p.findtext('div[@class="title"]/a').strip()
                            #~ url = p.find('div[@class="title"]/a').get('href')

                            tdict = self.checkout_program_dict()
                            tdict['source'] = u'nieuwsblad'
                            tdict['channelid'] = chanid
                            tdict['channel'] = config.channels[chanid].chan_name
                            tdict['detail_url'][self.proc_id] = p.find('div[@class="title"]/a').get('href')
                            #~ tdict['prog_ID'][self.proc_id] = u'tv-%s' % tdict['detail_url'][self.proc_id].split('/')[5]  if (tdict['detail_url'][self.proc_id] != '') else ''

                            # The Title
                            tdict['name'] = self.empersant(p.findtext('div[@class="title"]/a').strip())
                            if  tdict['name'] == None or tdict['name'] == '':
                                log('Can not determine program title for "%s"\n' % tdict['detail_url'][self.proc_id])
                                continue

                            # Get the starttime and make sure the midnight date change is properly crossed
                            start = p.findtext('div[@class="time"]')
                            if start == None or start == '':
                                log('Can not determine starttime for "%s"\n' % tdict['name'])
                                continue

                            prog_time = datetime.time(int(start.split(':')[0]), int(start.split(':')[1]), 0 ,0 ,CET_CEST)
                            if datetime.datetime.combine(scan_date, prog_time) < last_program:
                                date_offset = date_offset +1
                                scan_date = datetime.date.fromordinal(self.current_date + date_offset)

                            tdict['offset'] = date_offset
                            tdict['start-time'] = datetime.datetime.combine(scan_date, prog_time)
                            last_program = tdict['start-time']

                            # and append the program to the list of programs
                            tdict = self.check_title_name(tdict)
                            with self.source_lock:
                                self.program_data[channelid].append(tdict)

                        self.base_count += 1
                        self.day_loaded[channelid][offset] = True
                        # be nice to nieuwsblad.be
                        time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))

                    if len(self.program_data[channelid]) == 0:
                        self.set_ready(channelid)
                        continue

                    # Add starttime of the next program as the endtime
                    with self.source_lock:
                        self.program_data[channelid].sort(key=lambda program: (program['start-time']))
                        self.add_endtimes(channelid, 6)

                        for tdict in self.program_data[channelid]:
                            self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

                    if failure_count == 0 or retry == 1:
                        self.parse_programs(chanid, 0, 'None')
                        self.set_ready(channelid)
                        try:
                            infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                        except:
                            pass

        except:
            log(['\n', 'An unexpected error has occured in the %s thread\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()

# end nieuwsblad_HTML

class primo_HTML(FetchData):
    """
    Get all available days of programming for the requested channels
    from the primo.eu page. Based on FetchData Class
    """
    def init_channels(self):

        self.init_channel_source_ids()

    def get_url(self, offset = 0, detail = None):
        if offset == 'channels':
            return config.source_base_url[self.proc_id] + "/Tv%20programma's%20in%20volledig%20scherm%20bekijken"

        elif detail == None and isinstance(offset, int):
            date = self.get_datestamp(offset)
            return '%s/tv-programs-full-view/%s/all/all' % (config.source_base_url[self.proc_id], date)

        else:
            return u'%s/tvprograms/ajaxcallback/%s' % (config.source_base_url[self.proc_id],  detail)

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        try:
            strdata = config.get_page(self.get_url('channels'))
            if self.get_channel_lineup(strdata) == 69:
                log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
                return 69  # EX_UNAVAILABLE

        except:
            self.fail_count += 1
            log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
            return 69  # EX_UNAVAILABLE

    def get_channel_lineup(self, chandata):

        try:
            if not isinstance(chandata, (str, unicode)):
                chandata = config.get_page(self.get_url(0))

            strdata = self.get_source_regex(chandata, 0, 1, 1)
            strdata = self.clean_html(strdata)
            strdata = self.check_text_subs(strdata)
            htmldata = ET.fromstring(strdata.encode('utf-8'))
            htmldata = htmldata.find('div/div[@id="tvprograms-main"]/div[@id="tvprograms"]')
            for item in htmldata.findall('div[@id="program-channel-programs"]/div/div/div'):
                if item.get("style") != None:
                    continue

                chan_string = self.get_source_regex(item.get("class"), 1)
                channelid = chan_string.group(1)
                cname = chan_string.group(2)
                icon_search = 'div[@id="program-channels-list-main"]/div/ul/li/div/a/img[@class="%s"]' % channelid
                icon = htmldata.find(icon_search)
                if icon == None:
                    icon = ''

                else:
                    icon = re.split('/',icon.get("src"))[-1]

                if not channelid in self.all_channels.keys():
                    self.all_channels[channelid] = {}
                    self.all_channels[channelid]['name'] = cname
                    self.all_channels[channelid]['icon'] = icon
                    self.all_channels[channelid]['icongrp'] = 9

        except:
            self.fail_count += 1
            log(traceback.format_exc())
            return 69

    def load_pages(self):
        failure_count = 0
        if config.opt_dict['offset'] > 7:
            self.set_ready()
            return

        if len(self.chanids) == 0:
            return

        try:
            for retry in (0, 1):
                for offset in range( config.opt_dict['offset'], min((config.opt_dict['offset'] + config.opt_dict['days']), 7)):
                    if self.quit:
                        return

                    # Check if it is allready loaded
                    if self.day_loaded[0][offset] != False:
                        continue

                    log(['\n', 'Now fetching channels from primo.eu for day %s of %s\n' % \
                        (offset, config.opt_dict['days'])], 2)

                    # get the raw programming for the day
                    try:
                        channel_url = self.get_url(offset)
                        strdata = config.get_page(channel_url)

                        if strdata == None:
                            if retry == 0:
                                log("Skip day=%s on primo.eu. No data! First attempt.\n" % (offset))
                            else:
                                log("Skip day=%s on primo.eu. No data! Final attempt.\n" % (offset))

                            failure_count += 1
                            self.fail_count += 1
                            continue

                        if self.all_channels == {}:
                            self.get_channel_lineup(strdata)

                    except:
                        log('Error: "%s" reading the primo.eu basepage for day %s.\n' % \
                            (sys.exc_info()[1], offset))
                        failure_count += 1
                        self.fail_count += 1
                        continue

                    try:
                        strdata = self.get_source_regex(strdata, 0, 1, 1)
                        strdata = self.clean_html(strdata)
                        strdata = self.check_text_subs(strdata)
                        htmldata = ET.fromstring(strdata.encode('utf-8'))
                        htmldata = htmldata.find('div/div[@id="tvprograms-main"]/div[@id="tvprograms"]')
                        sel_date = htmldata.findtext('div[@id="program-header-top"]/div/div[@id="dates"]/ul/li[@class="selected-date"]/a/span[@class="day"]')
                        if sel_date in ('', None) or datetime.date.fromordinal(self.current_date + offset).day != int(sel_date):
                            log("Skip day=%d on Primo.eu. Wrong date: %s(timestamp: %s!\n" % (offset, sel_date, self.get_datestamp(offset)))
                            failure_count += 1
                            self.fail_count += 1
                            continue


                    except:
                        log(["Error extracting ElementTree for day:%s on primo.eu\n" % (offset)])

                        if config.write_info_files:
                            infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                            infofiles.write_raw_string(strdata)

                        failure_count += 1
                        self.fail_count += 1
                        continue

                    for chan in htmldata.findall('div[@id="program-channel-programs"]/div/div/div'):
                        if chan.get("style") != None:
                            continue

                        channelid = self.get_source_regex(chan.get("class"), 1, 1, 1)
                        if not channelid in self.chanids.keys():
                            continue

                        chanid = self.chanids[channelid]
                        date_offset = offset -1
                        last_end = datetime.datetime.combine(datetime.date.fromordinal(self.current_date + offset), \
                                                                                        datetime.time(hour=6, tzinfo=CET_CEST))
                        for d in chan.findall('div[@class="hour hour-"]'):
                            date_offset+=1
                            scan_date = datetime.date.fromordinal(self.current_date + date_offset)
                            for p in d.findall('div'):
                                tdict = self.checkout_program_dict()
                                tdict['source'] = u'primo'
                                tdict['channelid'] = chanid
                                tdict['channel'] = config.channels[chanid].chan_name
                                pid = p.find('h3').get('id')
                                tdict['prog_ID'][self.proc_id] = u'primo-%s' % pid  if pid != None else ''
                                tdict['detail_url'][self.proc_id] = self.get_url(detail = pid)  if pid != None else ''

                                # The Title
                                tdict['name'] = self.empersant(p.findtext('h3').strip())
                                if  tdict['name'] == None or tdict['name'] == '':
                                    log('Can not determine program title for "%s"\n' % tdict['detail_url'][self.proc_id])
                                    continue

                                # Get the starttime and make sure the midnight date change is properly crossed
                                ptime = p.findtext('span', '')
                                pduur = int(self.get_source_regex(p.get('style'), 2, 1, 1))*12
                                if ptime == '':
                                    tdict['start-time'] = last_end
                                    tdict['stop-time'] = last_end + datetime.timedelta(seconds=pduur)

                                else:
                                    ptime = ptime.split('-')
                                    pstart = ptime[0].strip().split('.')
                                    prog_time = datetime.time(hour=int(pstart[0]), minute=int(pstart[1]), tzinfo=CET_CEST)
                                    tdict['offset'] = date_offset
                                    tdict['start-time'] = datetime.datetime.combine(scan_date, prog_time)
                                    pstop = ptime[1].strip().split('.')
                                    prog_time = datetime.time(hour=int(pstop[0]), minute=int(pstop[1]), tzinfo=CET_CEST)
                                    tdict['offset'] = date_offset
                                    tdict['stop-time'] = datetime.datetime.combine(scan_date, prog_time)
                                    if tdict['stop-time'] < tdict['start-time']:
                                        tdict['stop-time'] = datetime.datetime.combine(datetime.date.fromordinal(self.current_date + date_offset + 1), prog_time)

                                last_end = tdict['stop-time']

                                # and append the program to the list of programs
                                tdict = self.check_title_name(tdict)
                                with self.source_lock:
                                    self.program_data[channelid].append(tdict)

                    self.base_count += 1
                    self.day_loaded[0][offset] = True
                    # be nice to primo.eu
                    time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))

                if failure_count == 0 or retry == 1:
                    for channelid, chanid in self.chanids.items():
                        self.program_data[channelid].sort(key=lambda program: (program['start-time'],program['stop-time']))
                        self.parse_programs(chanid, 0, 'None')
                        with self.source_lock:
                            for tdict in self.program_data[channelid]:
                                self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

                        self.set_ready(channelid)
                        try:
                            infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                        except:
                            pass

                    return

        except:
            log(['\n', 'An unexpected error has occured in the %s thread\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()

    def load_detailpage(self, tdict):
        try:
            strdata = config.get_page(tdict['detail_url'][self.proc_id], 'utf-8')
            if strdata == None:
                return

            strdata = self.check_text_subs(strdata)
            strdata = self.clean_html('<root>' + strdata + '</root>')
        except:
            log(['Error Fetching detailpage %s\n' % tdict['detail_url'][self.proc_id], traceback.format_exc()])
            return None

        try:
            htmldata = ET.fromstring(strdata.encode('utf-8'))

        except:
            log("Error extracting ElementTree from:%s on primo.eu\n" % (tdict['detail_url'][self.proc_id]))
            if config.write_info_files:
                infofiles.write_raw_string('Error: %s at line %s\n\n' % (sys.exc_info()[1], sys.exc_info()[2].tb_lineno))
                infofiles.write_raw_string(strdata + '\n')

            return None

        try:
            genre = ''
            subgenre = ''
            for d in htmldata.findall('div/div[@class="details"]/div'):
                dlabel = d.findtext('label')[:-1].lower().strip()
                ddata = self.empersant(d.findtext('span')).strip()
                if ddata in (None, '-'):
                    ddata = ''

                try:
                    if dlabel in ("programmanaam", "datum en tijd", "zender"):
                        continue

                    elif dlabel == "synopsis":
                        tdict['description'] = ddata

                    elif dlabel == "titel aflevering":
                        tdict['titel aflevering'] = ddata if ((ddata != tdict['name'])) else ''
                        tdict = self.check_title_name(tdict)

                    elif dlabel == "nr. aflevering":
                        tdict['episode'] = 0 if (ddata  == '') else int(ddata)

                    elif dlabel == "seizoen":
                        tdict['season'] = 0 if (ddata == '') else int(ddata)


                    elif dlabel in  config.roletrans.keys():
                        if not config.roletrans[dlabel] in tdict['credits']:
                            tdict['credits'][config.roletrans[dlabel]] = []

                        for p in d.findall('span'):
                            name = self.empersant(p.text).split('(')[0].strip()
                            if not name in tdict['credits'][config.roletrans[dlabel]]:
                                tdict['credits'][config.roletrans[dlabel]].append(name)

                    elif dlabel == "jaar":
                        tdict['jaar van premiere'] = ddata

                    elif dlabel == "land":
                        #~ tdict['country']
                        ddata = re.sub('\.', '', ddata).upper()
                        ddata = re.split(',', ddata)
                        for c in ddata:
                            if c in config.coutrytrans.values():
                                tdict['country'] = c
                                break

                            elif c in config.coutrytrans.keys():
                                tdict['country'] = config.coutrytrans[c]
                                break

                            elif config.write_info_files:
                                infofiles.addto_detail_list(u'new country => %s' % (c))

                    elif dlabel == "genre":
                        genre = ddata if len(ddata) > 2 else ''

                    elif dlabel == "samenvatting":
                        subgenre = ddata if len(ddata) <= 25 else ''

                    #~ elif dlabel == "rating":
                        #~ pass

                    #~ elif dlabel == "minimumleeftijd":
                        #~ pass

                    #~ elif dlabel == "":
                        #~ pass

                    elif config.write_info_files:
                        infofiles.addto_detail_list(u'new primo-tag => %s: %s' % (dlabel, ddata))

                except:
                    continue

            if (genre, subgenre) in config.source_cattrans[self.proc_id].keys():
                tdict['genre'] = config.source_cattrans[self.proc_id][(genre, subgenre)][0]
                if config.source_cattrans[self.proc_id][(genre, subgenre)][1] == '':
                    tdict['subgenre'] = subgenre

                else:
                    tdict['subgenre'] = config.source_cattrans[self.proc_id][(genre, subgenre)][1]

            elif genre in config.source_cattrans[self.proc_id].keys():
                tdict['genre'] = config.source_cattrans[self.proc_id][genre][0]
                if config.source_cattrans[self.proc_id][genre][1] == '':
                    tdict['subgenre'] = subgenre
                    if subgenre != '':
                        config.new_cattrans[self.proc_id][(genre, subgenre)] = (config.source_cattrans[self.proc_id][genre][0], subgenre)

                else:
                    tdict['subgenre'] = config.source_cattrans[self.proc_id][genre][1]

                if config.write_info_files and subgenre != '':
                    infofiles.addto_detail_list(u'new primo-subgenre => %s: %s' % (genre, subgenre))

            elif genre != '':
                tdict['genre'] = genre
                tdict['subgenre'] = subgenre
                config.new_cattrans[self.proc_id][(genre, subgenre)] = (genre, subgenre)
                if config.write_info_files and subgenre != '':
                    infofiles.addto_detail_list(u'new primo-genre => %s: %s' % (genre, subgenre))

            else:
                tdict['genre'] = 'overige'
                tdict['subgenre'] = ''

        except:
            log(['Error processing Primo.eu detailpage:%s\n' % (tdict['detail_url'][self.proc_id]), traceback.format_exc()])
            return

        tdict['ID'] = tdict['prog_ID'][self.proc_id]
        tdict[self.detail_check] = True

        return tdict

# end primo_HTML

class vrt_JSON(FetchData):
    def init_channels(self):
        self.init_channel_source_ids()

    def get_url(self, type = 'channels', offset = 0, channelid = None):

        scan_date = datetime.date.fromordinal(self.current_date + offset).strftime('%Y%m%d')

        if type == 'channels':
            return  [u'%schannel/s' % (config.source_base_url[self.proc_id]), 'application/vnd.channel.vrt.be.channels_1.1+json']

        elif type == 'genres':
            return  [u'%sepg/standardgenres' % (config.source_base_url[self.proc_id]), 'application/vnd.epg.vrt.be.standardgenres_1.0+json']

        elif type == 'week' and channelid == None:
            return  [u'%sepg/schedules/%s?type=week' % (config.source_base_url[self.proc_id], scan_date),
                            'application/vnd.epg.vrt.be.schedule_3.1+json']

        elif type == 'week':
            return  [u'%sepg/schedules/%s?type=week&channel_code=%s' % (config.source_base_url[self.proc_id], scan_date, channelid),
                            'application/vnd.epg.vrt.be.schedule_3.1+json']

        elif type == 'day' and channelid == None:
            return  [u'%sepg/schedules/%s?type=day' % (config.source_base_url[self.proc_id], scan_date),
                            'application/vnd.epg.vrt.be.schedule_3.1+json']

        elif type == 'day':
            return  [u'%sepg/schedules/%s?type=day&channel_code=%s' % (config.source_base_url[self.proc_id], scan_date, channelid),
                            'application/vnd.epg.vrt.be.schedule_3.1+json']

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        # download the json feed
        url = self.get_url()
        strdata = config.get_page(url[0], 'utf-8', url[1])
        if strdata == None:
            log("Unable to get channel info from %s\n" % self.source)
            return 69  # EX_UNAVAILABLE

        channel_list = json.loads(strdata)

        # and create a file with the channels
        self.all_channels ={}
        for channel in channel_list['channels']:
            if channel['state'] == 'inactive':
                continue

            channelid = channel['code']
            self.all_channels[channelid] = {}
            self.all_channels[channelid]['name'] = self.unescape(channel['displayName']).strip()
            if channel['type'] == 'tv':
                self.all_channels[channelid]['group'] = 2

            elif channel['type'] == 'radio':
                self.all_channels[channelid]['group'] = 12

            else:
                self.all_channels[channelid]['group'] = 99

            icon = channel['logoUrl'].split('/')
            if icon[2] == 'images.vrt.be':
                self.all_channels[channelid]['icon'] =  '%s/%s' % (icon[-2] , icon[-1])
                self.all_channels[channelid]['icongrp'] = 11

            if icon[2] == 'services.vrt.be':
                self.all_channels[channelid]['icon'] = icon[-1]
                self.all_channels[channelid]['icongrp'] = 10

    def get_datetime(self, date_string, round_down = True):
        date = datetime.datetime.strptime(date_string.split('.')[0], '%Y-%m-%dT%H:%M:%S').replace(tzinfo = UTC).astimezone(CET_CEST)
        seconds = date.second
        date = date.replace(second = 0)
        if seconds > 0 and not round_down:
            date = date + datetime.timedelta(minutes = 1)

        return date

    def get_standaardgenres(self):
        url = self.get_url('genres')
        strdata = config.get_page(url[0], 'utf-8', url[1])
        genredict = json.loads(strdata)
        vrt_genres = {}
        dvb_genres = {}
        ebu_genres = {}
        for g in genredict['standardGenres']:
            vrt_genres[g['code']] = {}
            vrt_genres[g['code']]['type'] = [g['type']]
            vrt_genres[g['code']]['eid'] = [g['eid']]
            vrt_genres[g['code']]['type'] = [g['type']]
            vrt_genres[g['code']]['name'] = [g['name']]
            eid = g['eid'].split('.')
            name =  g['name'].split('>')
            if g['type'] == 'DVB':
                if eid[1] == '0':
                    #~ eid[1] = ''
                    eid.pop(1)
                    name[1] = ''

                dvb_genres[tuple(eid)] = tuple(name)

            elif g['type'] == 'EBU':
                pass

        gkeys = dvb_genres.keys()
        for k in sorted(dvb_genres.keys()):
            print(u'%s: %s,' % (k, dvb_genres[k]))

    def load_pages(self):
        failure_count = 0
        radio_channels = ('13','12','03','31','32','41','55','56','11','21','22','23','24','25')
        if config.opt_dict['offset'] > 14:
            self.set_ready()
            return

        if len(self.chanids) == 0 :
            return

        first_fetch = True
        groupitems = {}
        week_loaded = {}
        fetch_dates = []
        first_day = int(datetime.date.fromordinal(self.current_date + config.opt_dict['offset']).strftime('%w'))
        first_day = config.opt_dict['offset'] + 1 - first_day if first_day > 0 else config.opt_dict['offset'] - 6
        fetch_range = range(first_day, (config.opt_dict['offset'] + config.opt_dict['days']), 7)
        for d in range(config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days'])):
            fetch_dates.append(datetime.date.fromordinal(self.current_date + d).strftime('%Y-%m-%d'))

        for channelid in self.chanids.keys():
            groupitems[channelid] = 0
            week_loaded[channelid] = {}
            for r in range(len(fetch_range)):
                week_loaded[channelid][r] = False

        try:
            for retry in (0, 1):
                channel_cnt = 0
                for channelid, chanid in self.chanids.items():
                    channel_cnt += 1
                    failure_count = 0
                    if self.quit:
                        return

                    for offset in range(len(fetch_range)):
                        if self.quit:
                            return

                        # Check if it is already loaded
                        if week_loaded[channelid][offset]:
                            continue

                        url = self.get_url('week', fetch_range[offset], channelid)
                        log(['\n', 'Now fetching %s(xmltvid=%s%s) from vrt.be\n' % \
                            (config.channels[chanid].chan_name, config.channels[chanid].xmltvid , \
                            config.channels[chanid].get_opt('compat')), \
                            '    (channel %s of %s) for week %s of %s).\n' % \
                            (channel_cnt, len(self.chanids), offset, len(fetch_range))], 2)

                        # be nice to vrt.be
                        if not first_fetch:
                            time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))
                            first_fetch = False

                        # get the raw programming for the day
                        try:
                            strdata = config.get_page(url[0], 'utf-8', url[1])

                            if strdata == None:
                                if retry == 0:
                                    log("No data on vrt.be for %s, week=%d! First attempt.\n" % (config.channels[chanid].chan_name, offset))
                                else:
                                    log("No data on vrt.be for %s, week=%d! Final attempt.\n" % (config.channels[chanid].chan_name, offset))
                                failure_count += 1
                                self.fail_count += 1
                                continue

                        except:
                            log('Error: "%s" reading the vrt.be json page for %s, week=%d.\n' % \
                                (sys.exc_info()[1], config.channels[chanid].chan_name, offset))
                            failure_count += 1
                            self.fail_count += 1
                            continue

                        self.base_count += 1
                        week_loaded[channelid][offset] = True
                        strdata = self.check_text_subs(strdata)
                        jsondata = json.loads(strdata)
                        for p in jsondata['events']:
                            if not (p['date'] in fetch_dates and p['channel']['code'] in self.chanids.keys()):
                                continue

                            tdict = self.checkout_program_dict()
                            tdict['prog_ID'][self.proc_id] = u'vrt-%s' % (p['code'])
                            self.json_by_id[tdict['prog_ID'][self.proc_id]] = p
                            tdict['source'] = 'vrt'
                            tdict['channelid'] = chanid
                            tdict['channel']  = config.channels[chanid].chan_name

                            # The Title
                            tdict['name'] = self.unescape(p['title'])

                            # The timing
                            tdict['start-time'] = self.get_datetime(p['startTime'])
                            tdict['stop-time']  = self.get_datetime(p['endTime'], False)
                            if  tdict['name'] == None or tdict['name'] == '' or tdict['start-time'] == None or tdict['stop-time'] == None:
                                continue

                            tdict['offset'] = self.get_offset(tdict['start-time'])
                            if 'shortDescription' in p.keys() and p['shortDescription'] not in ('', None):
                                tdict['description'] = p['shortDescription']

                            elif 'description' in p.keys() and p['description'] not in ('', None):
                                tdict['description'] = p['description']

                            if 'group' in p.keys():
                                tdict['group'] = p['group']
                                groupitems[channelid] +=1

                            if 'episodeTitle' in p and p['episodeTitle'] != None:
                                if p['episodeTitle'].lower().strip() != tdict['name'].lower().strip():
                                    tdict['titel aflevering'] = p['episodeTitle'].strip()

                            # Types
                            # Aflevering, Nieuws, Sport, Programma, Film, Groep, Weerbericht, radio, Kansspelen, Volatiel,
                            # MER, Feratel beelden, Volatiel programma - geen PDC, Main Transmission, Dia, NACHTLUS op MER
                            if p['type'] in ('Aflevering', 'Programma'):
                                if 'seasonNumber' in p and p['seasonNumber'] not in ('', None):
                                    try:
                                        tdict['season'] = int(p['seasonNumber'])

                                    except:
                                        pass

                                if 'episodeNumber' in p and p['episodeNumber'] not in ('', None):
                                    try:
                                        tdict['episode'] = int(p['episodeNumber'])

                                    except:
                                        pass

                            if 'presenters' in p and isinstance(p['presenters'], list):
                                if not 'presenter' in tdict['credits']:
                                    tdict['credits']['presenter'] = []

                                for d in p['presenters']:
                                    if 'name' in d:
                                        tdict['credits']['presenter'].append(d['name'])

                            if 'cast' in p and p['cast'] not in ('', None):
                                cast_items = self.get_string_parts(p['cast'])
                                for crole, cast in cast_items.items():
                                    if len(cast) == 0:
                                        continue

                                    elif crole in config.roletrans.keys():
                                        role = config.roletrans[crole]
                                        if not role in tdict['credits']:
                                            tdict['credits'][role] = []

                                        cast = re.sub('\) ([A-Z])', '), \g<1>', \
                                                re.sub(' en ', ', ', \
                                                re.sub('e\.a\.', '', cast[0]))).split(',')
                                        for cn in cast:
                                            tdict['credits'][role].append(cn.split('(')[0].strip())

                                    elif config.write_info_files:
                                        infofiles.addto_detail_list(u'new vrt cast item => %s = %s' % (crole, cast))

                            # standardGenres
                            # actua, sport, cultuur, film, docu, humor, series, ontspanning
                            if 'standardGenres' in p.keys() and isinstance(p['standardGenres'], dict):
                                if p['standardGenres']['type'] == 'DVB':
                                    pg = p['standardGenres']['eid'].split('.')
                                    pn =  p['standardGenres']['name'].split('>')
                                    if pg in config.source_cattrans[self.proc_id].keys():
                                        tdict['genre'] = config.source_cattrans[self.proc_id][pg][0].capitalize()
                                        tdict['subgenre'] = config.source_cattrans[self.proc_id][pg][1].capitalize()

                                    elif (pg[0].lower(), ) in config.source_cattrans[self.proc_id].keys():
                                        tdict['genre'] = config.source_cattrans[self.proc_id][(pg[0].lower(), )][0].capitalize()
                                        sg = config.source_cattrans[self.proc_id][(pg[0].lower(), )][1]
                                        tdict['subgenre'] = pn[1] if sg == '' else sg.capitalize()
                                        config.new_cattrans[self.proc_id][(pg[0], pg[1])] = (tdict['genre'], tdict['subgenre'])

                                    else:
                                        tdict['genre'] = u'overige'
                                        tdict['subgenre'] = pn[1]
                                        config.new_cattrans[self.proc_id][pg] = (u'Overige', pn[1])

                            if tdict['genre'] == u'overige' and channelid in radio_channels:
                                if 'muziek' in tdict['description'] or 'muziek' in tdict['name']:
                                    tdict['genre'] = u'muziek'

                                elif 'nieuws ' in tdict['description'] or 'nieuws ' in tdict['name']:
                                    tdict['genre'] = u'nieuws/actualiteiten'

                                elif 'sport' in tdict['description'] or 'sport' in tdict['name']:
                                    tdict['genre'] = u'sport'

                                elif 'actualiteiten' in tdict['description'] or 'actualiteiten' in tdict['name']:
                                    tdict['genre'] = u'nieuws/actualiteiten'

                            tdict['video']['breedbeeld'] = True if 'aspectRatio' in p.keys() and p['aspectRatio'] == '16:9' else False
                            tdict['video']['HD'] = True if 'videoFormat' in p.keys() and p['videoFormat'] == 'HD' else False
                            tdict['teletekst'] = True if 'hasTTSubTitles' in p.keys() and p['hasTTSubTitles'] else False
                            tdict['rerun'] = True if 'isRepeat' in p.keys() and p['isRepeat'] else False
                            if 'categories' in p.keys() and p['categories'] != None and p['categories'].strip() != '':
                                if p['categories'].strip() in  config.vrtkijkwijzer:
                                    tdict['kijkwijzer'].append(config.vrtkijkwijzer[p['categories'].strip()])

                                elif config.write_info_files:
                                    infofiles.addto_detail_list(u'new vrt categorie => %s' % (p['categories']))

                            if config.write_info_files:
                                for item in p.keys():
                                    if item.strip() not in (u'code', u'date', u'channel', u'programme', u'group', u'season', u'episode', u'brand',
                                            u'twitterHashTag', u'onDemandURL', u'episodeOnDemandURL', u'websiteURL', u'secondScreenURL',
                                            u'images', u'imagesLink', u'trailerURL', u'trailerPictureURL', u'playlistLink', u'playlistSiteURL',
                                            u'onAir', u'geoblocking', u'isLive', u'hidePrintedPress', u'bought', u'onDemand',
                                            u'updateFlag', u'whatsonProductId', u'reconcileId',u'pdc' ,
                                            u'title', u'shortTitle', u'startTime', u'endTime', u'duration', u'originalStartTime',
                                            u'seasonNumber', u'seasonEid', u'seasonTitle', u'seasonNumberOfEpisodes', u'hideSeasonNumber',
                                            u'episodeNumber', u'episodeEid', u'episodeSequenceNumber', u'episodeTitle', u'hideEpisodeNumber',
                                            u'aspectRatio', u'videoFormat', u'hasTTSubTitles', u'isRepeat',
                                            u'presenters', u'cast', u'type', u'categories', u'standardGenres',
                                            u'shortDescription', u'description', u'hasAudioDescription'):
                                        infofiles.addto_detail_list(u'new vrt key => %s = %s' % (item, p[item]))

                            tdict = self.check_title_name(tdict)
                            with self.source_lock:
                                self.program_data[channelid].append(tdict)

                    if failure_count == 0 or retry == 1:
                        with self.source_lock:
                            self.program_data[channelid].sort(key=lambda program: (program['start-time'],program['stop-time']))
                            # 1 or more groups were encountered
                            if groupitems[channelid] > 0:
                                group_start = False
                                for p in self.program_data[channelid][:]:
                                    if 'group' in p.keys():
                                        # Collecting the group
                                        if not group_start:
                                            group = []
                                            start = p['start-time']
                                            group_start = True

                                        group.append(p.copy())
                                        group_duur = p['stop-time'] - start

                                    elif group_start:
                                        # Repeating the group
                                        group_start = False
                                        group_eind = p['start-time']
                                        group_length = group_eind - start
                                        if group_length > datetime.timedelta(days = 1):
                                            # Probably a week was not grabbed
                                            group_eind -= datetime.timedelta(days = int(group_length.days))

                                        repeat = 0
                                        while True:
                                            repeat+= 1
                                            for g in group[:]:
                                                gdict = g.copy()
                                                gdict['prog_ID'][self.proc_id] = ''
                                                gdict['rerun'] = True
                                                gdict['start-time'] += repeat*group_duur
                                                gdict['stop-time'] += repeat*group_duur
                                                if gdict['start-time'] < group_eind:
                                                    if gdict['stop-time'] > group_eind:
                                                        gdict['stop-time'] = group_eind

                                                    self.program_data[channelid].append(gdict)

                                                else:
                                                    break

                                            else:
                                                continue

                                            break

                        self.parse_programs(chanid, 0, 'None')
                        self.set_ready(channelid)
                        for day in range( config.opt_dict['offset'], (config.opt_dict['offset'] + config.opt_dict['days'])):
                            self.day_loaded[channelid][day] = True
                        try:
                            infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                        except:
                            pass

        except:
            log(['\n', 'An unexpected error has occured in the %s thread:\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()
            return None

# end vrt_JSON

class Virtual_Channels(FetchData):
    """
    This source is for creating combined channels
    """
    def get_channels(self):
        self.all_channels = config.virtual_channellist

# end Virtual_Channels

class oorboekje_HTML(FetchData):
    """
    Get all available days of programming for the requested radio channels
    from the oorboekje.nl page. Based on FetchData Class
    """
    def init_channels(self):

        self.init_channel_source_ids()

    def get_url(self, type = None, offset = 0):

        week_day = datetime.date.fromordinal(self.current_date + offset).isoweekday()

        if type == 'channels':
            return  config.source_base_url[self.proc_id]

        else:
            return u'%sprogram.php?dag=%s' % (config.source_base_url[self.proc_id], week_day)

    def get_channels(self):
        """
        Get a list of all available channels and store these
        in all_channels.
        """

        try:
            strdata = config.get_page(self.get_url('channels'))
            if self.get_channel_lineup(strdata) == 69:
                log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
                return 69  # EX_UNAVAILABLE

        except:
            self.fail_count += 1
            log(["Unable to get channel info from %s\n" % self.source, traceback.format_exc()])
            return 69  # EX_UNAVAILABLE

    def get_channel_lineup(self, chandata):

        try:
            if not isinstance(chandata, (str, unicode)):
                return 69

            strdata = self.get_source_regex(chandata, 0, 1, 0)
            strdata = self.clean_html(strdata)
            chgroup = 11
            for ch in self.get_source_regex(strdata, 1, 2):
                if not '"/zenderinformatie/' in ch:
                    if self.get_source_regex(ch, 2) != None:
                        chgroup = 17

                    continue

                channelid = self.get_source_regex(ch, 3, 1, 1).lower()
                channame = self.get_source_regex(ch, 4)
                if channame:
                    chanlogo = channame.group(1)
                    channame = channame.group(2)

                else:
                    chanlogo = None
                    channame = self.get_source_regex(ch, 5, 1, 1)

                regionname = self.get_source_regex(channame, 6)
                channame = self.empersant(re.sub('<SPAN.*?</SPAN>', '', channame).strip())
                if regionname != None and not '(' in regionname.group(1):
                    channame = u'%s %s' % (channame, regionname.group(1))

                if not channelid in self.all_channels.keys():
                    self.all_channels[channelid] = {}
                    self.all_channels[channelid]['name'] = channame
                    self.all_channels[channelid]['group'] = chgroup
                    if chanlogo != None:
                        self.all_channels[channelid]['icon'] = chanlogo
                        self.all_channels[channelid]['icongrp'] = 14

        except:
            self.fail_count += 1
            log(traceback.format_exc())
            return 69

    def load_pages(self):
        failure_count = 0
        if config.opt_dict['offset'] > 7:
            self.set_ready()
            return

        if len(self.chanids) == 0:
            return

        try:
            for retry in (0, 1):
                failure_count = 0
                for offset in range( config.opt_dict['offset'], min((config.opt_dict['offset'] + config.opt_dict['days']), 7)):
                    if self.quit:
                        return

                    # Check if it is allready loaded
                    if self.day_loaded[0][offset] != False:
                        continue

                    log(['\n', 'Now fetching channels from oorboekje.nl for day %s of %s\n' % \
                        (offset, config.opt_dict['days'])], 2)

                    # get the raw programming for the day
                    try:
                        channel_url = self.get_url(offset=offset)
                        strdata = config.get_page(channel_url)

                        if strdata == None:
                            if retry == 0:
                                log("Skip day=%s on oorboekje.nl. No data! First attempt.\n" % (offset))
                            else:
                                log("Skip day=%s on oorboekje.nl. No data! Final attempt.\n" % (offset))

                            failure_count += 1
                            self.fail_count += 1
                            continue

                        #~ fetchdate = self.get_source_regex(strdata, 7)
                        #~ if fetchdate == None or datetime.date.fromordinal(self.current_date + offset) != \
                          #~ datetime.date(int(fetchdate.group(3)), int(fetchdate.group(2)), int(fetchdate.group(1))):
                            #~ log('Invalid date for oorboekje.nl for day %s.\n' % offset)

                    except:
                        log('Error: "%s" reading the oorboekje.nl basepage for day %s.\n' % \
                            (sys.exc_info()[1], offset))
                        failure_count += 1
                        self.fail_count += 1
                        continue

                    strdata = self.clean_html(strdata)
                    for ch in self.get_source_regex(strdata, 8, 2):
                        chan = self.get_source_regex(ch, 9)
                        if chan == None:
                            continue

                        channelid = chan.group(1).lower()
                        if channelid == 'r100procentnl':
                            channelid = '100procentnl'

                        channame = self.empersant(re.sub('<SPAN.*?</SPAN>', '', chan.group(2)).strip())
                        if not channelid in self.all_channels:
                             self.all_channels[channelid] ={}

                        self.all_channels[channelid]['name'] = channame
                        if not channelid in self.chanids.keys():
                            continue

                        chanid = self.chanids[channelid]
                        date_offset = offset
                        last_end = datetime.datetime.combine(datetime.date.fromordinal(self.current_date + offset),
                                                                                        datetime.time(hour=0, tzinfo=CET_CEST))
                        scan_date = datetime.date.fromordinal(self.current_date + date_offset)
                        pcount = 0
                        for p in self.get_source_regex(ch, 10, 2):
                            if 'style="text-indent: 0px;"' in p[0]:
                                continue

                            tdict = self.checkout_program_dict()
                            tdict['source'] = u'oorboekje'
                            tdict['channelid'] = chanid
                            tdict['channel'] = config.channels[chanid].chan_name

                            # The Title
                            tdict['name'] = self.empersant(p[4].strip())
                            if  tdict['name'] == None or tdict['name'] == '':
                                log('Can not determine program title\n')
                                continue

                            pcount+=1
                            ptime = datetime.time(int(p[1]), int(p[2]), tzinfo=CET_CEST)
                            tdict['offset'] = date_offset
                            tdict['start-time'] = datetime.datetime.combine(scan_date, ptime)
                            if tdict['start-time'] < last_end:
                                if pcount > 2:
                                    scan_date = datetime.date.fromordinal(self.current_date + date_offset+1)
                                    tdict['start-time'] = datetime.datetime.combine(scan_date, ptime)

                            last_end = tdict['start-time']
                            ptime = self.get_source_regex(p[3], 11)
                            if ptime != None:
                                ptime = datetime.time(int(ptime.group(1)), int(ptime.group(2)), tzinfo=CET_CEST)
                                tdict['stop-time'] = datetime.datetime.combine(scan_date, ptime)
                                if tdict['stop-time'] < last_end:
                                    scan_date = datetime.date.fromordinal(self.current_date + date_offset+1)
                                    tdict['stop-time'] = datetime.datetime.combine(scan_date, ptime)

                                last_end = tdict['stop-time']

                            for picon in self.get_source_regex(p[5], 12, 2):
                                if picon == "herhaling":
                                    tdict['rerun'] = True

                                elif picon == "nonstop":
                                    tdict['genre'] = u'muziek'

                            desc = re.sub('<.*?>', '', self.empersant(p[5])).strip()
                            if tdict['genre'] == u'overige':
                                if 'muziek' in desc or 'muziek' in tdict['name']:
                                    tdict['genre'] = u'muziek'

                                elif 'nieuws ' in desc or 'nieuws ' in tdict['name']:
                                    tdict['genre'] = u'nieuws/actualiteiten'

                                elif 'sport' in desc or 'sport' in tdict['name']:
                                    tdict['genre'] = u'sport'

                                elif 'actualiteiten' in desc or 'actualiteiten' in tdict['name']:
                                    tdict['genre'] = u'nieuws/actualiteiten'


                            tdict['description'] = desc
                            desc_items = self.get_string_parts(desc)
                            for crole, cast in desc_items.items():
                                if len(cast) == 0:
                                    continue

                                elif crole in config.roletrans.keys():
                                    role = config.roletrans[crole]
                                    if not role in tdict['credits']:
                                        tdict['credits'][role] = []

                                    cast = re.sub('\) ([A-Z])', '), \g<1>', \
                                            re.sub(' & ', ', ', \
                                            re.sub(' en ', ', ', \
                                            re.sub('e\.a\.', '', cast[0])))).split(',')
                                    for cn in cast:
                                        tdict['credits'][role].append(cn.split('(')[0].strip())

                                elif config.write_info_files:
                                    infofiles.addto_detail_list(u'new oorboekje desc item => %s = %s' % (crole, cast))



                            # and append the program to the list of programs
                            tdict = self.check_title_name(tdict)
                            with self.source_lock:
                                self.program_data[channelid].append(tdict)



                    self.base_count += 1
                    self.day_loaded[0][offset] = True
                    # be nice to oorboekje.nl
                    time.sleep(random.randint(config.nice_time[0], config.nice_time[1]))

                if failure_count == 0 or retry == 1:
                    for channelid, chanid in self.chanids.items():
                        # Add starttime of the next program as the endtime
                        with self.source_lock:
                            self.program_data[channelid].sort(key=lambda program: (program['start-time']))
                            self.add_endtimes(channelid, 7)

                        self.parse_programs(chanid, 0, 'None')
                        self.set_ready(channelid)
                        with self.source_lock:
                            for tdict in self.program_data[channelid]:
                                self.program_by_id[tdict['prog_ID'][self.proc_id]] = tdict

                        try:
                            infofiles.write_fetch_list(self.program_data[channelid], chanid, self.source, self.proc_id)

                        except:
                            pass

                    return

        except:
            log(['\n', 'An unexpected error has occured in the %s thread\n' %  (self.source), traceback.format_exc()], 0)
            self.set_ready()

# end oorboekje_HTML

class Channel_Config(Thread):
    """
    Class that holds the Channel definitions and manages the data retrieval and processing
    """
    def __init__(self, chanid = 0, name = '', group = 99):
        tname = ('channel-%s'% name).encode('utf-8', 'replace')
        Thread.__init__(self, name = tname)
        # Flag to stop the thread
        self.quit = False

        # Flags to indicate the data is in
        self.source_data = {}
        self.detail_data = Event()
        self.child_data = Event()
        self.cache_return = Queue()
        self.channel_lock = Lock()

        # Flag to indicate all data is processed
        self.ready = False
        self.state = 0
        self.source = None

        self.active = False
        self.is_child = False
        self.is_virtual_sub = False
        self.virtual_start = None
        self.virtual_end = None
        self.child_programs = []
        self.counter = 0
        self.chanid = chanid
        self.xmltvid = self.chanid
        self.chan_name = name
        self.group = group
        self.source_id = {}
        self.icon_source = -1
        self.icon = ''

        for index in range(xml_output.source_count):
            self.source_id[index] = ''
            self.source_data[index] = Event()

        self.counters = {}
        self.counters['none'] = 0
        self.counters['cache'] = 0
        self.counters['fail'] = 0
        self.counters['ttvdb'] = 0
        self.counters['ttvdb_fail'] = 0
        self.counters['fetch'] = {}
        self.counters['fetched'] = {}
        self.counters['fetch'][-1] = 0
        for index in xml_output.detail_sources:
            self.counters['fetch'][index] = 0
            self.counters['fetched'][index] = 0

        # This will contain the final fetcheddata
        self.all_programs = []
        self.current_prime = ''

        self.opt_dict = {}
        self.prevalidate_opt = {}
        self.opt_dict['xmltvid_alias'] = None
        self.opt_dict['prime_source'] = -1
        self.prevalidate_opt['prime_source'] = -1
        self.opt_dict['prefered_description'] = -1
        self.opt_dict['append_tvgidstv'] = True
        self.opt_dict['add_hd_id'] = False

        self.opt_dict['disable_source'] = []
        self.opt_dict['disable_detail_source'] = []
        self.opt_dict['disable_ttvdb'] = False

    def validate_settings(self):

        if not self.active and not self.is_child:
            return

        if self.prevalidate_opt['prime_source'] == -1:
            config.validate_option('prime_source', self)

        else:
            config.validate_option('prime_source', self, self.prevalidate_opt['prime_source'])

        config.validate_option('prefered_description', self)
        config.validate_option('overlap_strategy', self)
        config.validate_option('max_overlap', self)
        config.validate_option('desc_length', self)
        config.validate_option('slowdays', self)
        if self.group in config.ttvdb_disabled_groups:
            self.opt_dict['disable_ttvdb'] = True

        if self.opt_dict['xmltvid_alias'] != None:
            self.xmltvid = self.opt_dict['xmltvid_alias']

        elif (config.configversion < 2.208 or self.get_opt('legacy_xmltvids') == True):
            xmltvid = self.chanid.split('-',1)
            try:
                self.xmltvid = xmltvid[1] if int(xmltvid[0]) < 4 else self.chanid

            except:
                self.xmltvid = self.chanid

    def run(self):

        if not self.active and not self.is_child:
            self.ready = True
            for index in xml_output.source_order:
                self.source_data[index].set()

            self.detail_data.set()
            return

        if not self.is_child:
            self.child_data.set()

        try:
            # Create the merge order
            self.merge_order = []
            last_merge = []
            ps = self.opt_dict['prime_source']
            if (self.get_source_id(ps) != '') and not self.get_opt('disable_source', ps):
                if self.get_source_id(ps) in config.no_genric_matching[ps]:
                    last_merge.append(ps)

                else:
                    self.merge_order.append(ps)

            for index in xml_output.source_order:
                if (self.get_source_id(index) != '') and index != ps and not self.get_opt('disable_source', index):
                    if self.get_source_id(index) in config.no_genric_matching[index]:
                        last_merge.append(index)

                    else:
                        self.merge_order.append(index)

                elif index != ps:
                    self.source_data[index].set()

            self.merge_order.extend(last_merge)
            xml_data = False
            # Retrieve and merge the data from the available sources.
            for index in self.merge_order:
                channelid = self.get_source_id(index)
                self.source = index
                self.state = 1
                while not self.source_data[index].is_set():
                    # Wait till the event is set by the source, but check every 5 seconds for an unexpected break or wether the source is still alive
                    self.source_data[index].wait(5)
                    if self.quit:
                        self.ready = True
                        return

                    # Check if the source is still alive
                    if not xml_output.channelsource[index].is_alive():
                        self.source_data[index].set()
                        break

                self.state = 0
                self.source = None
                if self.source_data[index].is_set():
                    if len(xml_output.channelsource[index].program_data[channelid]) == 0:
                        if not ((index == 1 and 0 in self.merge_order) or index == 11):
                            log('No Data from %s for channel: %s\n'% (xml_output.channelsource[index].source, self.chan_name))
                            continue

                    if xml_data == False:
                        xml_data = True
                        prime_source = xml_output.channelsource[index].proc_id

                    xml_data = True
                    xml_output.channelsource[index].merge_sources(self.chanid,  prime_source, self.counter)
                    xml_output.channelsource[index].parse_programs(self.chanid, 1, 'None')
                    for i in range(0, len(self.all_programs)):
                        self.all_programs[i] = xml_output.channelsource[index].checkout_program_dict(self.all_programs[i])

            if self.chanid in config.combined_channels.keys():
                for c in config.combined_channels[self.chanid]:
                    if c['chanid'] in config.channels:
                        self.source = c['chanid']
                        self.state = 2
                        while not config.channels[c['chanid']].child_data.is_set():
                            # Wait till the event is set by the child, but check every 5 seconds for an unexpected break or wether the child is still alive
                            config.channels[c['chanid']].child_data.wait(5)
                            if self.quit:
                                self.ready = True
                                return

                            # Check if the child is still alive
                            if not config.channels[c['chanid']].is_alive():
                                break

                        self.state = 0
                        self.source = None
                        if len(config.channels[c['chanid']].child_programs) == 0:
                            log('No Data from %s for channel: %s\n'% (config.channels[c['chanid']].chan_name, self.chan_name))

                        elif self.child_data.is_set():
                            # We always merge as there might be restrictions
                            xml_data = True
                            xml_output.channelsource[0].merge_sources(self.chanid,  None, self.counter, c)
                            xml_output.channelsource[0].parse_programs(self.chanid, 1, 'None')
                            for i in range(0, len(self.all_programs)):
                                self.all_programs[i] = xml_output.channelsource[0].checkout_program_dict(self.all_programs[i])


            if self.is_child:
                self.child_programs = deepcopy(self.all_programs) if self.active else self.all_programs
                self.child_data.set()
                if not self.active:
                    self.ready = True
                    return

            # And get the detailpages
            if len(self.all_programs) == 0:
                self.detail_data.set()

            else:
                self.state = 3
                self.get_details()
                self.state = 4
                while not self.detail_data.is_set():
                    self.detail_data.wait(5)
                    #~ if config.write_info_files:
                        #~ infofiles.write_raw_string('%s is waiting for details' % ( self.chan_name))
                    if self.quit:
                        self.ready = True
                        return

                    # Check if the sources are still alive
                    for s in xml_output.detail_sources:
                        if xml_output.channelsource[s].is_alive():
                            break

                    else:
                        self.detail_data.set()
                        log('Detail sources: %s, %s and %s died.\n So we stop waiting for the pending details for channel %s\n' \
                            % (xml_output.channelsource[0].source, xml_output.channelsource[1].source, xml_output.channelsource[9].source, self.chan_name))

                self.state = 0
                self.all_programs = self.detailed_programs

            # And log the results
            with xml_output.output_lock:
                xml_output.cache_count += self.counters['cache']
                xml_output.ttvdb_count += self.counters['ttvdb']
                xml_output.ttvdb_fail_count += self.counters['ttvdb_fail']
                xml_output.progress_counter+= 1
                counter = xml_output.progress_counter

            log_array = ['\n', 'Detail statistics for %s (channel %s of %s)\n' % (self.chan_name, counter, config.chan_count)]
            log_array.append( '%6.0f cache hit(s)\n' % (self.counters['cache']))
            if self.get_opt('fast'):
                log_array.append('%6.0f without details in cache\n' % self.counters['none'])
                log_array.append('\n')
                log_array.append('%6.0f succesful ttvdb lookups\n' % self.counters['ttvdb'])
                log_array.append('%6.0f failed ttvdb lookups\n' % self.counters['ttvdb_fail'])

            else:
                log_array.append('%6.0f detail fetch(es) from tvgids.nl\n' % self.counters['fetched'][0])
                log_array.append('%6.0f detail fetch(es) from tvgids.tv\n' % self.counters['fetched'][1])
                log_array.append('%6.0f detail fetch(es) from primo.eu\n' % self.counters['fetched'][9])
                log_array.append('%6.0f failure(s)\n' % self.counters['fail'])
                log_array.append('%6.0f without detail info\n' % self.counters['none'])
                log_array.append('\n')
                log_array.append('%6.0f succesful ttvdb lookups\n' % self.counters['ttvdb'])
                log_array.append('%6.0f    failed ttvdb lookups\n' % self.counters['ttvdb_fail'])
                log_array.append('\n')
                log_array.append('%6.0f left in the tvgids.nl queue to process\n' % (xml_output.channelsource[0].detail_request.qsize()))
                log_array.append('%6.0f left in the tvgids.tv queue to process\n' % (xml_output.channelsource[1].detail_request.qsize()))
                log_array.append('%6.0f left in the primo.eu queue to process\n' % (xml_output.channelsource[9].detail_request.qsize()))

            log_array.append('\n')
            log(log_array, 4, 3)

            # a final check on the sanity of the data
            xml_output.channelsource[0].parse_programs(self.chanid, 1)

            # Split titles with colon in it
            # Note: this only takes place if all days retrieved are also grabbed with details (slowdays=days)
            # otherwise this function might change some titles after a few grabs and thus may result in
            # loss of programmed recordings for these programs.
            # Also check if a genric genre does aply
            for g, chlist in config.generic_channel_genres.items():
                if self.chanid in chlist:
                    gen_genre = g
                    break

            else:
                gen_genre = None

            for i, v in enumerate(self.all_programs):
                self.all_programs[i] = self.title_split(v)
                if gen_genre != None and self.all_programs[i]['genre'] in (u'overige', u''):
                    self.all_programs[i]['genre'] = gen_genre

            if self.get_opt('add_hd_id'):
                self.opt_dict['mark_hd'] = False
                xml_output.create_channel_strings(self.chanid, False)
                xml_output.create_program_string(self.chanid, False)
                xml_output.create_channel_strings(self.chanid, True)
                xml_output.create_program_string(self.chanid, True)

            else:
                xml_output.create_channel_strings(self.chanid)
                xml_output.create_program_string(self.chanid)

            if config.write_info_files:
                infofiles.write_raw_list()

            self.ready = True

        except:
            log(['\n', 'An unexpected error has occured in the %s thread:\n' %  (self.chan_name), traceback.format_exc(), \
                '\n', 'If you want assistence, please attach your configuration and log files!\n', \
                '     %s\n' % (config.config_file), '     %s\n' % (config.log_file)],0)

            self.ready = True
            for source in xml_output.channelsource.values():
                if source.is_alive():
                    source.cache_return.put('quit')
                    source.quit = True

            for channel in config.channels.values():
                if channel.is_alive():
                    channel.cache_return.put('quit')
                    channel.quit = True

            return(97)

    def use_cache(self, tdict, cached):
        # copy the cached information, except the start/end times, rating and clumping,
        # these may have changed.
        # But first checkout the dict
        cached = xml_output.channelsource[0].checkout_program_dict(cached)
        try:
            clump  = tdict['clumpidx']

        except LookupError:
            clump = False

        cached['start-time'] = tdict['start-time']
        cached['stop-time']  = tdict['stop-time']
        if clump:
            cached['clumpidx'] = clump

        # Make sure we do not overwrite fresh info with cashed info
        if tdict['description'] > cached['description']:
            cached['description'] = tdict['description']

        if not 'prefered description' in cached.keys():
            cached['prefered description'] = tdict['prefered description']

        elif tdict['prefered description'] > cached['prefered description']:
            cached['prefered description'] = tdict['prefered description']

        for fld in ('name', 'titel aflevering', 'originaltitle', 'jaar van premiere', 'airdate', 'country', 'star-rating', 'omroep'):
            if tdict[fld] != '':
                cached[fld] = tdict[fld]

        if re.sub('[-,. ]', '', cached['name']) == re.sub('[-,. ]', '', cached['titel aflevering']):
            cached['titel aflevering'] = ''

        for fld in ('season', 'episode'):
            if tdict[fld] != 0:
                cached[fld] = int(tdict[fld])

        if tdict['rerun'] == True:
            cached['rerun'] = True

        if len(tdict['kijkwijzer']) > 0:
            for item in tdict['kijkwijzer']:
                if not item in cached['kijkwijzer']:
                    cached['kijkwijzer'].append(item)

        return cached

    def update_counter(self, cnt_type, source_id=None, cnt_add=True, cnt_change=1):
        if not isinstance(cnt_change, int) or cnt_change == 0:
            return

        with self.channel_lock:
            if not cnt_type in self.counters:
                if source_id == None:
                    self.counters[cnt_type] = 0

                else:
                    self.counters[cnt_type] = {}
                    self.counters[cnt_type][source_id] = 0

            if isinstance(self.counters[cnt_type], int):
                if cnt_add:
                    self.counters[cnt_type] += cnt_change

                else:
                    self.counters[cnt_type] -= cnt_change

            elif isinstance(self.counters[cnt_type], dict):
                if source_id == None:
                    source_id = 0

                if isinstance(self.counters[cnt_type][source_id], int):
                    if cnt_add:
                        self.counters[cnt_type][source_id] += cnt_change

                    else:
                        self.counters[cnt_type][source_id] -= cnt_change

    def get_counter(self):
        with self.channel_lock:
            self.fetch_counter += 1
            return 100*float(self.fetch_counter)/float(self.nprograms)

    def get_source_id(self, source):
        if source in self.source_id.keys():
            return self.source_id[source]

        return ''

    def get_opt(self, opt, source_id = None):
        retval = None
        if opt in ('disable_source', 'disable_detail_source'):
            if source_id in self.opt_dict[opt] or source_id in config.opt_dict[opt]:
                return True

            else:
                return False

        if opt == 'disable_ttvdb':
            if config.opt_dict[opt] or self.opt_dict[opt]:
                return True

            else:
                False

        if opt == 'compat':
            if 'compat' in self.opt_dict.keys():
                if self.opt_dict['compat']:
                    return '.tvgids.nl'

                else:
                    return ''

            elif config.opt_dict['compat']:
                return '.tvgids.nl'

            else:
                return ''

        if opt in self.opt_dict.keys():
            retval = self.opt_dict[opt]

        elif opt in config.opt_dict.keys():
            retval = config.opt_dict[opt]

        if retval == None:
            if opt == 'slowdays':
                if self.get_opt('fast'):
                    retval = 0

                else:
                    retval = config.opt_dict['days']

        return retval

    def get_details(self):
        """
        Given a list of programs, from the several sources, retrieve program details
        """
        # Check if there is data
        self.detailed_programs = []
        if len(self.all_programs) == 0:
            return

        programs = self.all_programs[:]

        if self.get_opt('fast'):
            log(['\n', 'Now Checking cache for %s programs on %s(xmltvid=%s%s)\n' % \
                (len(programs), self.chan_name, self.xmltvid, self.get_opt('compat')), \
                '    (channel %s of %s) for %s days.\n' % (self.counter, config.chan_count, config.opt_dict['days'])], 2)

        else:
            log(['\n', 'Now fetching details for %s programs on %s(xmltvid=%s%s)\n' % \
                (len(programs), self.chan_name, self.xmltvid, self.get_opt('compat')), \
                '    (channel %s of %s) for %s days.\n' % (self.counter, config.chan_count, config.opt_dict['days'])], 2)

        # randomize detail requests
        self.fetch_counter = 0
        self.nprograms = len(programs)
        fetch_order = list(range(0,self.nprograms))
        random.shuffle(fetch_order)

        for i in fetch_order:
            if self.quit:
                self.ready = True
                return

            try:
                if programs[i] == None:
                    continue

            except:
                log(traceback.format_exc())
                if config.write_info_files:
                    infofiles.write_raw_string('Error: %s with index %s\n' % (sys.exc_info()[1], i))

                continue

            p = programs[i]
            logstring = u'%s-%s: %s' % \
                                (p['start-time'].strftime('%d %b %H:%M'), \
                                p['stop-time'].strftime('%H:%M'), \
                                p['name'])

            # We only fetch when we are in slow mode and slowdays is not set to tight
            no_fetch = (self.get_opt('fast') or p['offset'] >= (config.opt_dict['offset'] + self.get_opt('slowdays')))

            # check the cache for this program's ID
            # If not found, check the various ID's and (if found) make it the prime one
            xml_output.program_cache.cache_request.put({'task':'query_id', 'parent': self, 'program': p})
            cache_id = self.cache_return.get(True)
            if cache_id =='quit':
                self.ready = True
                return

            if cache_id != None:
                xml_output.program_cache.cache_request.put({'task':'query', 'parent': self, 'pid': cache_id})
                cached_program = self.cache_return.get(True)
                if cached_program =='quit':
                    self.ready = True
                    return

                # check if it contains detail info from tvgids.nl or (if no nl-url known, or in no_fetch mode) tvgids.tv
                if cached_program != None and \
                    (no_fetch or \
                        cached_program[xml_output.channelsource[0].detail_check] or \
                        (p['detail_url'][0] == '' and \
                            (cached_program[xml_output.channelsource[9].detail_check] or \
                                (p['detail_url'][9] == '' and \
                                cached_program[xml_output.channelsource[1].detail_check])))):
                        log(u'      [cached] %s:(%3.0f%%) %s\n' % (self.chan_name, self.get_counter(), logstring), 8, 1)
                        self.update_counter('cache')
                        p = self.use_cache(p, cached_program)
                        if not self.get_opt('disable_ttvdb'):
                            if p['genre'].lower() == u'serie/soap' and p['titel aflevering'] != '' and p['season'] == 0:
                                self.update_counter('fetch', -1)
                                xml_output.ttvdb.detail_request.put({'tdict':p, 'parent': self, 'task': 'update_ep_info'})
                                continue

                        self.detailed_programs.append(p)
                        continue

            # Either we are fast-mode, outside slowdays or there is no url. So we continue
            no_detail_fetch = (no_fetch or ((p['detail_url'][0] == '') and \
                                                                (p['detail_url'][9] == '') and \
                                                                (p['detail_url'][1] == '')))

            if no_detail_fetch:
                log(u'    [no fetch] %s:(%3.0f%%) %s\n' % (self.chan_name, self.get_counter(), logstring), 8, 1)
                self.update_counter('none')
                if not self.get_opt('disable_ttvdb'):
                    if p['genre'].lower() == u'serie/soap' and p['titel aflevering'] != '' and p['season'] == 0:
                        self.update_counter('fetch', -1)
                        xml_output.ttvdb.detail_request.put({'tdict':p, 'parent': self, 'task': 'update_ep_info'})
                        continue

                self.detailed_programs.append(p)

                continue

            for src_id in xml_output.detail_sources:
                if not self.get_opt('disable_detail_source', src_id) and p['detail_url'][src_id] != '':
                    self.update_counter('fetch', src_id)
                    xml_output.channelsource[src_id].detail_request.put({'tdict':p, 'cache_id': cache_id, 'logstring': logstring, 'parent': self})
                    break

        # Place terminator items in the queue
        for src_id in xml_output.detail_sources:
            if self.counters['fetch'][src_id] > 0:
                xml_output.channelsource[src_id].detail_request.put({'last_one': True, 'parent': self})
                break

        else:
            if not self.get_opt('disable_ttvdb'):
                xml_output.ttvdb.detail_request.put({'task': 'last_one', 'parent': self})

            else:
                self.detail_data.set()

    def title_split(self,program):
        """
        Some channels have the annoying habit of adding the subtitle to the title of a program.
        This function attempts to fix this, by splitting the name at a ': '.
        """
        # Some programs (BBC3 when this happened) have no genre. If none, then set to a default
        if program['genre'] is None:
            program['genre'] = 'overige';

        ptitle = program['name']
        psubtitle = program['titel aflevering']
        if  ptitle == None or ptitle == '':
            return program

        # exclude certain programs
        if  ('titel aflevering' in program and psubtitle != '')  \
          or ('genre' in program and program['genre'].lower() in ['movies','film']) \
          or (ptitle.lower() in config.notitlesplit):
            return program

        # and do the title split test
        p = ptitle.split(':')
        if len(p) >1:
            log('Splitting title \"%s\"\n' %  ptitle, 64)
            program['name'] = p[0].strip()
            program['titel aflevering'] = "".join(p[1:]).strip()
            if config.write_info_files:
                infofiles.addto_detail_list(unicode('Name split = %s + %s' % (program['name'] , program['titel aflevering'])))

        return program

# end Channel_Config

class XMLoutput:
    '''
    This class collects the data and creates the output
    '''
    def __init__(self):

        self.xmlencoding = 'UTF-8'
        # This will contain the cache
        self.program_cache = None
        # Thes will contain the seperate XML strings
        self.xml_channels = {}
        self.xml_programs = {}
        self.progress_counter = 0

        # We have several sources of logos, the first provides the nice ones, but is not
        # complete. We use the tvgids logos to fill the missing bits.
        self.logo_source_preference = [99, 4, 5, 6, 7, 8, 9, 10, 11, 3, 0, 1, 2]
        self.logo_provider = ['http://graphics.tudelft.nl/~paul/logos/gif/64x64/',
                                        'http://static.tvgids.nl/gfx/zenders/',
                                        'http://s4.cdn.sanomamedia.be/a/epg/q100/w60/h/',
                                        'http://staticfiles.rtl.nl/styles/img/logos/',
                                        'http://212.142.41.211/ChannelLogos/02/',
                                        'https://wp20-images-nl-dynamic.horizon.tv/ChannelLogos/02/',
                                        'http://img.humo.be/q100/w100/h100/epglogos/',
                                        'http://www-assets.npo.nl/uploads/',
                                        'http://2.nieuwsbladcdn.be/extra/assets/img/tvgids/',
                                        'http://www.primo.eu/sites/all/modules/primo_integrations/tv-logos/']

        self.source_count = 11
        self.sources = {0: 'tvgids.nl', 1: 'tvgids.tv', 2: 'rtl.nl', 3: 'teveblad.be', 4: 'npo.nl',
                                  5: 'horizon.tv', 6: 'humo.be', 7: 'vpro.nl', 8: 'nieuwsblad.be', 9:'primo.eu',
                                  10: 'vrt.be', 11: 'virtual', 12: 'oorboekje.nl'}
        self.source_order = [7, 0, 1, 5, 9, 6, 8, 2, 4, 10, 11, 12]
        self.sourceid_order = [0, 9, 1, 2, 4, 7, 5, 6, 7, 8, 10, 11, 12]
        self.source_count = len(self.sources)
        self.detail_sources = (0, 9, 1)
        self.prime_source_order = (2, 4, 7, 0, 5, 1, 9, 6, 8, 10, 11, 12)
        self.channelsource = {}
        self.channelsource[0] = tvgids_JSON(0, 'tvgids.nl', 'nl-ID', 'nl-url', True, 'tvgids-fetched', True)
        self.channelsource[1] = tvgidstv_HTML(1, 'tvgids.tv', 'tv-ID', 'tv-url', False, 'tvgidstv-fetched', True)
        self.channelsource[2] = rtl_JSON(2, 'rtl.nl', 'rtl-ID', 'rtl-url', True)
        #self.channelsource[3] = teveblad_HTML(3, 'teveblad.be', 'be-ID', 'be-url')
        self.channelsource[4] = npo_HTML(4, 'npo.nl', 'npo-ID', 'npo-url')
        self.channelsource[5] = horizon_JSON(5, 'horizon.tv', 'horizon-ID', 'horizon-url', True)
        self.channelsource[6] = humo_JSON(6, 'humo.be', 'humo-ID', 'humo-url', True)
        self.channelsource[7] = vpro_HTML(7, 'vpro.nl', 'vpro-ID', 'vpro-url')
        self.channelsource[8] = nieuwsblad_HTML(8, 'nieuwsblad.be', 'nb-ID', 'nb-url')
        self.channelsource[9] = primo_HTML(9, 'primo.eu', 'primo-ID', 'primo-url', False, 'primo-fetched', True)
        self.channelsource[10] = vrt_JSON(10, 'vrt.be', 'vrt-ID', 'vrt-url', True)
        self.channelsource[11] = Virtual_Channels(11, 'virtual', 'virtual-ID', 'virtual-url')
        self.channelsource[12] = oorboekje_HTML(12, 'oorboekje.nl', 'ob-ID', 'ob-url')
        self.output_lock = Lock()
        self.cache_return = Queue()
        self.ttvdb = theTVDB()
        self.cache_count = 0
        self.fetch_count = 0
        self.fail_count = 0
        self.ttvdb_count = 0
        self.ttvdb_fail_count = 0
        self.program_count = 0

    def xmlescape(self, s):
        """Escape <, > and & characters for use in XML"""
        return saxutils.escape(s)

    def remove_accents(self, name):
        name = re.sub('á','a', name)
        name = re.sub('é','e', name)
        name = re.sub('í','i', name)
        name = re.sub('ó','o', name)
        name = re.sub('ú','u', name)
        name = re.sub('ý','y', name)
        name = re.sub('à','a', name)
        name = re.sub('è','e', name)
        name = re.sub('ì','i', name)
        name = re.sub('ò','o', name)
        name = re.sub('ù','u', name)
        name = re.sub('ä','a', name)
        name = re.sub('ë','e', name)
        name = re.sub('ï','i', name)
        name = re.sub('ö','o', name)
        name = re.sub('ü','u', name)
        name = re.sub('ÿ','y', name)
        name = re.sub('â','a', name)
        name = re.sub('ê','e', name)
        name = re.sub('î','i', name)
        name = re.sub('ô','o', name)
        name = re.sub('û','u', name)
        name = re.sub('ã','a', name)
        name = re.sub('õ','o', name)
        name = re.sub('@','a', name)
        return name

    def format_timezone(self, td, use_utc=False, only_date=False ):
        """
        Given a datetime object, returns a string in XMLTV format
        """
        if use_utc:
            td = td.astimezone(UTC)

        if only_date:
            return td.strftime('%Y%m%d')

        else:
            return td.strftime('%Y%m%d%H%M%S %z')

    def add_starttag(self, tag, ident = 0, attribs = '', text = '', close = False):
        '''
        Add a starttag with optional attributestring, textstring and optionally close it.
        Give it the proper ident.
        '''
        if attribs != '':
            attribs = ' %s' % attribs

        if close and text == '':
            return u'%s<%s%s/>\n' % (''.rjust(ident), self.xmlescape(tag), self.xmlescape(attribs))

        if close and text != '':
            return u'%s<%s%s>%s</%s>\n' % (''.rjust(ident), self.xmlescape(tag), self.xmlescape(attribs), self.xmlescape(text), self.xmlescape(tag))

        else:
            return u'%s<%s%s>%s\n' % (''.rjust(ident), self.xmlescape(tag), self.xmlescape(attribs), self.xmlescape(text))

    def add_endtag(self, tag, ident = 0):
        '''
        Return a proper idented closing tag
        '''
        return u'%s</%s>\n' % (''.rjust(ident), self.xmlescape(tag))

    def create_channel_strings(self, chanid, add_HD = None):
        '''
        Create the strings for the channels we fetched info about
        '''
        if add_HD == True:
            xmltvid = '%s-hd' % config.channels[chanid].xmltvid

        else:
            xmltvid = config.channels[chanid].xmltvid

        self.xml_channels[xmltvid] = []
        self.xml_channels[xmltvid].append(self.add_starttag('channel', 2, 'id="%s%s"' % \
            (xmltvid, config.channels[chanid].get_opt('compat'))))
        self.xml_channels[xmltvid].append(self.add_starttag('display-name', 4, 'lang="nl"', \
            config.channels[chanid].chan_name, True))
        if config.channels[chanid].get_opt('logos'):
            if config.channels[chanid].icon_source in range(len(self.logo_provider)):
                lpath = self.logo_provider[config.channels[chanid].icon_source]
                lname = config.channels[chanid].icon
                if config.channels[chanid].icon_source == 5 and lpath[-16:] == 'ChannelLogos/02/':
                    if len(lname) > 16 and  lname[0:16] == 'ChannelLogos/02/':
                        lname = lname[16:].split('?')[0]

                    else:
                        lname = lname.split('?')[0]

                elif config.channels[chanid].icon_source == 5 and lpath[-16:] != 'ChannelLogos/02/':
                    if len(lname) > 16 and  lname[0:16] == 'ChannelLogos/02/':
                        lname = lname.split('?')[0]

                    else:
                        lpath = lpath + 'ChannelLogos/02/'
                        lname = lname.split('?')[0]

                full_logo_url = lpath + lname
                self.xml_channels[xmltvid].append(self.add_starttag('icon', 4, 'src="%s"' % full_logo_url, '', True))

            elif config.channels[chanid].icon_source == 99:
                self.xml_channels[xmltvid].append(self.add_starttag('icon', 4, 'src="%s"' % config.channels[chanid].icon, '', True))

        self.xml_channels[xmltvid].append(self.add_endtag('channel', 2))

    def create_program_string(self, chanid, add_HD = None):
        '''
        Create all the program strings
        '''
        if add_HD == True:
            xmltvid = '%s-hd' % config.channels[chanid].xmltvid

        else:
            xmltvid = config.channels[chanid].xmltvid
            with self.output_lock:
                self.program_count += len(config.channels[chanid].all_programs)

        self.xml_programs[xmltvid] = []
        config.channels[chanid].all_programs.sort(key=lambda program: (program['start-time'],program['stop-time']))
        for program in config.channels[chanid].all_programs[:]:
            xml = []

            # Start/Stop
            attribs = 'start="%s" stop="%s" channel="%s%s"' % \
                (self.format_timezone(program['start-time'], config.opt_dict['use_utc']), \
                self.format_timezone(program['stop-time'], config.opt_dict['use_utc']), \
                xmltvid, config.channels[chanid].get_opt('compat'))

            if 'clumpidx' in program and program['clumpidx'] != '':
                attribs += 'clumpidx="%s"' % program['clumpidx']

            xml.append(self.add_starttag('programme', 2, attribs))

            # Title
            xml.append(self.add_starttag('title', 4, 'lang="nl"', program['name'], True))
            if program['originaltitle'] != '' and program['country'] != '' and program['country'].lower() != 'nl' and program['country'].lower() != 'be':
                xml.append(self.add_starttag('title', 4, 'lang="%s"' % (program['country'].lower()), program['originaltitle'], True))

            # Subtitle
            if 'titel aflevering' in program and program['titel aflevering'] != '':
                xml.append(self.add_starttag('sub-title', 4, 'lang="nl"', program['titel aflevering'] ,True))

            # Add an available subgenre in front off the description or give it as description

            # A prefered description was set and found
            if len(program['prefered description']) > 100:
                program['description'] = program['prefered description']

            desc_line = u''
            if program['subgenre'] != '':
                 desc_line = u'%s: ' % (program['subgenre'])

            if program['omroep'] != ''and re.search('(\([A-Za-z \-]*?\))', program['omroep']):
                desc_line = u'%s%s ' % (desc_line, re.search('(\([A-Za-z \-]*?\))', program['omroep']).group(1))

            if program['description'] != '':
                desc_line = u'%s%s ' % (desc_line, program['description'])

            # Limit the length of the description
            if desc_line != '':
                desc_line = re.sub('\n', ' ', desc_line)
                if len(desc_line) > config.channels[chanid].get_opt('desc_length'):
                    spacepos = desc_line[0:config.channels[chanid].get_opt('desc_length')-3].rfind(' ')
                    desc_line = desc_line[0:spacepos] + '...'

                xml.append(self.add_starttag('desc', 4, 'lang="nl"', desc_line.strip(),True))

            # Process credits section if present.
            # This will generate director/actor/presenter info.
            if program['credits'] != {}:
                start_added = False
                for role in ('director', 'actor', 'writer', 'adapter', 'producer', 'composer', 'editor', 'presenter', 'commentator', 'guest'):
                    if role in program['credits']:
                        for name in program['credits'][role]:
                            if name != '':
                                if not start_added:
                                    xml.append(self.add_starttag('credits', 4))
                                    start_added = True

                                xml.append(self.add_starttag((role), 6, '', self.xmlescape(name),True))

                if start_added:
                    xml.append(self.add_endtag('credits', 4))

            # Original Air-Date
            if isinstance(program['airdate'], datetime.date):
                xml.append(self.add_starttag('date', 4, '',  \
                    self.format_timezone(program['airdate'], config.opt_dict['use_utc'],True), True))

            elif program['jaar van premiere'] != '':
                xml.append(self.add_starttag('date', 4, '', program['jaar van premiere'],True))

            # Genre
            if config.channels[chanid].get_opt('cattrans'):
                cat0 = ('', '')
                cat1 = (program['genre'].lower(), '')
                cat2 = (program['genre'].lower(), program['subgenre'].lower())
                if cat2 in config.cattrans.keys() and config.cattrans[cat2] != '':
                    cat = config.cattrans[cat2].capitalize()

                elif cat1 in config.cattrans.keys() and config.cattrans[cat1] != '':
                    cat = config.cattrans[cat1].capitalize()

                elif cat0 in config.cattrans.keys() and config.cattrans[cat0] != '':
                   cat = config.cattrans[cat0].capitalize()

                else:
                    cat = 'Unknown'

                xml.append(self.add_starttag('category', 4 , '', cat, True))

            else:
                cat = program['genre']
                if program['genre'] != '':
                    xml.append(self.add_starttag('category', 4, 'lang="nl"', program['genre'], True))

                else:
                    xml.append(self.add_starttag('category', 4 , '', 'Overige', True))

            # An available url
            if program['infourl'] != '':
                xml.append(self.add_starttag('url', 4, '', program['infourl'],True))

            if program['country'] != '':
                xml.append(self.add_starttag('country', 4, '', program['country'],True))

            # Only add season/episode if relevant. i.e. Season can be 0 if it is a pilot season, but episode never.
            # Also exclude Sports for MythTV will make it into a Series
            if cat.lower() != 'sports' and cat.lower() != 'sport':
                if program['season'] != 0 and program['episode'] != 0:
                    if program['season'] == 0:
                        text = ' . %d . '  % (int(program['episode']) - 1)

                    else:
                        text = '%d . %d . '  % (int(program['season']) - 1, int(program['episode']) - 1)

                    xml.append(self.add_starttag('episode-num', 4, 'system="xmltv_ns"', text,True))

            # Process video/audio/teletext sections if present
            if (program['video']['breedbeeld'] or program['video']['blackwhite'] \
              or (config.channels[chanid].get_opt('mark_hd') \
              or add_HD == True) and (program['video']['HD'])):
                xml.append(self.add_starttag('video', 4))

                if program['video']['breedbeeld']:
                    xml.append(self.add_starttag('aspect', 6, '', '16:9',True))

                if program['video']['blackwhite']:
                    xml.append(self.add_starttag('colour', 6, '', 'no',True))

                if (config.channels[chanid].get_opt('mark_hd') \
                  or add_HD == True) and (program['video']['HD']):
                    xml.append(self.add_starttag('quality', 6, '', 'HDTV',True))

                xml.append(self.add_endtag('video', 4))

            if program['audio'] != '':
                xml.append(self.add_starttag('audio', 4))
                xml.append(self.add_starttag('stereo', 6, '',program['audio'] ,True))
                xml.append(self.add_endtag('audio', 4))

            # It's been shown before
            if program['rerun']:
                xml.append(self.add_starttag('previously-shown', 4, '', '',True))

            # It's a first
            if program['premiere']:
                xml.append(self.add_starttag('premiere', 4, '', '',True))

            # It's the last showing
            if program['last-chance']:
                xml.append(self.add_starttag('last-chance', 4, '', '',True))

            # It's new
            if program['new']:
                xml.append(self.add_starttag('new', 4, '', '',True))

            # There are teletext subtitles
            if program['teletekst']:
                xml.append(self.add_starttag('subtitles', 4, 'type="teletext"', '',True))

            # Add any Kijkwijzer items
            if config.opt_dict['kijkwijzerstijl'] in ('long', 'short', 'single'):
                kstring = ''
                # First only one age limit from high to low
                for k in ('4', '3', '9', '2', '1'):
                    if k in program['kijkwijzer']:
                        if config.opt_dict['kijkwijzerstijl'] == 'single':
                            kstring += (config.kijkwijzer[k]['code'] + ': ')

                        else:
                            xml.append(self.add_starttag('rating', 4, 'system="kijkwijzer"'))
                            if config.opt_dict['kijkwijzerstijl'] == 'long':
                                xml.append(self.add_starttag('value', 6, '', config.kijkwijzer[k]['text'], True))

                            else:
                                xml.append(self.add_starttag('value', 6, '', config.kijkwijzer[k]['code'], True))

                            xml.append(self.add_starttag('icon', 6, 'src="%s"' % config.kijkwijzer[k]['icon'], '', True))
                            xml.append(self.add_endtag('rating', 4))
                        break

                # And only one of any of the others
                for k in ('g', 'a', 's', 't', 'h', 'd'):
                    if k in program['kijkwijzer']:
                        if config.opt_dict['kijkwijzerstijl'] == 'single':
                            kstring += k.upper()

                        else:
                            xml.append(self.add_starttag('rating', 4, 'system="kijkwijzer"'))
                            if config.opt_dict['kijkwijzerstijl'] == 'long':
                                xml.append(self.add_starttag('value', 6, '', config.kijkwijzer[k]['text'], True))

                            else:
                                xml.append(self.add_starttag('value', 6, '', config.kijkwijzer[k]['code'], True))

                            xml.append(self.add_starttag('icon', 6, 'src="%s"' % config.kijkwijzer[k]['icon'], '', True))
                            xml.append(self.add_endtag('rating', 4))

                if config.opt_dict['kijkwijzerstijl'] == 'single' and kstring != '':
                    xml.append(self.add_starttag('rating', 4, 'system="kijkwijzer"'))
                    xml.append(self.add_starttag('value', 6, '', kstring, True))
                    xml.append(self.add_endtag('rating', 4))

            # Set star-rating if applicable
            if program['star-rating'] != '':
                xml.append(self.add_starttag('star-rating', 4))
                xml.append(self.add_starttag('value', 6, '',('%s/10' % (program['star-rating'])).strip(),True))
                xml.append(self.add_endtag('star-rating', 4))

            xml.append(self.add_endtag('programme', 2))
            self.xml_programs[xmltvid].append(xml)

    def get_xmlstring(self):
        '''
        Compound the compleet XML output and return it
        '''
        if config.output == None:
            startstring =[u'<?xml version="1.0" encoding="%s"?>\n' % logging.local_encoding]

        else:
            startstring =[u'<?xml version="1.0" encoding="%s"?>\n' % self.xmlencoding]

        startstring.append(u'<!DOCTYPE tv SYSTEM "xmltv.dtd">\n')
        startstring.append(u'<tv generator-info-name="%s" generator-info-url="https://github.com/tvgrabbers/tvgrabnlpy">\n' % config.version(True))
        closestring = u'</tv>\n'

        xml = []
        xml.append(u"".join(startstring))

        for channel in config.channels.values():
            if channel.active and channel.xmltvid in self.xml_channels:
                xml.append(u"".join(self.xml_channels[channel.xmltvid]))
                if channel.get_opt('add_hd_id') and '%s-hd' % (channel.xmltvid) in self.xml_channels:
                    xml.append(u"".join(self.xml_channels['%s-hd' % channel.xmltvid]))

        for channel in config.channels.values():
            if channel.active and channel.xmltvid in self.xml_programs:
                for program in self.xml_programs[channel.xmltvid]:
                    xml.append(u"".join(program))

                if channel.get_opt('add_hd_id') and '%s-hd' % (channel.xmltvid) in self.xml_channels:
                    for program in self.xml_programs['%s-hd' % channel.xmltvid]:
                        xml.append(u"".join(program))

        xml.append(closestring)

        return u"".join(xml)

    def print_string(self):
        '''
        Print the compleet XML string to stdout or selected file
        '''
        xml = xml_output.get_xmlstring()

        if xml != None:
            if config.output == None:
                sys.stdout.write(xml.encode(logging.local_encoding, 'replace'))

            else:
                config.output.write(xml)

            if config.write_info_files:
                infofiles.write_xmloutput(xml)

# end XMLoutput
xml_output = XMLoutput()

def main():
    # We want to handle unexpected errors nicely. With a message to the log
    try:
        # Get the options, channels and other configuration
        start_time = datetime.datetime.now()
        x = config.validate_commandline()
        if x != None:
            return(x)

        log("The Netherlands: %s\n" % config.version(True), 1, 1)
        log('Start time of this run: %s\n' % (start_time.strftime('%Y-%m-%d %H:%M')),4, 1)

        # Start the seperate fetching threads
        for source in xml_output.channelsource.values():
            x = source.start()
            if x != None:
                return(x)

        # Start the Channel threads, but wait a second so the sources have properly initialized any child channel
        time.sleep(1)
        counter = 0
        channel_threads = []
        for channel in config.channels.values():
            if not (channel.active or channel.is_child):
                continue

            counter += 1
            channel.counter = counter
            x = channel.start()
            if x != None:
                return(x)

            channel_threads.append(channel)

        # Synchronize
        for index in (0, 1):
            xml_output.channelsource[index].join()

        for channel in channel_threads:
            if channel.is_alive():
                channel.join()

        # produce the results and wrap-up
        config.write_defaults_list()
        xml_output.print_string()

        # Report duration
        end_time = datetime.datetime.now()
        duration = end_time - start_time

        log_array = ['\n', 'Execution complete.\n', '\n']
        log_array.append('Fetch statistics for %s programms on %s channels:\n' % (xml_output.program_count, config.chan_count))
        log_array.append(' Start time: %s\n'% (start_time.strftime('%Y-%m-%d %H:%M')))
        log_array.append('   End time: %s\n' % (end_time.strftime('%Y-%m-%d %H:%M')))
        log_array.append('   Duration: %s\n' % (duration))
        log_array.append( '%6.0f page(s) fetched, of which %s failed\n' % (xml_output.fetch_count, xml_output.fail_count))
        log_array.append( '%6.0f cache hits\n' % (xml_output.cache_count))
        log_array.append( '%6.0f succesful ttvdb.com lookups\n' % (xml_output.ttvdb_count))
        log_array.append( '%6.0f    failed ttvdb.com lookups\n' % (xml_output.ttvdb_fail_count))
        if xml_output.fetch_count > 0:
            log_array.extend([' Time/fetch: %s seconds\n' % (duration.total_seconds()/xml_output.fetch_count), '\n'])
        log_array.append('%6.0f page(s) fetched from theTVDB.com\n' % (xml_output.ttvdb.fetch_count))
        log_array.extend(['%6.0f failure(s) on theTVDB.com\n' % (xml_output.ttvdb.fail_count), '\n'])
        for source in xml_output.channelsource.values():
            log_array.append('%6.0f   base page(s) fetched from %s\n' % (source.base_count, source.source))
            if source.detail_processor:
                log_array.append('%6.0f detail page(s) fetched from %s\n' % (source.detail_count, source.source))

            log_array.extend(['%6.0f failure(s) on %s\n' % (source.fail_count, source.source), '\n'])

        log(log_array, 4, 3)

    except:
        log(['\n', 'An unexpected error has occured:\n', traceback.format_exc(), \
            '\n', 'If you want assistence, please attach your configuration and log files!\n', \
            '     %s\n' % (config.config_file), '     %s\n' % (config.log_file)],0)

        return(99)

    # and return success
    return(0)
# end main()

# allow this to be a module
if __name__ == '__main__':
    x = main()
    config.close()
    sys.exit(x)