r7292 - in dumbhippo/trunk: firehose/firehose/jobs super super/firehose super/firehose/files super/firehose/files/conf super/firehose/files/scripts

[commits mugshot org]

Author: walters
Date: 2008-02-06 17:46:36 -0600 (Wed, 06 Feb 2008)
New Revision: 7292

Added:
   dumbhippo/trunk/firehose/firehose/jobs/__init__.py
   dumbhippo/trunk/super/firehose/
   dumbhippo/trunk/super/firehose/files/
   dumbhippo/trunk/super/firehose/files/conf/
   dumbhippo/trunk/super/firehose/files/conf/mugshot.cfg
   dumbhippo/trunk/super/firehose/files/data/
   dumbhippo/trunk/super/firehose/files/lib/
   dumbhippo/trunk/super/firehose/files/logs/
   dumbhippo/trunk/super/firehose/files/run/
   dumbhippo/trunk/super/firehose/files/scripts/
   dumbhippo/trunk/super/firehose/files/scripts/firehose-running.sh
   dumbhippo/trunk/super/firehose/files/scripts/firehose-start.sh
   dumbhippo/trunk/super/firehose/files/scripts/firehose-stop.sh
Modified:
   dumbhippo/trunk/firehose/firehose/jobs/master.py
   dumbhippo/trunk/firehose/firehose/jobs/poller.py
   dumbhippo/trunk/super/base.conf
Log:
Super-ify firehose.


Added: dumbhippo/trunk/firehose/firehose/jobs/__init__.py
===================================================================

Modified: dumbhippo/trunk/firehose/firehose/jobs/master.py
===================================================================
--- dumbhippo/trunk/firehose/firehose/jobs/master.py	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/firehose/firehose/jobs/master.py	2008-02-06 23:46:36 UTC (rev 7292)
@@ -75,7 +75,7 @@
         self.__client_url = config.get('firehose.clienturl')
         
         # Default to one slave on localhost
-        self.__worker_endpoints = ['localhost:%d' % (int(config.get('firehose.slaveport')),)]
+        self.__worker_endpoints = ['localhost:%d' % (int(config.get('firehose.localslaveport')),)]
         _logger.debug("worker endpoints are %r", self.__worker_endpoints)
         for bind in self.__worker_endpoints:
             (host,port) = bind.split(':')
@@ -156,6 +156,7 @@
     
     @log_except(_logger)
     def __push_changed(self):
+        extkey = config.get('firehose.externalServiceKey')
         try:
             self.__task_lock.acquire()
             self.__changed_thread_queued = False
@@ -166,7 +167,9 @@
         jsonstr = simplejson.dumps(changed)
         parsed = urlparse.urlparse(self.__client_url)
         conn = httplib.HTTPConnection(parsed.hostname, parsed.port)
-        conn.request('POST', parsed.path or '/', jsonstr)
+        path = parsed.path or '/'
+        path += '?esk=' + extkey
+        conn.request('POST', path, jsonstr)
         conn.close()        
 
     def __append_changed(self, changed):

Modified: dumbhippo/trunk/firehose/firehose/jobs/poller.py
===================================================================
--- dumbhippo/trunk/firehose/firehose/jobs/poller.py	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/firehose/firehose/jobs/poller.py	2008-02-06 23:46:36 UTC (rev 7292)
@@ -90,7 +90,7 @@
         return _instance
         
     def __init__(self):
-        bindport = int(config.get('firehose.slaveport'))
+        bindport = int(config.get('firehose.localslaveport'))
         self.__server = BaseHTTPServer.HTTPServer(('', bindport), TaskRequestHandler)
         self.__active_collectors = set()
         self.__master_hostport = config.get('firehose.masterhost')

Modified: dumbhippo/trunk/super/base.conf
===================================================================
--- dumbhippo/trunk/super/base.conf	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/super/base.conf	2008-02-06 23:46:36 UTC (rev 7292)
@@ -484,7 +484,31 @@
 		<parameter name="stopCommand">$targetdir/scripts/imbot-stop.sh</parameter>
 		<parameter name="statusCommand">$targetdir/scripts/imbot-running.sh</parameter>
 	</service>
+	
+    <service name="firehose">
+        <parameter name="targetdir">$basedir/firehose</parameter>
+        <parameter name="firehosedir">$svndir/firehose</parameter>
 
+        
+
+        <merge src="$superdir/firehose/files"
+               expand="yes"/>
+        <merge src="$firehosedir/start-firehose.py"
+               dest="start-firehose.py"/>
+        <merge src="$firehosedir/firehose"
+               dest="firehose"/>
+
+        <targetAttributes pattern="/logs/*"  ignore="yes" preserve="yes"/>
+        <targetAttributes pattern="/run/*"   ignore="yes"/>
+        <targetAttributes pattern="/data/*"  ignore="yes" preserve="yes"/>        
+        
+        <parameter name="firehoseMasterPort">$((baseport+80))</parameter>
+        <parameter name="firehoseLocalSlavePort">$((baseport+81))</parameter>         
+
+        <parameter name="startCommand">$targetdir/scripts/firehose-start.sh</parameter>
+        <parameter name="stopCommand">$targetdir/scripts/firehose-stop.sh</parameter>
+    </service>	
+
 	<service name="hungry">
 		<parameter name="targetdir">$basedir/hungry</parameter>
 

Added: dumbhippo/trunk/super/firehose/files/conf/mugshot.cfg
===================================================================
--- dumbhippo/trunk/super/firehose/files/conf/mugshot.cfg	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/super/firehose/files/conf/mugshot.cfg	2008-02-06 23:46:36 UTC (rev 7292)
@@ -0,0 +1,72 @@
+[global]
+# This is where all of your settings go for your development environment
+# Settings that are the same for both development and production
+# (such as template engine, encodings, etc.) all go in 
+# firehose/config/app.cfg
+
+# DATABASE
+
+# pick the form for your database
+# sqlobject.dburi="postgres://username hostname/databasename"
+# sqlobject.dburi="mysql://username:password hostname:port/databasename"
+# sqlobject.dburi="sqlite:///file_name_and_path"
+
+firehose.taskdbpath="%(current_dir_uri)s/data/dev-tasks.sqlite"
+firehose.masterhost="@@serverHost@@:@@firehoseMasterPort@@"
+firehose.localslaveport="@@firehoseLocalSlavePort@@"
+firehose.clienturl="http://@@serverHost@:@@tomcatHttpPort@@/extservice/notify-polling-tasks";
+firehose.externalServiceKey="@@externalServiceKey@@"
+
+# if you are using a database or table type without transactions
+# (MySQL default, for example), you should turn off transactions
+# by prepending notrans_ on the uri
+# sqlobject.dburi="notrans_mysql://username:password hostname:port/databasename"
+
+# for Windows users, sqlite URIs look like:
+# sqlobject.dburi="sqlite:///drive_letter:/path/to/file"
+
+# SERVER
+
+# Some server parameters that you may want to tweak
+server.socket_port=@@firehoseMasterPort@@
+
+# Enable the debug output at the end on pages.
+# log_debug_info_filter.on = False
+
+# Auto-Reload after code modification
+autoreload.on = False
+
+# Set to True if you'd like to abort execution if a controller gets an
+# unexpected parameter. False by default
+tg.strict_parameters = True
+
+tg.scheduler = True
+
+# LOGGING
+# Logging configuration generally follows the style of the standard
+# Python logging module configuration. Note that when specifying
+# log format messages, you need to use *() for formatting variables.
+# Deployment independent log configuration is in firehose/config/log.cfg
+[logging]
+
+[[handlers]]
+
+[[[server_log]]]
+# set the filename as the first argument below
+args="('logs/server.log',)"
+class='FileHandler'
+level='DEBUG'
+formatter='full_content'
+
+[[loggers]]
+[[[firehose]]]
+level='DEBUG'
+qualname='firehose'
+handlers=['server_log']
+propagate=0
+
+[[[access]]]
+level='DEBUG'
+qualname='turbogears.access'
+handlers=['server_log']
+propagate=0

Added: dumbhippo/trunk/super/firehose/files/scripts/firehose-running.sh
===================================================================
--- dumbhippo/trunk/super/firehose/files/scripts/firehose-running.sh	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/super/firehose/files/scripts/firehose-running.sh	2008-02-06 23:46:36 UTC (rev 7292)
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+set -e
+
+targetdir=@@targetdir@@
+
+pid=`cat $targetdir/run/imbot.pid`
+
+if ps -p $pid > /dev/null ; then
+	exit 0;
+else
+	exit 1;
+fi


Property changes on: dumbhippo/trunk/super/firehose/files/scripts/firehose-running.sh
___________________________________________________________________
Name: svn:executable
   + *

Added: dumbhippo/trunk/super/firehose/files/scripts/firehose-start.sh
===================================================================
--- dumbhippo/trunk/super/firehose/files/scripts/firehose-start.sh	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/super/firehose/files/scripts/firehose-start.sh	2008-02-06 23:46:36 UTC (rev 7292)
@@ -0,0 +1,22 @@
+#!/bin/sh
+
+targetdir=@@targetdir@@
+
+echo "Starting Firehose"
+
+cd $targetdir
+
+./start-firehose.py ./conf/mugshot.cfg &
+
+pid=$!
+
+# catch an immediate failure if any
+sleep 2
+if ps -p $pid > /dev/null ; then : ; else
+	echo "... failed"
+	exit 1
+fi
+
+echo "$pid" > $targetdir/run/firehose.pid
+echo "...sucessfully started"
+exit 0


Property changes on: dumbhippo/trunk/super/firehose/files/scripts/firehose-start.sh
___________________________________________________________________
Name: svn:executable
   + *

Added: dumbhippo/trunk/super/firehose/files/scripts/firehose-stop.sh
===================================================================
--- dumbhippo/trunk/super/firehose/files/scripts/firehose-stop.sh	2008-02-06 18:11:20 UTC (rev 7291)
+++ dumbhippo/trunk/super/firehose/files/scripts/firehose-stop.sh	2008-02-06 23:46:36 UTC (rev 7292)
@@ -0,0 +1,34 @@
+#!/bin/sh
+
+targetdir=@@targetdir@@
+
+pidfile=$targetdir/run/firehose.pid
+
+echo "Stopping firehose..."
+
+if [ \! -f $pidfile ] ; then 
+    echo "... not running"
+    exit 0
+fi
+
+pid=`cat $pidfile`
+
+if kill $pid ; then : ; else
+    echo "... not running"
+    exit 0
+fi
+
+timeout=30
+interval=1
+while [ $timeout -gt 0 ] ; do
+    if ps -p $pid > /dev/null ; then : ; else
+	echo "... stopped"
+	exit 0
+    fi
+
+    sleep $interval
+    let timeout="$timeout - $interval"
+done
+
+echo "...timed out"
+exit 1


Property changes on: dumbhippo/trunk/super/firehose/files/scripts/firehose-stop.sh
___________________________________________________________________
Name: svn:executable
   + *