====== Overview ======
{{tag>project software}}
========= Overview ==========
Genericore is a generic message queueing interface to provide an easily extensible stats processing infrastructure.

It is modelled after the EVA principle (Eingabe Verarbeitung Ausgabe). The system roughly contains three kinds of modules :

  * Input Modules (like E-MAIL gathering and simple parsing)
  * Processing Modules (like calculating the number of mails a user on the mailing list sends)
  * Output Modules (like a webserver which shows the calculated values)
The process we use is publish-subscribe (see below)


========= Message Queue ============
The message queue is an [[http://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol|AMQP]]-based [[https://www.rabbitmq.com/|RabbitMQ]] which runs on:

  * ''188.40.39.66'',
  * default port,
  * defaul username and password (if you don't know it then simply ask someone at shackspace).

You might want to read [[https://www.rabbitmq.com/getstarted.html|Getting started with RabbitMQ]] (which is pretty good). Especially see [[https://www.rabbitmq.com/tutorial-three-python.html|Publish/Subscribe]].

Two things are important:

  - Queues: FIFO Pipelines which hold message
  - Exchanges, which define how messages are emitted to queues

===== Default Values =====
Unless explicitly defined by the modules, the following default values are used:

1. Exchanges:
<code>
type = 'fanout'
passive = False
durable = False
auto_delete = False
internal = False
nowait = False
</code>

2. Queue:
<code>
name = ''
passive = False
durable = False
exclusive = True
auto_delete = True
nowait = False
</code>

====== Configure Rabbitmq ==========
This is how the current rabbitmq server is configured to grant permission for user shack and do expose its interface to the network.

Download and install rabbitmq (there are plenty tutorials on the web)

In /etc/rabbitmq/rabbitmq.conf
<code>
NODENAME=rabbit                                                                    
NODE_IP_ADDRESS=YOUR.IP
NODE_PORT=5672

LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia
</code>

start the rabbitmq instance (e.g. /etc/rc.d/rabbitmq start), then change permissions 
<code>
sudo rabbitmqctl delete_user guest
sudo rabbitmqctl add_user shack your_shack_password
sudo rabbitmqctl set_permissions shack ".*" ".*" ".*"
</code>

=========== Current Modules ========= 
Currently the following modules are available:

===== Input Modules ======

==== genericore-ircspy ====
  * Description: Read IRC PRIVMSGs and publish JSON-representations.
  * Repository: [[https://github.com/shackspace/genericore-ircspy]]
  * Maintainer: [[leute:tv]]
  * Output Exchange: ircspy
  * Output Protocol: [[https://github.com/shackspace/genericore/blob/master/protocols/ircspy|see examples]]

==== genericore-mailsrc ====
  * Description: Receive mails via raw TCP and publish JSON-representations.
  * Repository: [[https://github.com/shackspace/genericore-mailsrc]]
  * Maintainer: [[leute:tv]]
  * Output Exchange: mailsrc
  * Output Protocol: [[https://github.com/shackspace/genericore/blob/master/protocols/mailsrc/mailsrc.json%2Btype|mailsrc.json+type]]

==== genericore-rabbithole ====
  * Description : Module which periodically sends Backend Informations of the AMQP Server for Maintenance and Debugging purpose.
  * using : rabbitmqctl, python-genericore
  * Repository: [[https://github.com/shackspace/genericore-rabbithole]]
  * Maintainer : [[leute:felix]]
  * Output Exchange : rabbithole

==== genericore-snmp_users ====
  * Description: Polls an SNMP-Router for it's ARP Table
  * Repository: [[https://github.com/shackspace/genericore-snmp_users]]
  * Maintainer: [[leute:felix]]
  * Output Exchange: snmp_users
  * Output Protocol:
<code>{ "type": "snmp", "subtype":0, "data": { TODO }}</code>

===== Processing Modules ======

==== genericore-mail_proc ====
  * Description: Processes mail to generate statistics from new mails by saving and loading stats locally. Especially for given Users at the Mailing List
  * Stats generated:
    * Individual :
      * Number of Posts
      * First Mail date
      * Last Mail date
  * Repository: [[https://github.com/shackspace/genericore-mail_proc]]
  * Maintainer: [[leute:felix]]
  * Backend: mongodb
  * Input Exchange: mailsrc
  * Output Exchange: mail_proc
  * Output Protocol: [[https://github.com/shackspace/genericore/tree/master/protocols/mail_proc|see examples]]

==== genericore-mail_stats ====

  * Description: Generates Statistics of the Mailing list
  * Stats generated:
    * Absolute Time:
      * Number of mails in the last month,week,day,hour (probably year?)
  * Repository: [[https://github.com/shackspace/genericore-mail_stats]]
  * Maintainer: [[leute:felix]]
  * Backend: mongodb
  * Input Exchange: mailsrc
  * Output Exchange: mail_stats
  * Output Protocol: [[https://github.com/shackspace/genericore/tree/master/protocols/mail_stats|see examples]]

====== Output Modules ======

==== genericore-bunghole ====
  * Description: Web based output system with plugin system for different processing modules.
  * Repository: [[https://github.com/shackspace/genericore-bunghole]]
  * Maintainer: [[leute:svenpfleiderer]]
  * Backend: node-dirty, express.js
  * Input Exchange: mail_proc
  * Input Protocol:
<code> { "type": "log", "subtype":0, "data": { TODO } } </code>

==== genericore-shiny =====
  * Description: (Currently not so) Shiny Web frontend for genericore (not yet plugable but functional prototype)
  * Repository: [[https://github.com/shackspace/genericore-shiny]]
  * Maintainer: [[leute:felix]]
  * Backend: cherrypy
  * Input Exchange: snmp_src,mail_stats,irc_something
  * Input Protocol
<code> Whatever the processing modules cough out </code>

==== genericore-irclog ====
  * Description: Forward Genericore modules' logs to IRC.
  * Repository: [[https://github.com/shackspace/genericore-irclog]]
  * Maintainer: [[leute:tv]]
  * Input Exchange: log

=========== Libraries =========
===== Javascript libraries ======
==== node-genericore ====
  * Repository: [[https://github.com/shackspace/node-genericore]]
  * Maintainer: [[leute:tv]]
===== Python libraries  ======
==== python-genericore =====
  * Repository: [[https://github.com/shackspace/python-genericore]]
  * Maintainer: [[leute:felix]]
  * Version: <code brainfuck>+++++++[>++++++++>+++++++>+++++++<<<-]>--.>---.>.</code>

=========== TODO Modules =========
describe the modules "todo" in this section:
===== Input Modules ======
==== genericore-shackopen ====
  * Description : a simple source module which publishes the "Open"-Status of the Hackerspace (see wiki-status bar)



===== Processing Modules ======
==== genericore-mail_live_stats ====
  * Description: Generates live stats every N minutes from given input
  * Repository: [[https://github.com/shackspace/genericore-mail_live_stats]]
  * Maintainer: [[leute:felix]]
  * Stats generated:
    * Mails in the Hour,Day,Week
    * Most Active User in Hour,Day,Week
    * Biggest Thread in Hour,Day,Week
    * Biggest Mail in ...
    * ... EXTEND ME
  * Backend: mongodb
  * Input Exchange: mail_proc
  * Output Exchange: mail_live_stats (,log)
  * Output Protocol:
<code> { "type": "mail", "subtype":1, "data": { TODO } } </code>
===== Output Modules ======
==== genericore-milkshake ====
* Description: Gateway between AMQP and Websockets
* Backend: Tornado,python-genericore
* Maintainer: [[leute:felix]]