Genericore is a generic message queueing interface to provide an easily extensible stats processing infrastructure.

It is modelled after the EVA principle (Eingabe Verarbeitung Ausgabe). The system roughly contains three kinds of modules :

• Input Modules (like E-MAIL gathering and simple parsing)
• Processing Modules (like calculating the number of mails a user on the mailing list sends)
• Output Modules (like a webserver which shows the calculated values)

The process we use is publish-subscribe (see below)

The message queue is an AMQP-based RabbitMQ which runs on:

• 188.40.39.66,
• default port,
• defaul username and password (if you don't know it then simply ask someone at shackspace).

You might want to read Getting started with RabbitMQ (which is pretty good). Especially see Publish/Subscribe.

Two things are important:

1. Queues: FIFO Pipelines which hold message
2. Exchanges, which define how messages are emitted to queues

Unless explicitly defined by the modules, the following default values are used:

1. Exchanges:

type = 'fanout'
passive = False
durable = False
auto_delete = False
internal = False
nowait = False

2. Queue:

name = ''
passive = False
durable = False
exclusive = True
auto_delete = True
nowait = False

This is how the current rabbitmq server is configured to grant permission for user shack and do expose its interface to the network.

Download and install rabbitmq (there are plenty tutorials on the web)

In /etc/rabbitmq/rabbitmq.conf

NODENAME=rabbit                                                                    
NODE_IP_ADDRESS=YOUR.IP
NODE_PORT=5672

LOG_BASE=/var/log/rabbitmq
MNESIA_BASE=/var/lib/rabbitmq/mnesia

start the rabbitmq instance (e.g. /etc/rc.d/rabbitmq start), then change permissions

sudo rabbitmqctl delete_user guest
sudo rabbitmqctl add_user shack your_shack_password
sudo rabbitmqctl set_permissions shack ".*" ".*" ".*"

Currently the following modules are available:

• Description: Read IRC PRIVMSGs and publish JSON-representations.
• Repository: https://github.com/shackspace/genericore-ircspy
• Maintainer: tv
• Output Exchange: ircspy
• Output Protocol: see examples

• Description: Receive mails via raw TCP and publish JSON-representations.
• Repository: https://github.com/shackspace/genericore-mailsrc
• Maintainer: tv
• Output Exchange: mailsrc
• Output Protocol: mailsrc.json+type

• Description : Module which periodically sends Backend Informations of the AMQP Server for Maintenance and Debugging purpose.
• using : rabbitmqctl, python-genericore
• Repository: https://github.com/shackspace/genericore-rabbithole
• Maintainer : felix
• Output Exchange : rabbithole

• Description: Polls an SNMP-Router for it's ARP Table
• Repository: https://github.com/shackspace/genericore-snmp_users
• Maintainer: felix
• Output Exchange: snmp_users
• Output Protocol:

{ "type": "snmp", "subtype":0, "data": { TODO }}

• Description: Processes mail to generate statistics from new mails by saving and loading stats locally. Especially for given Users at the Mailing List
• Stats generated:
  • Individual :
    • Number of Posts
    • First Mail date
    • Last Mail date
• Repository: https://github.com/shackspace/genericore-mail_proc
• Maintainer: felix
• Backend: mongodb
• Input Exchange: mailsrc
• Output Exchange: mail_proc
• Output Protocol: see examples

• Description: Generates Statistics of the Mailing list
• Stats generated:
  • Absolute Time:
    • Number of mails in the last month,week,day,hour (probably year?)
• Repository: https://github.com/shackspace/genericore-mail_stats
• Maintainer: felix
• Backend: mongodb
• Input Exchange: mailsrc
• Output Exchange: mail_stats
• Output Protocol: see examples

• Description: Web based output system with plugin system for different processing modules.
• Repository: https://github.com/shackspace/genericore-bunghole
• Maintainer: svenpfleiderer
• Backend: node-dirty, express.js
• Input Exchange: mail_proc
• Input Protocol:

 { "type": "log", "subtype":0, "data": { TODO } } 

• Description: (Currently not so) Shiny Web frontend for genericore (not yet plugable but functional prototype)
• Repository: https://github.com/shackspace/genericore-shiny
• Maintainer: felix
• Backend: cherrypy
• Input Exchange: snmp_src,mail_stats,irc_something
• Input Protocol

 Whatever the processing modules cough out 

• Description: Forward Genericore modules' logs to IRC.
• Repository: https://github.com/shackspace/genericore-irclog
• Maintainer: tv
• Input Exchange: log

• Repository: https://github.com/shackspace/node-genericore
• Maintainer: tv

• Repository: https://github.com/shackspace/python-genericore
• Maintainer: felix
• Version:

  +++++++[>++++++++>+++++++>+++++++<<<-]>--.>---.>.

describe the modules „todo“ in this section:

• Description : a simple source module which publishes the „Open“-Status of the Hackerspace (see wiki-status bar)

• Description: Generates live stats every N minutes from given input
• Repository: https://github.com/shackspace/genericore-mail_live_stats
• Maintainer: felix
• Stats generated:
  • Mails in the Hour,Day,Week
  • Most Active User in Hour,Day,Week
  • Biggest Thread in Hour,Day,Week
  • Biggest Mail in …
  • … EXTEND ME
• Backend: mongodb
• Input Exchange: mail_proc
• Output Exchange: mail_live_stats (,log)
• Output Protocol:

 { "type": "mail", "subtype":1, "data": { TODO } } 

* Description: Gateway between AMQP and Websockets * Backend: Tornado,python-genericore * Maintainer: felix