High Scalable & Available part 3: Designing DB 2 MQ Connectors

We already have our MQ server running, now lets install our first DB Cluster by following my tutorials in setting up MySQL Cluster, starts from here.

Assumed that we successfully installed our DB Cluster, before we start writing codes of our DB 2 MQ Connectors, lets go back to the requirements, then move to the connectors design.

A. Requirements:

1. I want to distribute and spread all users’ session into many sharded DB servers equally, meaning: I want the first user session that logged-in into my Application to be persisted in my 1st DB Cluster, the next logged-in user will goes to the 2nd DB Cluster and so on. When the last of my DB Cluster instance reached, the next logged-in user will goes back to the 1st DB Cluster, in other words: I want to spread the load equally to all my DB Cluster instances by doing a round robin,
2. I want my Application to be able to selects, updates and deletes a user’s session, without any knowledge: in which DB Cluster instance that user is registered.

By using features of the RabbitMQ, the above requirements are easy to be done, so before we move on, there are some prerequisites knowledge we must acquired regarding:

1. Work queues message routing in RabbitMQ,
2. Publish/subscribe message routing in RabbitMQ, and
3. Topic message routing in (again) RabbitMQ

Please do learn all those features listed above, in order to understand the codes we are going to do in the next steps.

B. Connector Design for Handling “INSERT” Data

Our first DB 2 MQ Connector is for handling data insertion. For data insertion, we are going to use the same approached with the Work queues message routing example. This way, our requirement to equally spreads the load to all available DB Cluster instances can be achieved. This concept can be illustrated as below pic.

C. Connector Design for Handling “SELECT, UPDATE and DELETE” Data

Our second DB 2 MQ Connector is for handling data selection, updates and deletion. For doing these db operations, we are going to use the same approached  with the Publish/subscribe message routing example, with FANOUT mode. This way, our requirement on hiding where the required data is resides from my Application can be achieved. This concept can be illustrated as below pic.

D. Connector Design for Returning “SELECT” Result

Our third or last DB 2 MQ Connector is for returning a result set on data selection operation. For doing this, we are going to use the same approached with the Topic message routing example. This way, each of our DB Cluster instance can return the selection result to the right Application instance. This concept can be illustrated as below pic.

Ok that’s it guys… next, I’m going to introduce how to use both Java AMQP client library and ClusterJ, for developing our DB 2 MQ Connectors, as I designed above.

High Scalable & Available part 2: Installing RabbitMQ

I discussed the architecture which I’m going to build already, now, we are going to setup our Message Queuing  service.

Important:

1. OS which I’m using for this MQ node is: Ubuntu Server v.12.04,
2. MQ which I’m using is: RabbitMQ v.2.8.2 for debian, and
3. Some prerequisite knowledge on installing MySQL Cluster, which you can read in my previous tutorials.

A. Preparations:

1. Creates a new VM of Ubuntu Server
2. Disabling IP Filtering in this Ubuntu (read my post here)

B. Steps:

1. Login as root
2. Download RabbitMQ binary distribution for debian package

cd /var/tmp/
wget www.rabbitmq.com/releases/rabbitmq-server/v2.8.4/rabbitmq-server_2.8.4-1_all.deb

3. Install erlang-nox package

apt-get install erlang-nox

4. Installing RabbitMQ

cd /var/tmp/
dpkg -i rabbitmq-server_2.8.2-1_all.deb

If everything ok, you can see something like below pic, at your server console.

5. Installing rabbitmq web management console

cd /usr/lib/rabbitmq/bin
rabbitmq-plugins enable rabbitmq_management
rabbitmq-server restart

If everything ok, you can see something like below pic, at your server console.

6. To test our installation, just open our browser, and type our MQ server web admin console address.

7. Login using default RabbitMQ username & password, then you can see something like below pic.

That’s it guys, now we already have our message queuing server running, and ready for use.

Next we are going to build a connector between our RabbitMQ server and our in-memory-db MuSQL cluster server.

High Scalable & Available part 1: Architecture

Hi, it has been a long time since I haven’t write any tutorial, and after I made some experiment with MySQL Cluster and RabbitMQ, its time for me to build a high scalable & available IT infrastructure.

Actually, there are lots of approaches out there to gain a highly scalable and available IT infrastructure.

Here, I just want to discuss the one which is suits to my business and technical requirements.

A. My Requirements are:

1. I wanna have a system that can serves millions of users
2. I wanna be able to serves my users in almost real-time
3. I wanna start small, but my system should be able to scale in parallel with the amount of our end-users

B. My Architecture:

The above diagram shows the high-level concept of the infra I’m about to built. The explanations are:

1. All the green boxes are the nodes we are going to built,
2. All the yellow boxes are the nodes that replicates the green nodes, when we want to scale our system’s capacity,
3. The Message Queuing node, is the node that will enable our system to scale horizontally.

In detail, each of our Application Server will consists of:

1. Our custom web application,
2. An MQ connector module (inside our custom app), see below diagram

Meanwhile, each of our DB Cluster will consists of:

1. One node that will host an ndb_mgmd node, and an MQ connector,
2. Two nodes that will act as Master & Slave, that each will work as both ndbd node and mysqld, see below diagram.

C. Technologies

Technologies that I want to use, in building this infrastructure are:

1. For my custom application, I will use: Mochiweb, an Erlang web server and web application framework,
2. For my message queing, I will use: RabbitMQ, an Erlang based message queue server,
3. For building my App to MQ Connector, I will use: Erlang AMQP client library,
4. For building my DB to MQ Connector, I will use: Java AMQP client library and ClusterJ,
5. For building my Master/Slave DB layer, I will use MySQL Cluster.

Ok that’s it for now, next we are going to start our small project with RabbitMQ installation in Ubuntu server v.12.04.

Warning:

For other beginners like me: this architecture, is designed by me, to cater my current personal project’s requirements (some requirements, are not mentioned here, as those are classified). This architecture may not suitable for many other use cases out there.