Sunday, April 28, 2013

Performance check twemproxy and redis

Hi friends today m sharing with you Performance check  twemproxy and redis

Redis is an open source, BSD licensed, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets. Redis in itself is  a great data store supporting a variety of datastructure and highly rich feature base. Redis is a event based system running single thread giving throughput  as high as 500k on localhost.

twemproxy (pronounced "two-em-proxy"), aka nutcracker is a fast and lightweight proxy for memcached and redis protocol. It was primarily built to reduce the connection count on the backend caching servers. This is also an event based system running single thread giving a high throughput.
With twemproxy it is also possible to shard data over multiple redis instances.

We performed  various testes with twemproxy and redis  to find out the drop in through by applying twemproxy before redis. Since both proxy and redis runs in single thread, to optimize on cpu resources it is also possible to pinn any of these process to a single core.
following are the test result performed on our laptops with intel i5  4 core porcessors.
Non-Pipelined Pinned Non-pipelined Non-Pinned Pipelined Pipelined and Pinned
Zero Hops - Direct Redis 120k 120k 450k 500k
Zero Hops - with twemproxy 75k 80k 370k 370k
One Hop - Client to (Twemproxy & redis) 63k 66k 108k 109k
One Hop -b/w Twemproxy & redis 88k 90k 117k 118k
One Hop - Client to direct Redis 64k 65k 119k 119k
Two hops (client redis and proxy on diff machine) 45k 46k 180k 180k

 it is clear that redis alone can give a QPS of 500k when both client and server are running on same host. This performance drops by ~20% but still very high. It drops as number of network hops increases for obvious reasons.



Saturday, April 27, 2013

Simple redirections 


A little note for seeing this things: with the less command you can view both stdout (which will remain on the buffer) and the stderr that will be printed on the screen, but erased as you try to 'browse' the buffer.

3.2 Sample: stdout 2 file

This will cause the ouput of a program to be written to a file.

        ls -l > ls-l.txt
        
Here, a file called 'ls-l.txt' will be created and it will contain what you would see on the screen if you type the command 'ls -l' and execute it.

3.3 Sample: stderr 2 file

This will cause the stderr ouput of a program to be written to a file.

        grep da * 2> grep-errors.txt
        
Here, a file called 'grep-errors.txt' will be created and it will contain what you would see the stderr portion of the output of the 'grep da *' command.

3.4 Sample: stdout 2 stderr

This will cause the stderr ouput of a program to be written to the same filedescriptor than stdout.

        grep da * 1>&2 
        
Here, the stdout portion of the command is sent to stderr, you may notice that in differen ways.

3.5 Sample: stderr 2 stdout

This will cause the stderr ouput of a program to be written to the same filedescriptor than stdout.

        grep * 2>&1
        
Here, the stderr portion of the command is sent to stdout, if you pipe to less, you'll see that lines that normally 'dissapear' (as they are written to stderr) are being kept now (because they're on stdout).

3.6 Sample: stderr and stdout 2 file

This will place every output of a program to a file. This is suitable sometimes for cron entries, if you want a command to pass in absolute silence.

        rm -f $(find / -name core) &> /dev/null 
        

Saturday, April 20, 2013

redirect TCP connections with redir

Sometimes you need an easy way to redirect incoming connections to another system. For example when migrating an old box to a new one. redir: Redirect TCP connections It can run under inetd or stand alone (in which case it handles multiple connections). Its 8 bit clean, not limited to line mode, is small and light. Supports FTP redirects and supports transparency support. redir is all you need to redirect traffic across firewalls authenticate based on an IP address etc etc. No need for the firewall toolkit. The functionality of inetd/tcpd and "redir" will allow you to do everything you need without screwy telnet/ftp etc gateways. (I assume you are running IP Masquerading of course.)
You’ll need the ‘redir’ program for this to work:
apt-get install redir

Redir redirects tcp connections coming in to a local port to a specified address/port combination.

To redirect TCP connections with redir utility,  Simply type the
command such as below

redir --laddr=<listen_address> --lport=<listen_port>  --caddr=<connect_address>  --cport=<connect_port>

Suppose that the ip address of our system is 60.50.40.31 and we woud like to connect to 64.64.64.64  ip address and 3120 port over our system. Simply, we want redirutiliy to redirect connections coming to 60.50.40.31 on port 3120 to 64.64.64.64  port 3120. We have to run redir such as below to do so:
redir --laddr=60.50.40.31 --lport=3120  --caddr=64.64.64.64 --cport=3120