By a Combiner function I mean even if u get multiple copy of same key from Cluster you going to make reading this blog, this function should allow either to choose one or combine all.Since you dont know if one of the redis instance fails, to which the twemproxy will write data, you need to read all redis instances and merge the data collected from each of the instances hence i put in requirements Combiner function
If you have machine having multiple cores,
- choose to run multiple instances of redis on same machine.
- chose to run multiple such machines. (this makes architecture more fault tolerant )
- its better to run master slave replication with each redis instance, i.e half of them wil be slaves and half of them will be master.
- it is better to keep configuration such that master and slave are not on same machine.
- it is advisable to even keep slaves of same machine redis instances on different machine if you can. to minimize loss of data
- make masters non persistent, and slave persistent.
- choose to run multiple instances of twemproxy and shard your writes on them, if you feel you have higher traffic than a twemproxy can handle. Multiple instances are good to make system more fault tolerant, just in case one twemprozy dies for some or other reason
- Configure all twemproxy to write to all redis masters instances and with same deterministic technique, consistent hash, and hashing algorithm.
If somthing bad can happen, it will happen
Here are few advantages that I find using this configuration.
- You can pin masters to some core, this will make your system look like each core listen to a port.