Hi, I plan to build a SIP TCP server (no UDP) based on Unicorn/Rainbows! HTTP server. The main different between a SIP server and HTTP server are: - SIP uses persistent TCP connections, so I should use Rainbows!. - For a SIP server it''s not valid a simple request-response model. Different workers could handle SIP messages (requests and responses) belonging to the same SIP session so I need a shared memory between all the workers. Another option is using EventMachine, perhaps more suitable for this purpose by design as it uses a single Ruby process so sharing memory is not a problem. In the other side using a single process in a multicore server is a pain. I would like to use Unicorn/Rainbows as I love its design: by far it''s the more reliable and efficient Ruby HTTP server and it takes advantages of Unix''s features. I don''t want to use a DB server neither MemCache as "shared memory" as it would be too slow. Is there any way to share RAM memory between different Unicorn/Rainbows! workers in a *safe* way? I could create a Hash or Array of SIP sessions into the master process so all the workers reuse it, but I don''t think it would be safe to access/write into it from different Ruby processes. For that I would also need a semaphore system (perhaps again a shared variable between all workers in order to lock the shared Array/Hash when a worker writes into it). Any tip about it? suggstions? Thanks a lot. -- I?aki Baz Castillo <ibc at aliax.net>
I?aki Baz Castillo <ibc at aliax.net> wrote:> Hi, I plan to build a SIP TCP server (no UDP) based on > Unicorn/Rainbows! HTTP server. The main different between a SIP server > and HTTP server are: > > - SIP uses persistent TCP connections, so I should use Rainbows!. > - For a SIP server it''s not valid a simple request-response model. > Different workers could handle SIP messages (requests and responses) > belonging to the same SIP session so I need a shared memory between > all the workers. > > Another option is using EventMachine, perhaps more suitable for this > purpose by design as it uses a single Ruby process so sharing memory > is not a problem. In the other side using a single process in a > multicore server is a pain. > I would like to use Unicorn/Rainbows as I love its design: by far it''s > the more reliable and efficient Ruby HTTP server and it takes > advantages of Unix''s features. > > I don''t want to use a DB server neither MemCache as "shared memory" as > it would be too slow. Is there any way to share RAM memory between > different Unicorn/Rainbows! workers in a *safe* way? I could create a > Hash or Array of SIP sessions into the master process so all the > workers reuse it, but I don''t think it would be safe to access/write > into it from different Ruby processes. For that I would also need a > semaphore system (perhaps again a shared variable between all workers > in order to lock the shared Array/Hash when a worker writes into it). > > Any tip about it? suggstions?Hi I?aki, First off I wouldn''t call Memcached slow... How many requests do you need out of it and what kind of access patterns are you doing? If you''re on a modern Linux 2.6 box, then anything on the local filesystem is basically shared memory if you have enough RAM to store your working set (and it sounds like you do). If you''re willing to be confined to a single box, I''ve had great experiences with TokyoCabinet the past few years. One system I built shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn (previously Mongrel) workers. TokyoCabinet uses mmap and pread/pwrite, so you can safely share DB handles across any number of native threads/processes. Writes in TC need filesystem locking and you can only have one active writer, but that was still plenty fast enough in my experience. If fsync() becomes a bottleneck, then for non-critical data either: 1) disable it (TC lets you) 2) or use libeatmydata if you choose something that doesn''t let you disable fsync() <soapbox> My personal opinion is that fsync()/O_SYNC are completely overrated. All important machines these days have redundant storage, redundant power supplies, ECC memory, hardware health monitoring/alerting and backup power. Any remaining hardware/kernel failure possibilities are far less likely than application bugs that corrupt data :) But, if you really have critical data, /then/ get an SSD or a battery-backed write cache. </soapbox> -- Eric Wong
2010/4/26 Eric Wong <normalperson at yhbt.net>:> Hi I?aki, > > First off I wouldn''t call Memcached slow... ?How many requests do you > need out of it and what kind of access patterns are you doing?Well, I want to do as efficient as possible. This is not the main purpose (if so I wouldn''t use Ruby) but it''s important. For example: - 8 workers. - A SIP request arrives and it handled by worker 1. - The server must store in *shared* memory info about such request (status, last SIP response sereplied for it....). The request transaction remains in memory for some seconds after sending the response. - Due to a non so unusual SIP issue, a retransmission of the previous request arrives to the server but now it is handled by worker 2. - Worker 2 must check the request "branch" id into shared memory, find the previously created entry, check that a response was already sent for it, and re-send such response. Well, in fact this is really common in SIP under UDP. Under TCP usually there are no retransmissions (but they could be so the worker must check it).> If you''re on a modern Linux 2.6 box, then anything on the local > filesystem is basically shared memory if you have enough RAM to store > your working set (and it sounds like you do).And what about mounting a filesystem under RAM? :would it be ever faster? :)> If you''re willing to be confined to a single box, I''ve had great > experiences with TokyoCabinet the past few years. ?One system I built > shares 10G of read-mostly data (on 16G boxes) across 8 Unicorn > (previously Mongrel) workers. ?TokyoCabinet uses mmap and pread/pwrite, > so you can safely share DB handles across any number of native > threads/processes.I didn''t know it. I''ve just read the doc, installed and tested it. Really impressive!> Writes in TC need filesystem locking and you can only have one active > writer, but that was still plenty fast enough in my experience.If you use a "on memory database" there wouldn''t be filesystem locking, right?> If fsync() becomes a bottleneck, then for non-critical data either: > > ?1) disable it (TC lets you) > ?2) or use libeatmydata if you choose something that doesn''t let > ? ? you disable fsync() > > <soapbox> > ?My personal opinion is that fsync()/O_SYNC are completely overrated. > ?All important machines these days have redundant storage, redundant > ?power supplies, ECC memory, hardware health monitoring/alerting and > ?backup power. ?Any remaining hardware/kernel failure possibilities > ?are far less likely than application bugs that corrupt data :) > > ?But, if you really have critical data, /then/ get an SSD or > ?a battery-backed write cache. > </soapbox>Thanks, but the fact is that I don''t need persistent storage. This is, the data I will need to share between Ruby processes (workers) is about realtime data, so if the server is stopped and restarted I don''t want to retrieve such information. Well, it could be useful for some cases however, I need to think about it. I''ve seen a method "sync" in the Tokyo Tyrant client API which forces the server to store/sync the on memory data to the file (if a file is being used). So thanks a lot. Definitely I''ll learn about TokioCabinet as it seems a really nice solution for my needs. Regards. -- I?aki Baz Castillo <ibc at aliax.net>
> If you use a "on memory database" there wouldn''t be filesystem locking, right?fsync throughput (a.k.a. commit rate) is easy to test: sudo aptitude install sysbench sysbench --test=fileio --file-fsync-freq=1 --file-num=1 \ --file-total-size=16384 --file-test-mode=rndwr run \ | grep "Requests/sec" You can then see if any additional tuning (RAM drive, etc.) is even necessary for your requirements. Unless something is doing write-back caching in between the app and the drive this rate will be a function of RPM of your drive, e.g. ~120/s for 7200RPM We did a bunch of benchmarking when setting up a couple of DB servers recently and found that consumer-grade drives (SATA) lie and have write-back caching enabled by default, and you get 1-3k/s. It''s not considered safe to run a database on such configuration as HDD cache lacks battery backup. For comparison we get ~10k/s on a 4 disk RAID 10 SAS array (w/ battery backup unit) with write-back caching turned on in the controller and off in the drives. So, the point is that consumer hardware lies and you are quite likely to get good fsync performance. Since you don''t care about data safety for your setup this should be fine. Best regards, Gleb