Hello, I manage a www (apache), database (postgresql) and e-mail (qmail) with about 100 Internet domains. We recently purchased a DELL MD3000i server with data access through ISCSI protocol. I installed a new server with FreeBSD 7.0 compiled kernel with ISCSI (v. 2.1) and SCHED_ULE scheduler. The server's local disk configuration is: 512 MB / 1024 MB SWAP 254 MB / var 2048MB / var/ qmail 512 MB /tmp ~ 70 GB /usr The apache data (/usr/local/www), postgre (/usr/local/pgsql) and qmail (/usr/local/vpopmail) are instead on the partition on ISCSI. After installing all the necessary softwares I tested the machine as mail server for a one domain and everything worked fine for 2-3 days. I wanted to test the correct functioning of qmail and of the reading/writing on ISCSI. On the end of testing I decided to transfer all of data (www, DB and e-mail) on the FreeBSD 7.0. I compressed all the data on the original server (on tar.gz format). I transferred them on the new server and started to decompress the files. Www data have been decompressed correctly on the ISCSI disk. But while the messages were being decompressed (~80GB of tar.gz) and all the DB were being restored, the server crashed. The error shown was: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 31, size: 4096 This message repeated every 30 seconds. While restarting with the single user mode I checked there wasn't any message after the crash in /var/log/messages. I think 1GB of swap should be enough, since with this configuration on FreeBSD 5.5 everything worked for over 270 days without reboot. The local disks are 2 of ~80GB in RAID 1 (mirror) with Gmirror before installing FreeBSD I tested the 2 disks with the badblocks software on Slackware which didn't find any r/w error. Can somebody help me understand why the system crashed this way or how to avoid future crash? It's a production server and I can't afford an out-of-service or data loss. Thank you very much and kind regards. -- Sossi Andrej ------------------------- DotCom Information technology Via Biancospino, 9 34151 - Opicina (TS) Italy tel: +39 040 2158191 fax: +39 040 0641954 E-mail: asossi@dotcom.ts.it ---------------------------- Ai sensi del D.lgs n. 196 del 30.06.03 (Codice Privacy) si precisa che le informazioni contenute in questo messaggio sono riservate e ad uso esclusivo del destinatario. Qualora il messaggio in parola Le fosse pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non inoltrarlo a terzi, dandocene gentilmente comunicazione. Grazie This message, for the D.lgs n. 196 / 30.06.03 (Privacy Code), may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
Paul B. Mahol wrote:> On 11/17/08, Sossi Andrej <asossi@dotcom.ts.it> wrote: > >> Hello, >> I manage a www (apache), database (postgresql) and e-mail (qmail) with >> about 100 Internet domains. >> We recently purchased a DELL MD3000i server with data access through >> ISCSI protocol. >> I installed a new server with FreeBSD 7.0 compiled kernel with ISCSI >> (v. 2.1) and SCHED_ULE scheduler. >> The server's local disk configuration is: >> 512 MB / >> 1024 MB SWAP >> 254 MB / var >> 2048MB / var/ qmail >> 512 MB /tmp >> ~ 70 GB /usr >> >> The apache data (/usr/local/www), postgre (/usr/local/pgsql) and qmail >> (/usr/local/vpopmail) are instead on the partition on ISCSI. After >> installing all the necessary softwares I tested the machine as mail >> server for a one domain and everything worked fine for 2-3 days. >> I wanted to test the correct functioning of qmail and of the >> reading/writing on ISCSI. On the end of testing I decided to transfer >> all of data (www, DB and e-mail) on the FreeBSD 7.0. I compressed all >> the data on the original server (on tar.gz format). I transferred them >> on the new server and started to decompress the files. Www data have >> been decompressed correctly on the ISCSI disk. But while the messages >> were being decompressed (~80GB of tar.gz) and all the DB were being >> restored, the server crashed. >> > > Which application were used for decompression? >I use this command: tar -xvzf filename.tar.gz (for file decompression as root) and psql -f filename.sql postgres (for restore all databases)>> The error shown was: >> >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 31, size: 4096 >> > > I believe this can happen any time when to much of swap partition is used. > If this sort of thing happens often, review memory management of application(s) > that is causing such scenario, if that is application fault - you have > two choices: > get more RAM (this may not always help) or use another application. > Increasing swap partition will not help much .... >During decompression i sometimes monitor the system with command top and I don't see the system use swap at all, but I didn't monitor constantly. If the problem is caused by tar, what choices do i have to transport files from original server to another quickly?>> This message repeated every 30 seconds. >> While restarting with the single user mode I checked there wasn't any >> message after the crash in /var/log/messages. >> >> I think 1GB of swap should be enough, since with this configuration on >> FreeBSD 5.5 everything worked for over 270 days without reboot. >> The local disks are 2 of ~80GB in RAID 1 (mirror) with Gmirror >> before installing FreeBSD I tested the 2 disks with the badblocks >> software on Slackware which didn't find any r/w error. >> >> Can somebody help me understand why the system crashed this way or how >> to avoid future crash? >> It's a production server and I can't afford an out-of-service or data loss. >> >> Thank you very much and kind regardsThank for your reply. -- Sossi Andrej ------------------------- DotCom Information technology Via Biancospino, 9 34151 - Opicina (TS) Italy tel: +39 040 2158191 fax: +39 040 0641954 E-mail: asossi@dotcom.ts.it ---------------------------- Ai sensi del D.lgs n. 196 del 30.06.03 (Codice Privacy) si precisa che le informazioni contenute in questo messaggio sono riservate e ad uso esclusivo del destinatario. Qualora il messaggio in parola Le fosse pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non inoltrarlo a terzi, dandocene gentilmente comunicazione. Grazie This message, for the D.lgs n. 196 / 30.06.03 (Privacy Code), may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation.
On Mon, Nov 17, 2008 at 05:18:33PM +0100, Sossi Andrej wrote:> Hello, > I manage a www (apache), database (postgresql) and e-mail (qmail) with > about 100 Internet domains. > We recently purchased a DELL MD3000i server with data access through > ISCSI protocol. > I installed a new server with FreeBSD 7.0 compiled kernel with ISCSI > (v. 2.1) and SCHED_ULE scheduler. > The server's local disk configuration is: > 512 MB / > 1024 MB SWAP > 254 MB / var > 2048MB / var/ qmail > 512 MB /tmp > ~ 70 GB /usr > > The apache data (/usr/local/www), postgre (/usr/local/pgsql) and qmail > (/usr/local/vpopmail) are instead on the partition on ISCSI. After > installing all the necessary softwares I tested the machine as mail > server for a one domain and everything worked fine for 2-3 days. > I wanted to test the correct functioning of qmail and of the > reading/writing on ISCSI. On the end of testing I decided to transfer > all of data (www, DB and e-mail) on the FreeBSD 7.0. I compressed all > the data on the original server (on tar.gz format). I transferred them > on the new server and started to decompress the files. Www data have > been decompressed correctly on the ISCSI disk. But while the messages > were being decompressed (~80GB of tar.gz) and all the DB were being > restored, the server crashed. > The error shown was: > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 31, size: 4096 > > This message repeated every 30 seconds.Is this of any help? http://lists.freebsd.org/pipermail/freebsd-scsi/2008-February/003383.html http://lists.freebsd.org/pipermail/freebsd-scsi/2008-February/003387.html -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
On 11/17/08, Sossi Andrej <asossi@dotcom.ts.it> wrote:> Hello, > I manage a www (apache), database (postgresql) and e-mail (qmail) with > about 100 Internet domains. > We recently purchased a DELL MD3000i server with data access through > ISCSI protocol. > I installed a new server with FreeBSD 7.0 compiled kernel with ISCSI > (v. 2.1) and SCHED_ULE scheduler. > The server's local disk configuration is: > 512 MB / > 1024 MB SWAP > 254 MB / var > 2048MB / var/ qmail > 512 MB /tmp > ~ 70 GB /usr > > The apache data (/usr/local/www), postgre (/usr/local/pgsql) and qmail > (/usr/local/vpopmail) are instead on the partition on ISCSI. After > installing all the necessary softwares I tested the machine as mail > server for a one domain and everything worked fine for 2-3 days. > I wanted to test the correct functioning of qmail and of the > reading/writing on ISCSI. On the end of testing I decided to transfer > all of data (www, DB and e-mail) on the FreeBSD 7.0. I compressed all > the data on the original server (on tar.gz format). I transferred them > on the new server and started to decompress the files. Www data have > been decompressed correctly on the ISCSI disk. But while the messages > were being decompressed (~80GB of tar.gz) and all the DB were being > restored, the server crashed.Which application were used for decompression?> The error shown was: > > swap_pager: indefinite wait buffer: bufobj: 0, blkno: 31, size: 4096I believe this can happen any time when to much of swap partition is used. If this sort of thing happens often, review memory management of application(s) that is causing such scenario, if that is application fault - you have two choices: get more RAM (this may not always help) or use another application. Increasing swap partition will not help much ....> This message repeated every 30 seconds. > While restarting with the single user mode I checked there wasn't any > message after the crash in /var/log/messages. > > I think 1GB of swap should be enough, since with this configuration on > FreeBSD 5.5 everything worked for over 270 days without reboot. > The local disks are 2 of ~80GB in RAID 1 (mirror) with Gmirror > before installing FreeBSD I tested the 2 disks with the badblocks > software on Slackware which didn't find any r/w error. > > Can somebody help me understand why the system crashed this way or how > to avoid future crash? > It's a production server and I can't afford an out-of-service or data loss. > > Thank you very much and kind regards. > > -- > Sossi Andrej > ------------------------- > DotCom Information technology > > Via Biancospino, 9 > 34151 - Opicina (TS) > Italy > > tel: +39 040 2158191 > fax: +39 040 0641954 > E-mail: asossi@dotcom.ts.it > ---------------------------- > > Ai sensi del D.lgs n. 196 del 30.06.03 (Codice Privacy) si precisa che > le informazioni contenute in questo messaggio sono riservate e ad uso > esclusivo del destinatario. Qualora il messaggio in parola Le fosse > pervenuto per errore, La preghiamo di eliminarlo senza copiarlo e di non > inoltrarlo a terzi, dandocene gentilmente comunicazione. Grazie > > This message, for the D.lgs n. 196 / 30.06.03 (Privacy Code), may > contain confidential and/or privileged information. If you are not the > addressee or authorized to receive this for the addressee, you must not > use, copy, disclose or take any action based on this message or any > information herein. If you have received this message in error, please > advise the sender immediately by reply e-mail and delete this message. > Thank you for your cooperation. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Sossi Andrej schrieb:> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 31, size: 4096 > > Can somebody help me understand why the system crashed this way or how > to avoid future crash?This happens when it takes more than 20 seconds to swap out a page (http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/troubleshoot.html#INDEFINITE-WAIT-BUFFER). I had the same problem some time ago with FreeBSD 6.0 when making Backups (so the disks were busy). For me it helped to increase the timeout from 20 to 40 seconds in vm/swap_pager.c (http://fxr.watson.org/fxr/source/vm/swap_pager.c?v=FREEBSD60#L1103) -- Martin