Lustre discuss - May 2006 - Bad performance on IA64

On Tue, 2004-05-04 at 20:00, sky wrote:
> I have patched lustre into 2.4.20 kernel with little hacked code. It
> worked on IA64.
> But I achieved bad performance. Result of bonnie show that bandwidth
> of block write is about 20MB/s.But NFS is about 40MB/s.Is it normal? 
> Any suggestion? Maybe something to be tuned to make it faster?
> 
> BTW Our paltform hardware: IA64+IPoIB (infiniband)+Ultra320 SCSI disk
It''s hard to be sure exactly what is going on without more details of
your configuration, but the problem is probably related to IP over I/B.

We are focused on writing a native I/B driver, and not running over IP,
so we have done zero tuning or testing to see if this can be made to
work well.  So far, our customers are content to wait for a native nal.

Progress on the I/B nal is not fast, largely because of factors outside
of our control (like unstable drivers).  It will almost certainly be a
couple of months before we have a nal ready to be seriously tested, or
perhaps slightly sooner if some additional funding becomes available.

Sorry that I can''t give you a good answer right now -- hopefully soon!

-Phil

I have patched lustre into 2.4.20 kernel with little hacked code. It worked on
IA64.
But I achieved bad performance. Result of bonnie show that bandwidth of block
write is about 20MB/s.But NFS is about 40MB/s.Is it normal?
Any suggestion? Maybe something to be tuned to make it faster?

BTW Our paltform hardware: IA64+IPoIB (infiniband)+Ultra320 SCSI disk





=========================263µç×ÓÓʼþ£­ÐÅÀµÓÊ×Ôרҵ

This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

---559023410-824023566-1085418421=:25446
Content-Type: TEXT/PLAIN; charset=US-ASCII


Hi,

   I have set up Lustre version 1.0.4 on a two node cluster. Node 1 is
configured as MDS, LOV with 2 OSTS, and Client. Node is only configured as
client. When I run parallel I/O programs (MPI-IO) which does repeated
file I/O on a single file, it works fine if being run only on Node 2. When
I include Node 1 as well, the application hangs. Sometimes it completes
when number of parallel process executed on each node = 1 (but reads are
very slow). When number of process allocated to each node is increased to
2 or more, the application hangs and crashes the file system (Lustre file
system gives Input/Output Error). The log message is attached.

   Is this bug related to Bug #2227 since the messages seem to correspond.
There is known bugs listed in Lustre Website "Runnign client and OST on
the same node is still not 100% stable. Application or system hangs are
possible". Although, I run on nodes configured as OST and clients, however
I perform I/O on same file with chunk size 4KB and application hangs if
each process on node reads/writes to same file multiple times (16-32).
Each node has sufficient memory (128 MB RAM). Any advise/comments will be
greatly appreciated.

Thanks,
-Kums
---559023410-824023566-1085418421=:25446
Content-Type: TEXT/PLAIN; charset=US-ASCII;
name="lustre_v1_0_4_hang_log.txt"
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.GSO.4.58.0405241207010.25446@mpi.mpi-softtech.com>
Content-Description: 
Content-Disposition: attachment; filename="lustre_v1_0_4_hang_log.txt"

TWF5IDI0IDIyOjI1OjIxIGx1c3RyZTAga2VybmVsOiBMdXN0cmVFcnJvcjog
MjIyNjooZmlsdGVyX2lvLmM6MTkzOmZpbHRlcl9wcmVwcndfcmVhZCgpKSBz
bG93IHN0YXJ0X3BhZ2VfcmVhZCAxMDBzDQpNYXkgMjQgMjI6MjU6MjEgbHVz
dHJlMCBrZXJuZWw6IEx1c3RyZUVycm9yOiAyMjI2OihmaWx0ZXJfaW8uYzoy
MTE6ZmlsdGVyX3ByZXByd19yZWFkKCkpIHNsb3cgZmluaXNoX3BhZ2VfcmVh
ZCAxMDBzDQpNYXkgMjQgMjI6MjU6MjEgbHVzdHJlMCBrZXJuZWw6IEx1c3Ry
ZUVycm9yOiAyMjI1OihmaWx0ZXJfaW8uYzoxOTM6ZmlsdGVyX3ByZXByd19y
ZWFkKCkpIHNsb3cgc3RhcnRfcGFnZV9yZWFkIDEwMHMNCk1heSAyNCAyMjoy
NToyMSBsdXN0cmUwIGtlcm5lbDogTHVzdHJlRXJyb3I6IDIyMjU6KGZpbHRl
cl9pby5jOjIxMTpmaWx0ZXJfcHJlcHJ3X3JlYWQoKSkgc2xvdyBmaW5pc2hf
cGFnZV9yZWFkIDEwMHMNCg0KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQ0K
DQpNYXkgMjQgMjI6MjU6NDYgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVycm9y
OiAyMjI1OihzZXJ2aWNlLmM6MzkzOnB0bHJwY19tYWluKCkpDQpEcm9wcGlu
ZyByZXF1ZXN0IGZyb20gTklEIDB4YTAwMDEyZCBiZWNhdXNlIGl0J3MgMTI1
IHNlY29uZHMgb2xkLg0KTWF5IDI0IDIyOjI1OjQ2IGx1c3RyZTAga2VybmVs
OiBMdXN0cmVFcnJvcjogMjIyNzoob3N0X2hhbmRsZXIuYzo0Njg6b3N0X2Jy
d19yZWFkKCkpIEBAQCB0aW1lb3V0IG9uIGJ1bGsgUFVUIHJlcUBjNjc4Mzgw
MCB4NjIwMS90MCBvMy0+PD8+QE5JRF8xMC4wLjEuNDVfVVVJRDotMSBsZW5z
IDMwNC8yNDAgcmVmIDAgZmwgP3BoYXNlPzovMC8wIHJjIDAvMA0KTWF5IDI0
IDIyOjI1OjQ2IGx1c3RyZTAga2VybmVsOiBMdXN0cmVFcnJvcjogMjIyNzoo
c2VydmljZS5jOjQwNjpwdGxycGNfbWFpbigpKQ0KcmVxdWVzdCA2MjAxIGZy
b20gTklEIDB4YTAwMDEyZCBwcm9jZXNzZWQgaW4gMTI1MDY2MTExdXMgKDEy
NTA3MTY2OHVzIHRvdGFsKQ0KTWF5IDI0IDIyOjI1OjQ2IGx1c3RyZTAga2Vy
bmVsOiBMdXN0cmVFcnJvcjogMjA0MDoobGliLW1vdmUuYzoxMzQ6bGliX2Zp
bmRfbWUoKSkgMTY3NzcyNDYxOiBEcm9wcGluZyBQVVQgZnJvbSAxNjc3NzI0
NjEuMCBwb3J0YWwgNCBtYXRjaCAweDE4M2Igb2Zmc2V0IDAgbGVuZ3RoDQoy
NDA6IG5vIG1hdGNoDQpNYXkgMjQgMjI6MjU6NDYgbHVzdHJlMCBrZXJuZWw6
IEx1c3RyZUVycm9yOiAyMDQwOihsaWItbW92ZS5jOjEzNDpsaWJfZmluZF9t
ZSgpKSAxNjc3NzI0NjE6IERyb3BwaW5nIFBVVCBmcm9tIDE2Nzc3MjQ2MS4w
IHBvcnRhbCA0IG1hdGNoIDB4MTgzYSBvZmZzZXQgMCBsZW5ndGgNCjI0MDog
bm8gbWF0Y2gNCk1heSAyNCAyMjoyNTo0NiBsdXN0cmUwIGtlcm5lbDogTHVz
dHJlRXJyb3I6IDIwNDE6KGxpYi1tb3ZlLmM6MTM0OmxpYl9maW5kX21lKCkp
IDE2Nzc3MjQ2MTogRHJvcHBpbmcgUFVUIGZyb20gMTY3NzcyNDYxLjAgcG9y
dGFsIDQgbWF0Y2ggMHgxODM5IG9mZnNldCAwIGxlbmd0aA0KMjQwOiBubyBt
YXRjaA0KTWF5IDI0IDIyOjI3OjQ0IGx1c3RyZTAgcGFtX3Job3N0c19hdXRo
WzcyOThdOiBhbGxvd2VkIHRvIGt1bXNAbHVzdHJlMS5tcGktc29mdHRlY2gu
Y29tIGFzIGt1bXMNCk1heSAyNCAyMjoyNzo0NCBsdXN0cmUwIHJzaChwYW1f
dW5peClbNzI5OF06IHNlc3Npb24gb3BlbmVkIGZvciB1c2VyIGt1bXMgYnkg
KHVpZD0wKQ0KTWF5IDI0IDIyOjI3OjQ0IGx1c3RyZTAgcnNoKHBhbV91bml4
KVs3Mjk4XTogc2Vzc2lvbiBjbG9zZWQgZm9yIHVzZXIga3Vtcw0KTWF5IDI0
IDIyOjI4OjAxIGx1c3RyZTAga2VybmVsOiBMdXN0cmVFcnJvcjogNzMwMzoo
bGxpdGVfbGliLmM6OTA2OmxsX3N0YXRmc19pbnRlcm5hbCgpKSBvYmRfc3Rh
dGZzIGZhaWxzOiByYyA9IC01DQoNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tDQoNCk1heSAyNCAyMjozNTo1MSBsdXN0cmUwIGtlcm5lbDogTHVz
dHJlRXJyb3I6IDIyMjU6KGZpbHRlcl9pby5jOjE5MzpmaWx0ZXJfcHJlcHJ3
X3JlYWQoKSkgc2xvdyBzdGFydF9wYWdlX3JlYWQgMjBzDQpNYXkgMjQgMjI6
MzU6NTEgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVycm9yOiAyMjI1OihmaWx0
ZXJfaW8uYzoyMTE6ZmlsdGVyX3ByZXByd19yZWFkKCkpIHNsb3cgZmluaXNo
X3BhZ2VfcmVhZCAyMHMNCk1heSAyNCAyMjozNTo1MSBsdXN0cmUwIGtlcm5l
bDogTHVzdHJlRXJyb3I6IDIyMjc6KGZpbHRlcl9pby5jOjE5MzpmaWx0ZXJf
cHJlcHJ3X3JlYWQoKSkgc2xvdyBzdGFydF9wYWdlX3JlYWQgMjBzDQpNYXkg
MjQgMjI6MzU6NTEgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVycm9yOiAyMjI3
OihmaWx0ZXJfaW8uYzoyMTE6ZmlsdGVyX3ByZXByd19yZWFkKCkpIHNsb3cg
ZmluaXNoX3BhZ2VfcmVhZCAyMHMNCk1heSAyNCAyMjozNTo1MSBsdXN0cmUw
IGtlcm5lbDogTHVzdHJlRXJyb3I6IDIyMjY6KGZpbHRlcl9pby5jOjE5Mzpm
aWx0ZXJfcHJlcHJ3X3JlYWQoKSkgc2xvdyBzdGFydF9wYWdlX3JlYWQgNjBz
DQpNYXkgMjQgMjI6MzU6NTEgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVycm9y
OiAyMjI2OihmaWx0ZXJfaW8uYzoyMTE6ZmlsdGVyX3ByZXByd19yZWFkKCkp
IHNsb3cgZmluaXNoX3BhZ2VfcmVhZCA2MHMNCk1heSAyNCAyMjozNTo1MSBs
dXN0cmUwIGtlcm5lbDogTHVzdHJlRXJyb3I6IDIwNDA6KGxpYi1tb3ZlLmM6
MTM0OmxpYl9maW5kX21lKCkpIDE2Nzc3MjQ2MTogRHJvcHBpbmcgUFVUIGZy
b20gMTY3NzcyNDYxLjAgcG9ydGFsIDggbWF0Y2ggMHgxYzY3IG9mZnNldCAw
IGxlbmd0aA0KNTI0Mjg4OiBubyBtYXRjaA0KTWF5IDI0IDIyOjM1OjUxIGx1
c3RyZTAgcnNoKHBhbV91bml4KVs3NjM0XTogc2Vzc2lvbiBjbG9zZWQgZm9y
IHVzZXIga3Vtcw0KTWF5IDI0IDIyOjM2OjE1IGx1c3RyZTAga2VybmVsOiBM
dXN0cmVFcnJvcjogMjIyNToob3N0X2hhbmRsZXIuYzo0Njg6b3N0X2Jyd19y
ZWFkKCkpIEBAQCB0aW1lb3V0IG9uIGJ1bGsgUFVUIHJlcUBjNjc4M2MwMCB4
OTIxL3QwIG8zLT48Pz5AOi0xIGxlbnMgMjg4LzI0MCByZWYgMCBmbCA/cGhh
c2U/Oi8wLzAgcmMgMC8wDQpNYXkgMjQgMjI6MzY6MTUgbHVzdHJlMCBrZXJu
ZWw6IEx1c3RyZUVycm9yOiAyMjI1OihzZXJ2aWNlLmM6MzkzOnB0bHJwY19t
YWluKCkpDQpEcm9wcGluZyByZXF1ZXN0IGZyb20gTklEIDB4YTAwMDEyZSBi
ZWNhdXNlIGl0J3MgMTI0IHNlY29uZHMgb2xkLg0KTWF5IDI0IDIyOjM2OjE1
IGx1c3RyZTAgbGFzdCBtZXNzYWdlIHJlcGVhdGVkIDIgdGltZXMNCk1heSAy
NCAyMjozNjoxNSBsdXN0cmUwIGtlcm5lbDogTHVzdHJlRXJyb3I6IDIyMjc6
KG9zdF9oYW5kbGVyLmM6NDY4Om9zdF9icndfcmVhZCgpKSBAQEAgdGltZW91
dCBvbiBidWxrIFBVVCByZXFAYzY3ODM4MDAgeDkyMC90MCBvMy0+PD8+QDot
MSBsZW5zIDI4OC8yNDAgcmVmIDAgZmwgP3BoYXNlPzovMC8wIHJjIDAvMA0K
TWF5IDI0IDIyOjM2OjE1IGx1c3RyZTAga2VybmVsOiBMdXN0cmVFcnJvcjog
MjA0MDoobGliLW1vdmUuYzoxMzQ6bGliX2ZpbmRfbWUoKSkgMTY3NzcyNDYx
OiBEcm9wcGluZyBQVVQgZnJvbSAxNjc3NzI0NjEuMCBwb3J0YWwgOCBtYXRj
aCAweDFjNmEgb2Zmc2V0IDAgbGVuZ3RoDQo1MjQyODg6IG5vIG1hdGNoDQpN
YXkgMjQgMjI6MzY6MTUgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVycm9yOiAy
MjI2Oihvc3RfaGFuZGxlci5jOjQ2ODpvc3RfYnJ3X3JlYWQoKSkgQEBAIHRp
bWVvdXQgb24gYnVsayBQVVQgcmVxQGM2NzgzYTAwIHg3MjcxL3QwIG8zLT48
Pz5ATklEXzEwLjAuMS40NV9VVUlEOi0xIGxlbnMgMjg4LzI0MCByZWYgMCBm
bCA/cGhhc2U/Oi8wLzAgcmMgMC8wDQpNYXkgMjQgMjI6MzY6MTUgbHVzdHJl
MCBrZXJuZWw6IEx1c3RyZUVycm9yOiAyMDQwOihsaWItbW92ZS5jOjEzNDps
aWJfZmluZF9tZSgpKSAxNjc3NzI0NjE6IERyb3BwaW5nIFBVVCBmcm9tIDE2
Nzc3MjQ2MS4wIHBvcnRhbCA0IG1hdGNoIDB4MWM2NyBvZmZzZXQgMCBsZW5n
dGgNCjI0MDogbm8gbWF0Y2gNCk1heSAyNCAyMjozNjo0MCBsdXN0cmUwIGtl
cm5lbDogTHVzdHJlRXJyb3I6IDIyMjU6KG9zdF9oYW5kbGVyLmM6NDY4Om9z
dF9icndfcmVhZCgpKSBAQEAgdGltZW91dCBvbiBidWxrIFBVVCByZXFAYzY3
ODNjMDAgeDcyNzQvdDAgbzMtPjw/PkBOSURfMTAuMC4xLjQ1X1VVSUQ6LTEg
bGVucyAyODgvMjQwIHJlZiAwIGZsID9waGFzZT86LzAvMCByYyAwLzANCk1h
eSAyNCAyMjozNjo0MCBsdXN0cmUwIGtlcm5lbDogTHVzdHJlRXJyb3I6IDIy
Mjc6KG9zdF9oYW5kbGVyLmM6NDY4Om9zdF9icndfcmVhZCgpKSBAQEAgdGlt
ZW91dCBvbiBidWxrIFBVVCByZXFAYzY3ODM4MDAgeDkyNS90MCBvMy0+PD8+
QDotMSBsZW5zIDI4OC8yNDAgcmVmIDANCiBmbCA/cGhhc2U/Oi8wLzAgcmMg
MC8wDQpNYXkgMjQgMjI6MzY6NDAgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVy
cm9yOiAyMjI3OihzZXJ2aWNlLmM6MzkzOnB0bHJwY19tYWluKCkpDQpEcm9w
cGluZyByZXF1ZXN0IGZyb20gTklEIDB4YTAwMDEyZSBiZWNhdXNlIGl0J3Mg
MTEwIHNlY29uZHMgb2xkLg0KTWF5IDI0IDIyOjM2OjQwIGx1c3RyZTAga2Vy
bmVsOiBMdXN0cmVFcnJvcjogMjA0MDoobGliLW1vdmUuYzoxMzQ6bGliX2Zp
bmRfbWUoKSkgMTY3NzcyNDYxOiBEcm9wcGluZyBQVVQgZnJvbSAxNjc3NzI0
NjEuMCBwb3J0YWwgNCBtYXRjaCAweDFjNmEgb2Zmc2V0IDAgbGVuZ3RoDQoy
NDA6IG5vIG1hdGNoDQpNYXkgMjQgMjI6MzY6NDAgbHVzdHJlMCBrZXJuZWw6
IEx1c3RyZUVycm9yOiAyMDQwOihsaWItbW92ZS5jOjEzNDpsaWJfZmluZF9t
ZSgpKSAxNjc3NzI0NjE6IERyb3BwaW5nIFBVVCBmcm9tIDE2Nzc3MjQ2MS4w
IHBvcnRhbCA4IG1hdGNoIDB4MWM2ZiBvZmZzZXQgMCBsZW5ndGgNCjUyNDI4
ODogbm8gbWF0Y2gNCk1heSAyNCAyMjozNjo0MCBsdXN0cmUwIGtlcm5lbDog
THVzdHJlRXJyb3I6IDIwNDE6KGxpYi1tb3ZlLmM6MTM0OmxpYl9maW5kX21l
KCkpIDE2Nzc3MjQ2MTogRHJvcHBpbmcgUFVUIGZyb20gMTY3NzcyNDYxLjAg
cG9ydGFsIDggbWF0Y2ggMHgxYzcwIG9mZnNldCAwIGxlbmd0aA0KNTI0Mjg4
OiBubyBtYXRjaA0KTWF5IDI0IDIyOjM2OjQwIGx1c3RyZTAga2VybmVsOiBM
dXN0cmVFcnJvcjogMjIyNjoob3N0X2hhbmRsZXIuYzo0Njg6b3N0X2Jyd19y
ZWFkKCkpIEBAQCB0aW1lb3V0IG9uIGJ1bGsgUFVUIHJlcUBjNjc4M2EwMCB4
OTI2L3QwIG8zLT48Pz5AOi0xIGxlbnMgMjg4LzI0MCByZWYgMCBmbCA/cGhh
c2U/Oi8wLzAgcmMgMC8wDQpNYXkgMjQgMjI6Mzc6MDUgbHVzdHJlMCBrZXJu
ZWw6IEx1c3RyZUVycm9yOiAyMjI1Oihvc3RfaGFuZGxlci5jOjQ2ODpvc3Rf
YnJ3X3JlYWQoKSkgQEBAIHRpbWVvdXQgb24gYnVsayBQVVQgcmVxQGM2Nzgz
YzAwIHg3Mjc5L3QwIG8zLT48Pz5ATklEXzEwLjAuMS40NV9VVUlEOi0xIGxl
bnMgMjg4LzI0MCByZWYgMCBmbCA/cGhhc2U/Oi8wLzAgcmMgMC8wDQpNYXkg
MjQgMjI6Mzc6MDUgbHVzdHJlMCBrZXJuZWw6IEx1c3RyZUVycm9yOiAyMjI3
Oihvc3RfaGFuZGxlci5jOjQ2ODpvc3RfYnJ3X3JlYWQoKSkgQEBAIHRpbWVv
dXQgb24gYnVsayBQVVQgcmVxQGM2NzgzODAwIHg3MjgwL3QwIG8zLT48Pz5A
TklEXzEwLjAuMS40NV9VVUlEOi0xIGxlbnMgMjg4LzI0MCByZWYgMCBmbCA/
cGhhc2U/Oi8wLzAgcmMgMC8wDQoNCg0KDQoNCg=
---559023410-824023566-1085418421=:25446--

On Mon, 2004-05-24 at 13:07, Kumaran Rajaram wrote:
> 
>    I have set up Lustre version 1.0.4 on a two node cluster. Node 1 is
> configured as MDS, LOV with 2 OSTS, and Client. Node is only configured as
> client. When I run parallel I/O programs (MPI-IO) which does repeated
> file I/O on a single file, it works fine if being run only on Node 2. When
> I include Node 1 as well, the application hangs. Sometimes it completes
> when number of parallel process executed on each node = 1 (but reads are
> very slow). When number of process allocated to each node is increased to
> 2 or more, the application hangs and crashes the file system (Lustre file
> system gives Input/Output Error). The log message is attached.
> 
>    Is this bug related to Bug #2227 since the messages seem to correspond.
> There is known bugs listed in Lustre Website "Runnign client and OST
on
> the same node is still not 100% stable. Application or system hangs are
> possible". Although, I run on nodes configured as OST and clients,
however
> I perform I/O on same file with chunk size 4KB and application hangs if
> each process on node reads/writes to same file multiple times (16-32).
> Each node has sufficient memory (128 MB RAM). Any advise/comments will be
> greatly appreciated.
You said it yourself: running a client and OST on the same node is not
yet stable.

-Phil

If you are running the InfiniCon InfiniBand stack, you should be able to improve
your performance significantly by increasing the number of receive buffers
allocated to the IPoIB protocol. This is done by modifying the "options
ipoib" line in /etc/modules.conf (on all your InfiniBand hosts) to look
like:

options ipoib IpoibRecvBuffers=3D2000 IpoibXmitBuffers=3D100

You will need to restart the ipoib driver or reboot to make these new settings
effective.

Duane.


-----Original Message-----
From: sky [mailto:eqzhou@263.net]
Sent: Tuesday, May 04, 2004 8:01 AM
To: lustre-discuss@lists.clusterfs.com
Subject: [Lustre-discuss] Bad performance on IA64


I have patched lustre into 2.4.20 kernel with little hacked code. It worked on
IA64.
But I achieved bad performance. Result of bonnie show that bandwidth of block
write is about 20MB/s.But NFS is about 40MB/s.Is it normal?=20
Any suggestion? Maybe something to be tuned to make it faster?

BTW Our paltform hardware: IA64+IPoIB (infiniband)+Ultra320 SCSI disk





=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
263=B5=E7=D7=D3=D3=CA=BC=FE=A3=AD=D0=C5=C0=B5=D3=CA=D7=D4=D7=A8=D2=B5.?=AD=E7b=B1=CB=AC=B2f=A2=96)=E0=96+-.?=AD=E7b=B1=CB=AC=B2X=AC=B6?=BA=CB^=AD=FB=1C=A2hm=B6=9B?=FEX=AC=B6?=BA=CB^=AD=FB=1C=A2o=E6j)fj=7F=E5=8A=CBb=9D=FA?=96=EB-=AD=E7b=B1=CB