Sorry for the noise. This request can be cancelled. I left out one
very important step in restoring the mdt, namely
rm OBJECTS/* CATALOGS
DOH! Umounted, removed via ldiskfs mount, remounted, and now all OST
are showing UP instead of IN.
bob
On 7/16/2013 9:22 AM, Bob Ball wrote:> I am just finishing an upgrade from Lustre 1.8.4 to 2.1.6, on SL5.8 to
> SL6.4. All OSS, MGS and MDT machines were upgraded. Following the
> upgrade I find that none of the OST will change to RW state. Could I
> get some advice from someone on this?
>
> We have separate mgs/mdt machines, and 8 OSS with ~70 OST attached.
> All OSS were upgraded and the OST were mounted. This morning, I
> started the MGS then the MDT. The MGS log entries in
> /var/log/messages shown here first, followed by MDT log entries, and
> the "lctl dl" output on the mdt.
>
> I made no file system changes on the OSS nodes. I rebuilt the mdt to
> a smaller size, using the setfattr/tar method described in the Lustre
> 1.8 manuals, with the mkfs.lustre issued from the newly built, 2.1.6 mdt.
>
> Have I missed anything here? Can someone tell me what is wrong with
> my system?
>
> Thanks,
> bob
>
> ===================== mgs log =================>
2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162354.732167]
> Lustre: Lustre: Build Version:
> RC2--PRISTINE-2.6.32-358.11.1.el6_lustre.x86_64
> 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162355.043533]
> Lustre: Added LNI 10.10.1.140@tcp [8/256/0/180]
> 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162355.043545]
> Lustre: Added LNI 141.213.133.196@tcp2 [8/256/0/180]
> 2013-07-16T08:28:15-04:00 mgs.aglt2.org kernel: [162355.043593]
> Lustre: Accept secure, port 988
> 2013-07-16T08:28:16-04:00 mgs.aglt2.org kernel: [162355.398195]
> Lustre: Lustre OSC module (ffffffffa08cc2a0).
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.433725]
> Lustre: Lustre LOV module (ffffffffa094a280).
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.544341]
> Lustre: Lustre client module (ffffffffa0a34920).
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.557307]
> LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck
> is recommended
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.557998]
> LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts:
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.590277]
> LDISKFS-fs (sdb): warning: maximal mount count reached, running e2fsck
> is recommended
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.591095]
> LDISKFS-fs (sdb): mounted filesystem with ordered data mode. Opts:
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.690541]
> Lustre: MGS MGS started
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.690795]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from 51233eb1-32bc-cb6f-3bbe-5008265553b7@0@lo t0 exp
> (null) cur 1373977697 last 0
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162356.697639]
> Lustre: MGC10.10.1.140@tcp: Reactivating import
> 2013-07-16T08:28:17-04:00 mgs.aglt2.org kernel: [162357.003523]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
2f75f773-faaf-d242-de3e-18443c3c45db-MJIdNl17AxNPYjMLAWrGXw@public.gmane.org@tcp
t0
> exp (null) cur 1373977697 last 0
> 2013-07-16T08:28:26-04:00 mgs.aglt2.org kernel: [162365.602114]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
f49a6fb6-030d-b53b-f515-6df58cceb09d-MJIdNl17AxOziwXqQIhgAg@public.gmane.org@tcp
t0
> exp (null) cur 1373977706 last 0
> 2013-07-16T08:28:54-04:00 mgs.aglt2.org kernel: [162394.024874]
> Lustre: 438424:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
bc883142-33e6-cef3-630e-0e3322f19033-MJIdNl17AxOGm3HmKFSLxA@public.gmane.org@tcp
t0
> exp (null) cur 1373977734 last 0
>
> [ The 10.10.1.157 was an old, 1.8.4 machine that I
> missed. I shut it down ]
> 2013-07-16T08:28:58-04:00 mgs.aglt2.org kernel: [162397.566576]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
847c3782-0835-5695-93e9-4276f7d7152a-cb8LArtinR6wOrX8ILxvRA@public.gmane.org@tcp
> t0 exp (null) cur 1373977738 last 0
>
> 2013-07-16T08:28:58-04:00 mgs.aglt2.org kernel: [162397.566580]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 1
> previous similar message
> 2013-07-16T08:29:05-04:00 mgs.aglt2.org kernel: [162405.195835]
> Lustre: 438424:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
118c0806-972c-3e9a-5de2-8bc902817bfe-MJIdNl17AxPR0h2xWRnnyg@public.gmane.org@tcp
t0
> exp (null) cur 1373977745 last 0
> 2013-07-16T08:29:14-04:00 mgs.aglt2.org kernel: [162413.238112]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
dbae7713-3f95-057a-28e1-7b6519506f64-MJIdNl17AxOlbpGMeQAFrQ@public.gmane.org@tcp
t0
> exp (null) cur 1373977754 last 0
>
> [ Now I start the mdt machine, and these log ]
> 2013-07-16T08:46:23-04:00 mgs.aglt2.org kernel: [163439.535234]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) MGS:
> connection from
12f9f864-0ad6-624f-115d-8801b67ca538-cb8LArtinR6yXd1OKpXlSw@public.gmane.org@tcp
t0
> exp (null) cur 1373978783 last 0
> 2013-07-16T08:46:23-04:00 mgs.aglt2.org kernel: [163439.535258]
> Lustre: 438423:0:(ldlm_lib.c:952:target_handle_connect()) Skipped 3
> previous similar messages
> 2013-07-16T08:46:23-04:00 mgs.aglt2.org kernel: [163439.590237]
> Lustre: 440552:0:(mgs_llog.c:249:mgs_fsdb_handler()) MDT using 1.8 OSC
> name scheme
>
> ======================= mdt log from /var/log/messages
=================> [ First mount attempt failed, added that
unsupported "-o
> update" I found ]
>
> 2013-07-16T08:45:43-04:00 lmd02.aglt2.org kernel: [61935.936127]
> Lustre: Lustre:
> Build Version: RC2--PRISTINE-2.6.32-358.11.1.el6_lustre.x86_64
> 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61936.409749]
> Lustre: Added L
> NI 10.10.1.49@tcp [8/256/0/180]
> 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61936.409845]
> Lustre: Accept
> secure, port 988
> 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61936.833435]
> Lustre: Lustre
> OSC module (ffffffffa0bd22a0).
> 2013-07-16T08:45:44-04:00 lmd02.aglt2.org kernel: [61937.003602]
> Lustre: Lustre
> LOV module (ffffffffa0c64280).
> 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.195301]
> Lustre: Lustre
> client module (ffffffffa0d4e920).
> 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.344529]
> LDISKFS-fs (dm-
> 7): Unrecognized mount option "upgrade" or missing value
> 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.360076]
> LustreError: 210489:0:(obd_mount.c:1360:server_kernel_mount())
> premount /dev/mapper/sun-mdt3:0x0 ldiskfs failed: -22 Is the ldiskfs
> module available?
> 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.389113]
> LustreError: 210489:0:(obd_mount.c:1700:server_fill_super()) Unable to
> mount device /dev/mapper
> /sun-mdt3: -22
> 2013-07-16T08:45:45-04:00 lmd02.aglt2.org kernel: [61937.411161]
> LustreError: 210489:0:(obd_mount.c:2203:lustre_fill_super()) Unable to
> mount (-22)
> [ end failed mount attempt ]
>
> 2013-07-16T08:46:22-04:00 lmd02.aglt2.org kernel: [61974.840288]
> LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts:
> 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.189666]
> LDISKFS-fs (dm-7): mounted filesystem with ordered data mode. Opts:
> 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.239509]
> Lustre: MGC10.10.1.140@tcp: Reactivating import
> 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.466165]
> Lustre: 210764:0:(obd_config.c:1396:class_config_llog_handler()) For
> 1.8 interoperability, rename obd type from mds to mdt
> 2013-07-16T08:46:23-04:00 lmd02.aglt2.org kernel: [61975.764633]
> Lustre: Enabling ACL
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.319567]
> Lustre: umt3-MDT0000: used disk, loading
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.343261]
> Lustre: Mounting umt3-MDT0000 at first time on 1.8 FS, remove all
> clients for interop needs
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.408147]
> LustreError: 210764:0:(llog_lvfs.c:616:llog_lvfs_create()) error
> looking up logfile 0x1f0fc001:0x45460a07: rc -116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.431151]
> LustreError: 210764:0:(llog_obd.c:220:llog_setup_named()) obd
> umt3-OST0000-osc ctxt 2 lop_setup
> =ffffffffa0777ca0 failed -116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.455805]
> LustreError: 210764:0:(osc_request.c:4278:__osc_llog_init()) failed
> LLOG_MDS_OST_ORIG_CTXT
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.474526]
> LustreError: 210764:0:(osc_request.c:4295:__osc_llog_init()) osc
> ''umt3-OST0000-osc'' tgt
''mdd_obd-umt3-MDT0000'' catid ffff8803ab8678b0
> rc=-116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.502137]
> LustreError: 210764:0:(osc_request.c:4297:__osc_llog_init()) logid
> 0x1f0fc001:0x45460a07
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.520514]
> LustreError: 210764:0:(osc_request.c:4325:osc_llog_init()) rc: -116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.535211]
> LustreError: 210764:0:(lov_log.c:248:lov_llog_init()) error
> osc_llog_init idx 0 osc ''umt3-OST0000-osc'' tgt
''mdd_obd-umt3-MDT0000''
> (rc=-116)
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.562854]
> LustreError: 210764:0:(llog_lvfs.c:616:llog_lvfs_create()) error
> looking up logfile 0x1f0fc002:0x45460a08: rc -116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.585795]
> LustreError: 210764:0:(llog_obd.c:220:llog_setup_named()) obd
> umt3-OST0001-osc ctxt 2 lop_setup=ffffffffa0777ca0 failed -116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610490]
> LustreError: 210764:0:(osc_request.c:4278:__osc_llog_init()) failed
> LLOG_MDS_OST_ORIG_CTXT
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610779]
> LustreError: 210793:0:(lov_log.c:160:lov_llog_origin_connect()) error
> osc_llog_connect tgt 0 (-
> 107)
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610785]
> LustreError: 210793:0:(mds_lov.c:832:__mds_lov_synchronize())
> umt3-OST0000_UUID failed at llog_origin_connect: -107
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610790]
> LustreError: 210793:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
> umt3-OST0000_UUID failed -107
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.610794]
> LustreError: 210793:0:(mds_lov.c:864:__mds_lov_synchronize())
> deactivating umt3-OST0000_UUID
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.711378]
> LustreError: 210764:0:(osc_request.c:4295:__osc_llog_init()) osc
> ''umt3-OST0001-osc'' tgt
''mdd_obd-umt3-MDT0000'' catid ffff8803ab8678b0
> rc=-116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.739028]
> LustreError: 210764:0:(osc_request.c:4297:__osc_llog_init()) logid
> 0x1f0fc002:0x45460a08
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.757420]
> LustreError: 210764:0:(osc_request.c:4325:osc_llog_init()) rc: -116
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.772127]
> LustreError: 210764:0:(lov_log.c:248:lov_llog_init()) error
> osc_llog_init idx 1 osc ''umt3-OST0001-osc'' tgt
''mdd_obd-umt3-MDT0000''
> (rc=-116)
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.800849]
> LustreError: 210794:0:(lov_log.c:160:lov_llog_origin_connect()) error
> osc_llog_connect tgt 1 (-107)
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.801867]
> LustreError: 21
> 0796:0:(mds_lov.c:832:__mds_lov_synchronize()) umt3-OST0003_UUID
> failed at llog_origin_connect: -107
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.801872]
> LustreError: 210796:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
> umt3-OST0003_UUID failed -107
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.801877]
> LustreError: 210796:0:(mds_lov.c:864:__mds_lov_synchronize())
> deactivating umt3-OST0003_UUID
> 2013-07-16T08:46:24-04:00 lmd02.aglt2.org kernel: [61976.883376]
> LustreError: 210794:0:(lov_log.c:160:lov_llog_origin_connect())
> Skipped 2 previous similar messages
> 2013-07-16T08:48:23-04:00 lmd02.aglt2.org kernel: [62095.558232]
> Lustre: 210777:0:(ldlm_lib.c:952:target_handle_connect())
> umt3-MDT0000: connection from
>
f1e54b83-bfa4-2f99-dfd2-c9a903f97369-nhsZqJ3hdYzNi1J8u2S7Uw@public.gmane.org@tcp2
> t533204841976 exp (null) cur 1373978903 last 0
>
> [ Noticed that all mounts were RO, try to activate as subset
> as RW ]
> [ eg, /usr/sbin/lctl --device umt3-OSG0022-osc activate ]
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.496490]
> Lustre: umt3-OST0022-osc: Connection to umt3-OST0022 (at
> 10.10.2.21@tcp) was lost; in progress operations using this service
> will wait for recovery to complete
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.496953]
> Lustre: umt3-OST0022-osc: Connection restored to umt3-OST0022 (at
> 10.10.2.21@tcp)
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.497680]
> LustreError: 213631:0:(lov_log.c:160:lov_llog_origin_connect()) error
> osc_llog_connect tgt 34 (
> -107)
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.498190]
> Lustre: umt3-OST0023-osc: Connection to umt3-OST0023 (at
> 10.10.2.21@tcp) was lost; in progress operations using this service
> will wait for recovery to complete
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.518189]
> LustreError: 213631:0:(lov_log.c:160:lov_llog_origin_connect())
> Skipped 62 previous similar messages
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.518270]
> Lustre: umt3-OST0023-osc: Connection restored to umt3-OST0023 (at
> 10.10.2.21@tcp)
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519058]
> LustreError: 213633:0:(mds_lov.c:832:__mds_lov_synchronize())
> umt3-OST0023_UUID failed at llog_origin_connect: -107
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519062]
> LustreError: 213633:0:(mds_lov.c:832:__mds_lov_synchronize()) Skipped
> 68 previous similar messages
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519067]
> LustreError: 213633:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
> umt3-OST0023_UUID failed -107
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519070]
> LustreError: 213633:0:(mds_lov.c:860:__mds_lov_synchronize()) Skipped
> 68 previous similar messages
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519075]
> LustreError: 213633:0:(mds_lov.c:864:__mds_lov_synchronize())
> deactivating umt3-OST0023_UUID
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.519078]
> LustreError: 213633:0:(mds_lov.c:864:__mds_lov_synchronize()) Skipped
> 68 previous similar messages
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.660863]
> LustreError: 213631:0:(mds_lov.c:832:__mds_lov_synchronize())
> umt3-OST0022_UUID failed at llog_origin_connect: -107
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.661660]
> LustreError: 213635:0:(lov_log.c:160:lov_llog_origin_connect()) error
> osc_llog_connect tgt 36 (-107)
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.661669]
> LustreError: 213635:0:(mds_lov.c:860:__mds_lov_synchronize()) sync
> umt3-OST0024_UUID failed -107
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.661674]
> LustreError: 213635:0:(mds_lov.c:864:__mds_lov_synchronize())
> deactivating umt3-OST0024_UUID
> 2013-07-16T08:59:04-04:00 lmd02.aglt2.org kernel: [62736.743206]
> LustreError: 213631:0:(mds_lov.c:832:__mds_lov_synchronize()) Skipped
> 1 previous similar message
>
> ====================================> On the mdt node, I see this.
> [root@lmd02 ~]# lctl dl
> 0 UP mgc MGC10.10.1.140@tcp 12f9f864-0ad6-624f-115d-8801b67ca538 5
> 1 UP lov umt3-mdtlov umt3-mdtlov_UUID 4
> 2 UP mdt umt3-MDT0000 umt3-MDT0000_UUID 5
> 3 UP mds mdd_obd-umt3-MDT0000 mdd_obd_uuid-umt3-MDT0000 3
> 4 IN osc umt3-OST0000-osc umt3-mdtlov_UUID 5
> 5 IN osc umt3-OST0001-osc umt3-mdtlov_UUID 5
> 6 IN osc umt3-OST0002-osc umt3-mdtlov_UUID 5
> 7 IN osc umt3-OST0003-osc umt3-mdtlov_UUID 5
> 8 IN osc umt3-OST0004-osc umt3-mdtlov_UUID 5
> 9 IN osc umt3-OST0005-osc umt3-mdtlov_UUID 5
> 10 IN osc umt3-OST0006-osc umt3-mdtlov_UUID 5
> 11 IN osc umt3-OST0007-osc umt3-mdtlov_UUID 5
> 12 IN osc umt3-OST0010-osc umt3-mdtlov_UUID 5
> 13 IN osc umt3-OST0011-osc umt3-mdtlov_UUID 5
> 14 IN osc umt3-OST0012-osc umt3-mdtlov_UUID 5
> 15 IN osc umt3-OST0013-osc umt3-mdtlov_UUID 5
> 16 IN osc umt3-OST0014-osc umt3-mdtlov_UUID 5
> 17 IN osc umt3-OST0015-osc umt3-mdtlov_UUID 5
> 18 IN osc umt3-OST0016-osc umt3-mdtlov_UUID 5
> 19 IN osc umt3-OST0017-osc umt3-mdtlov_UUID 5
> 20 IN osc umt3-OST0018-osc umt3-mdtlov_UUID 5
> 21 IN osc umt3-OST0019-osc umt3-mdtlov_UUID 5
> 22 IN osc umt3-OST001a-osc umt3-mdtlov_UUID 5
> 23 IN osc umt3-OST001b-osc umt3-mdtlov_UUID 5
> 24 IN osc umt3-OST0008-osc umt3-mdtlov_UUID 5
> 25 IN osc umt3-OST0009-osc umt3-mdtlov_UUID 5
> 26 IN osc umt3-OST000a-osc umt3-mdtlov_UUID 5
> 27 IN osc umt3-OST000b-osc umt3-mdtlov_UUID 5
> 28 IN osc umt3-OST000c-osc umt3-mdtlov_UUID 5
> 29 IN osc umt3-OST000d-osc umt3-mdtlov_UUID 5
> 30 IN osc umt3-OST000e-osc umt3-mdtlov_UUID 5
> 31 IN osc umt3-OST000f-osc umt3-mdtlov_UUID 5
> 32 IN osc umt3-OST001c-osc umt3-mdtlov_UUID 5
> 33 IN osc umt3-OST001d-osc umt3-mdtlov_UUID 5
> 34 IN osc umt3-OST001e-osc umt3-mdtlov_UUID 5
> 35 IN osc umt3-OST001f-osc umt3-mdtlov_UUID 5
> 36 IN osc umt3-OST0020-osc umt3-mdtlov_UUID 5
> 37 IN osc umt3-OST0021-osc umt3-mdtlov_UUID 5
> 38 IN osc umt3-OST0022-osc umt3-mdtlov_UUID 5
> 39 IN osc umt3-OST0023-osc umt3-mdtlov_UUID 5
> 40 IN osc umt3-OST0024-osc umt3-mdtlov_UUID 5
> 41 IN osc umt3-OST0025-osc umt3-mdtlov_UUID 5
> 42 IN osc umt3-OST0026-osc umt3-mdtlov_UUID 5
> 43 IN osc umt3-OST0027-osc umt3-mdtlov_UUID 5
> 44 IN osc umt3-OST0028-osc umt3-mdtlov_UUID 5
> 45 IN osc umt3-OST0029-osc umt3-mdtlov_UUID 5
> 46 IN osc umt3-OST002a-osc umt3-mdtlov_UUID 5
> 47 IN osc umt3-OST002b-osc umt3-mdtlov_UUID 5
> 48 IN osc umt3-OST002c-osc umt3-mdtlov_UUID 5
> 49 IN osc umt3-OST002d-osc umt3-mdtlov_UUID 5
> 50 IN osc umt3-OST002e-osc umt3-mdtlov_UUID 5
> 51 IN osc umt3-OST002f-osc umt3-mdtlov_UUID 5
> 52 IN osc umt3-OST0030-osc umt3-mdtlov_UUID 5
> 53 IN osc umt3-OST0031-osc umt3-mdtlov_UUID 5
> 54 IN osc umt3-OST0032-osc umt3-mdtlov_UUID 5
> 55 IN osc umt3-OST0033-osc umt3-mdtlov_UUID 5
> 56 IN osc umt3-OST0034-osc umt3-mdtlov_UUID 5
> 57 IN osc umt3-OST0035-osc umt3-mdtlov_UUID 5
> 58 IN osc umt3-OST0036-osc umt3-mdtlov_UUID 5
> 59 IN osc umt3-OST0037-osc umt3-mdtlov_UUID 5
> 60 IN osc umt3-OST0038-osc umt3-mdtlov_UUID 5
> 61 IN osc umt3-OST0039-osc umt3-mdtlov_UUID 5
> 62 IN osc umt3-OST003a-osc umt3-mdtlov_UUID 5
> 63 IN osc umt3-OST003b-osc umt3-mdtlov_UUID 5
> 64 IN osc umt3-OST003c-osc umt3-mdtlov_UUID 5
> 65 IN osc umt3-OST003d-osc umt3-mdtlov_UUID 5
> 66 IN osc umt3-OST003e-osc umt3-mdtlov_UUID 5
> 67 IN osc umt3-OST003f-osc umt3-mdtlov_UUID 5
> 68 IN osc umt3-OST0040-osc umt3-mdtlov_UUID 5
> 69 IN osc umt3-OST0041-osc umt3-mdtlov_UUID 5
> 70 IN osc umt3-OST0042-osc umt3-mdtlov_UUID 5
> 71 IN osc umt3-OST0043-osc umt3-mdtlov_UUID 5
> 72 IN osc umt3-OST0044-osc umt3-mdtlov_UUID 5
> 73 IN osc umt3-OST0045-osc umt3-mdtlov_UUID 5
>
>