On Jul 12, 2005 18:56 +0200, Roland Fehrenbacher wrote:> in lustre 1.2.4 lvfs/fsfilt_ext3.c the kernel function > journal_callback_set is called. This function and with it the struct > journal_callback has been removed from the kernel at around 2.6.7. Is > there a patch or susbtitute for this function, or how is one supposed > to get this to compile? > > ----------------- Compile error -------------------------------------- > CC [M] /lustre-1.2.4/lvfs/fsfilt-ldiskfs.s > /lustre-1.2.4/lvfs/fsfilt-ldiskfs.c:61:error: field `cb_jcb'' has incomplete type > /lustre-1.2.4/lvfs/fsfilt-ldiskfs.c: Infunction `fsfilt_ldiskfs_add_journal_cb'': > /lustre-1.2.4/lvfs/fsfilt-ldiskfs.c:670: warning: implicit declaration of function `journal_callback_set'' > make[4]: *** [/lustre-1.2.4/lvfs/fsfilt-ldiskfs.s] Error 1 > ----------------------------------------------------------------------Sadly, this was removed from ext3 behind my back, not for any particularly good reason so reinstating the old code is the right thing to do. None of the CFS supported kernels is missing this yet. There is a patch called jbd-2.4.18-jcberr.patch which added this feature into the 2.4.18 kernel originally. The code in that area is relatively static, so the original patch should still apply. Failing that, you can get the code out of the last 2.6 kernel that still has it. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
On Jul 13, 2005 00:06 +0200, Roland Fehrenbacher wrote:> Thanks for the quick reply. Would you mind having a look at the > appended patch which is against 2.6.11.11. Does it look ok?Looks like it did in 2.6.9, should be fine. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.
--hXXMjwULbw Content-Type: text/plain; charset=us-ascii Content-Description: message body text Content-Transfer-Encoding: 7bit>>>>> "Andreas" == Andreas Dilger <adilger@clusterfs.com> writes:Andreas> On Jul 12, 2005 18:56 +0200, Roland Fehrenbacher wrote: >> in lustre 1.2.4 lvfs/fsfilt_ext3.c the kernel function >> journal_callback_set is called. This function and with it the >> struct journal_callback has been removed from the kernel at >> around 2.6.7. Is there a patch or susbtitute for this function, >> or how is one supposed to get this to compile? Andreas> Sadly, this was removed from ext3 behind my back, not for Andreas> any particularly good reason so reinstating the old code Andreas> is the right thing to do. None of the CFS supported Andreas> kernels is missing this yet. Andreas> There is a patch called jbd-2.4.18-jcberr.patch which Andreas> added this feature into the 2.4.18 kernel originally. Andreas> The code in that area is relatively static, so the Andreas> original patch should still apply. Failing that, you can Andreas> get the code out of the last 2.6 kernel that still has Andreas> it. Thanks for the quick reply. Would you mind having a look at the appended patch which is against 2.6.11.11. Does it look ok? Cheers, Roland --hXXMjwULbw Content-Type: text/plain Content-Disposition: inline; filename="ql-journal-callback-2.6.12.patch" Content-Transfer-Encoding: 7bit --- linux-2.6.11.11-ibgd/fs/jbd/transaction.c Mon May 30 10:00:20 2005 +++ linux-2.6.11.11-ibgd-lustre/fs/jbd/transaction.c Mon May 30 10:00:02 2005 @@ -50,7 +50,9 @@ transaction->t_state = T_RUNNING; transaction->t_tid = journal->j_transaction_sequence++; transaction->t_expires = jiffies + journal->j_commit_interval; + INIT_LIST_HEAD(&transaction->t_jcb); spin_lock_init(&transaction->t_handle_lock); + spin_lock_init(&transaction->t_jcb_lock); /* Set up the commit timer for the new transaction. */ journal->j_commit_timer->expires = transaction->t_expires; @@ -241,6 +243,7 @@ memset(handle, 0, sizeof(*handle)); handle->h_buffer_credits = nblocks; handle->h_ref = 1; + INIT_LIST_HEAD(&handle->h_jcb); return handle; } @@ -1282,6 +1285,36 @@ } /** + * void journal_callback_set() - Register a callback function for this handle. + * @handle: handle to attach the callback to. + * @func: function to callback. + * @jcb: structure with additional information required by func() , and + * some space for jbd internal information. + * + * The function will be + * called when the transaction that this handle is part of has been + * committed to disk with the original callback data struct and the + * error status of the journal as parameters. There is no guarantee of + * ordering between handles within a single transaction, nor between + * callbacks registered on the same handle. + * + * The caller is responsible for allocating the journal_callback struct. + * This is to allow the caller to add as much extra data to the callback + * as needed, but reduce the overhead of multiple allocations. The caller + * allocated struct must start with a struct journal_callback at offset 0, + * and has the caller-specific data afterwards. + */ +void journal_callback_set(handle_t *handle, + void (*func)(struct journal_callback *jcb, int error), + struct journal_callback *jcb) +{ + spin_lock(&handle->h_transaction->t_jcb_lock); + list_add_tail(&jcb->jcb_list, &handle->h_jcb); + spin_unlock(&handle->h_transaction->t_jcb_lock); + jcb->jcb_func = func; +} + +/** * int journal_stop() - complete a transaction * @handle: tranaction to complete. * @@ -1346,6 +1379,11 @@ if (journal->j_barrier_count) wake_up(&journal->j_wait_transaction_locked); } + + /* Move callbacks from the handle to the transaction. */ + spin_lock(&transaction->t_jcb_lock); + list_splice(&handle->h_jcb, &transaction->t_jcb); + spin_unlock(&transaction->t_jcb_lock); /* * If the handle is marked SYNC, we need to set another commit --- linux-2.6.11.11-ibgd/fs/jbd/journal.c Wed Mar 2 14:33:32 2005 +++ linux-2.6.11.11-ibgd-lustre/fs/jbd/journal.c Wed Apr 20 15:49:02 2005 @@ -55,6 +55,7 @@ #endif EXPORT_SYMBOL(journal_flush); EXPORT_SYMBOL(journal_revoke); +EXPORT_SYMBOL(journal_callback_set); EXPORT_SYMBOL(journal_init_dev); EXPORT_SYMBOL(journal_init_inode); @@ -71,6 +72,7 @@ EXPORT_SYMBOL(journal_errno); EXPORT_SYMBOL(journal_ack_err); EXPORT_SYMBOL(journal_clear_err); +EXPORT_SYMBOL(log_start_commit); EXPORT_SYMBOL(log_wait_commit); EXPORT_SYMBOL(journal_start_commit); EXPORT_SYMBOL(journal_force_commit_nested); --- linux-2.6.11.11-ibgd/fs/jbd/commit.c Wed Mar 2 14:33:54 2005 +++ linux-2.6.11.11-ibgd-lustre/fs/jbd/commit.c Thu Apr 21 13:06:32 2005 @@ -689,6 +689,30 @@ if (err) __journal_abort_hard(journal); + /* + * Call any callbacks that had been registered for handles in this + * transaction. It is up to the callback to free any allocated + * memory. + * + * The spinlocking (t_jcb_lock) here is surely unnecessary... + */ + spin_lock(&commit_transaction->t_jcb_lock); + if (!list_empty(&commit_transaction->t_jcb)) { + struct list_head *p, *n; + int error = is_journal_aborted(journal); + + list_for_each_safe(p, n, &commit_transaction->t_jcb) { + struct journal_callback *jcb; + + jcb = list_entry(p, struct journal_callback, jcb_list); + list_del(p); + spin_unlock(&commit_transaction->t_jcb_lock); + jcb->jcb_func(jcb, error); + spin_lock(&commit_transaction->t_jcb_lock); + } + } + spin_unlock(&commit_transaction->t_jcb_lock); + jbd_debug(3, "JBD: commit phase 7\n"); J_ASSERT(commit_transaction->t_sync_datalist == NULL); --- linux-2.6.11.11-ibgd/include/linux/jbd.h Wed Mar 2 14:34:01 2005 +++ linux-2.6.11.11-ibgd-lustre/include/linux/jbd.h Thu Apr 21 12:58:33 2005 @@ -354,6 +354,12 @@ bit_spin_unlock(BH_JournalHead, &bh->b_state); } +struct journal_callback { + struct list_head jcb_list; /* t_jcb_lock */ + void (*jcb_func)(struct journal_callback *jcb, int error); + /* user data goes here */ +}; + struct jbd_revoke_table_s; /** @@ -387,6 +393,13 @@ /* operations */ int h_err; + /* + * List of application registered callbacks for this handle. The + * function(s) will be called after the transaction that this handle is + * part of has been committed to disk. [t_jcb_lock] + */ + struct list_head h_jcb; + /* Flags [no locking] */ unsigned int h_sync: 1; /* sync-on-close */ unsigned int h_jdata: 1; /* force data journaling */ @@ -551,6 +564,15 @@ */ int t_handle_count; + /* + * Protects the callback list + */ + spinlock_t t_jcb_lock; + /* + * List of registered callback functions for this transaction. + * Called when the transaction is committed. [t_jcb_lock] + */ + struct list_head t_jcb; }; /** @@ -883,6 +905,9 @@ extern int journal_try_to_free_buffers(journal_t *, struct page *, int); extern int journal_stop(handle_t *); extern int journal_flush (journal_t *); +extern void journal_callback_set(handle_t *handle, + void (*fn)(struct journal_callback *,int), + struct journal_callback *jcb); extern void journal_lock_updates (journal_t *); extern void journal_unlock_updates (journal_t *); --hXXMjwULbw--
>>>>> "Andreas" == Andreas Dilger <adilger@clusterfs.com> writes:Andreas> On Jul 13, 2005 00:06 +0200, Roland Fehrenbacher wrote: >> Thanks for the quick reply. Would you mind having a look at the >> appended patch which is against 2.6.11.11. Does it look ok? Andreas> Looks like it did in 2.6.9, should be fine. I have this kernel running now with lustre 1.2.4. Setup worked fine. I have one server for mds, 1 server for oss, and two servers as client for some initial tests. When I put some load on the client side, I start getting the following kernel messages on the oss. What is going wrong here? LustreError: 2324:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:240:filter_commitrw_write()) slow direct_io 17s LustreError: 2324:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:252:filter_commitrw_write()) slow commitrw commit 17s LustreError: 2301:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 20s LustreError: 2304:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 21s LustreError: 2314:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 22s LustreError: 2321:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 21s LustreError: 2314:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:240:filter_commitrw_write()) slow direct_io 22s LustreError: 2321:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:240:filter_commitrw_write()) slow direct_io 22s LustreError: 2321:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:252:filter_commitrw_write()) slow commitrw commit 22s LustreError: 2304:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:240:filter_commitrw_write()) slow direct_io 21s LustreError: 2304:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:252:filter_commitrw_write()) slow commitrw commit 21s LustreError: 2301:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:240:filter_commitrw_write()) slow direct_io 21s LustreError: 2301:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:252:filter_commitrw_write()) slow commitrw commit 21s LustreError: 2314:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:252:filter_commitrw_write()) slow commitrw commit 22s LustreError: 2327:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 21s LustreError: 2293:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 20s LustreError: 2297:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 20s LustreError: 2295:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 20s Thanks for your help, Roland
Hi, in lustre 1.2.4 lvfs/fsfilt_ext3.c the kernel function journal_callback_set is called. This function and with it the struct journal_callback has been removed from the kernel at around 2.6.7. Is there a patch or susbtitute for this function, or how is one supposed to get this to compile? ----------------- Compile error -------------------------------------- CC [M] /lustre-1.2.4/lvfs/fsfilt-ldiskfs.s /lustre-1.2.4/lvfs/fsfilt-ldiskfs.c:61:error: field `cb_jcb'' has incomplete type /lustre-1.2.4/lvfs/fsfilt-ldiskfs.c: Infunction `fsfilt_ldiskfs_add_journal_cb'': /lustre-1.2.4/lvfs/fsfilt-ldiskfs.c:670: warning: implicit declaration of function `journal_callback_set'' make[4]: *** [/lustre-1.2.4/lvfs/fsfilt-ldiskfs.s] Error 1 ---------------------------------------------------------------------- Thanks, Roland
On Jul 18, 2005 17:40 +0200, Roland Fehrenbacher wrote:> When I put some load on the client side, I > start getting the following kernel messages on the oss. What is going > wrong here? > > LustreError: 2324:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:240:filter_commitrw_write()) slow direct_io 17s > LustreError: 2324:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:252:filter_commitrw_write()) slow commitrw commit 17s > LustreError: 2301:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 20s > LustreError: 2304:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 21s > LustreError: 2314:0:(/scratch/build/system/beowulf/lustre/new/lustre-1.2.4/obdfilter/filter_io_26.c:159:filter_commitrw_write()) slow brw_start 22sThis means that your disk backend isn''t performing very well (taking 20s to complete writes). One source of this problem is if you are running a bad io scheduler. You should try booting the OST with "elevator=deadline" for the kernel boot option. Cheers, Andreas -- Andreas Dilger Principal Software Engineer Cluster File Systems, Inc.